Oct 30 20:37:15.080837 ip-10-0-138-149 systemd[1]: Starting Kubernetes Kubelet... Oct 30 20:37:15.454935 ip-10-0-138-149 kubenswrapper[2573]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Oct 30 20:37:15.454935 ip-10-0-138-149 kubenswrapper[2573]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Oct 30 20:37:15.454935 ip-10-0-138-149 kubenswrapper[2573]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Oct 30 20:37:15.454935 ip-10-0-138-149 kubenswrapper[2573]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Oct 30 20:37:15.461404 ip-10-0-138-149 kubenswrapper[2573]: W1030 20:37:15.461321 2573 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 20:37:15.461404 ip-10-0-138-149 kubenswrapper[2573]: W1030 20:37:15.461405 2573 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 20:37:15.461476 ip-10-0-138-149 kubenswrapper[2573]: W1030 20:37:15.461409 2573 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 20:37:15.461476 ip-10-0-138-149 kubenswrapper[2573]: W1030 20:37:15.461412 2573 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 20:37:15.461476 ip-10-0-138-149 kubenswrapper[2573]: W1030 20:37:15.461416 2573 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 20:37:15.461476 ip-10-0-138-149 kubenswrapper[2573]: W1030 20:37:15.461418 2573 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 20:37:15.461476 ip-10-0-138-149 kubenswrapper[2573]: W1030 20:37:15.461422 2573 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 20:37:15.461476 ip-10-0-138-149 kubenswrapper[2573]: W1030 20:37:15.461429 2573 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 20:37:15.461476 ip-10-0-138-149 kubenswrapper[2573]: W1030 20:37:15.461432 2573 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 20:37:15.461476 ip-10-0-138-149 kubenswrapper[2573]: W1030 20:37:15.461435 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 20:37:15.461476 ip-10-0-138-149 kubenswrapper[2573]: W1030 20:37:15.461437 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 20:37:15.461476 ip-10-0-138-149 kubenswrapper[2573]: W1030 20:37:15.461440 2573 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 20:37:15.461476 ip-10-0-138-149 kubenswrapper[2573]: W1030 20:37:15.461443 2573 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 20:37:15.461476 ip-10-0-138-149 kubenswrapper[2573]: W1030 20:37:15.461445 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 20:37:15.461476 ip-10-0-138-149 kubenswrapper[2573]: W1030 20:37:15.461448 2573 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 20:37:15.461476 ip-10-0-138-149 kubenswrapper[2573]: W1030 20:37:15.461451 2573 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 20:37:15.461476 ip-10-0-138-149 kubenswrapper[2573]: W1030 20:37:15.461453 2573 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 20:37:15.461476 ip-10-0-138-149 kubenswrapper[2573]: W1030 20:37:15.461456 2573 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 20:37:15.461476 ip-10-0-138-149 kubenswrapper[2573]: W1030 20:37:15.461459 2573 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 20:37:15.461476 ip-10-0-138-149 kubenswrapper[2573]: W1030 20:37:15.461461 2573 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 20:37:15.461476 ip-10-0-138-149 kubenswrapper[2573]: W1030 20:37:15.461464 2573 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 20:37:15.461930 ip-10-0-138-149 kubenswrapper[2573]: W1030 20:37:15.461466 2573 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 20:37:15.461930 ip-10-0-138-149 kubenswrapper[2573]: W1030 20:37:15.461469 2573 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 20:37:15.461930 ip-10-0-138-149 kubenswrapper[2573]: W1030 20:37:15.461472 2573 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 20:37:15.461930 ip-10-0-138-149 kubenswrapper[2573]: W1030 20:37:15.461475 2573 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 20:37:15.461930 ip-10-0-138-149 kubenswrapper[2573]: W1030 20:37:15.461477 2573 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 20:37:15.461930 ip-10-0-138-149 kubenswrapper[2573]: W1030 20:37:15.461480 2573 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 20:37:15.461930 ip-10-0-138-149 kubenswrapper[2573]: W1030 20:37:15.461485 2573 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 20:37:15.461930 ip-10-0-138-149 kubenswrapper[2573]: W1030 20:37:15.461489 2573 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 20:37:15.461930 ip-10-0-138-149 kubenswrapper[2573]: W1030 20:37:15.461492 2573 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 20:37:15.461930 ip-10-0-138-149 kubenswrapper[2573]: W1030 20:37:15.461496 2573 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 20:37:15.461930 ip-10-0-138-149 kubenswrapper[2573]: W1030 20:37:15.461498 2573 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 20:37:15.461930 ip-10-0-138-149 kubenswrapper[2573]: W1030 20:37:15.461501 2573 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 20:37:15.461930 ip-10-0-138-149 kubenswrapper[2573]: W1030 20:37:15.461504 2573 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 20:37:15.461930 ip-10-0-138-149 kubenswrapper[2573]: W1030 20:37:15.461506 2573 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 20:37:15.461930 ip-10-0-138-149 kubenswrapper[2573]: W1030 20:37:15.461509 2573 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 20:37:15.461930 ip-10-0-138-149 kubenswrapper[2573]: W1030 20:37:15.461512 2573 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 20:37:15.461930 ip-10-0-138-149 kubenswrapper[2573]: W1030 20:37:15.461514 2573 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 20:37:15.461930 ip-10-0-138-149 kubenswrapper[2573]: W1030 20:37:15.461516 2573 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 20:37:15.461930 ip-10-0-138-149 kubenswrapper[2573]: W1030 20:37:15.461519 2573 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 20:37:15.462374 ip-10-0-138-149 kubenswrapper[2573]: W1030 20:37:15.461522 2573 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 20:37:15.462374 ip-10-0-138-149 kubenswrapper[2573]: W1030 20:37:15.461525 2573 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 20:37:15.462374 ip-10-0-138-149 kubenswrapper[2573]: W1030 20:37:15.461527 2573 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 20:37:15.462374 ip-10-0-138-149 kubenswrapper[2573]: W1030 20:37:15.461530 2573 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 20:37:15.462374 ip-10-0-138-149 kubenswrapper[2573]: W1030 20:37:15.461532 2573 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 20:37:15.462374 ip-10-0-138-149 kubenswrapper[2573]: W1030 20:37:15.461535 2573 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 20:37:15.462374 ip-10-0-138-149 kubenswrapper[2573]: W1030 20:37:15.461537 2573 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 20:37:15.462374 ip-10-0-138-149 kubenswrapper[2573]: W1030 20:37:15.461540 2573 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 20:37:15.462374 ip-10-0-138-149 kubenswrapper[2573]: W1030 20:37:15.461544 2573 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 20:37:15.462374 ip-10-0-138-149 kubenswrapper[2573]: W1030 20:37:15.461548 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 20:37:15.462374 ip-10-0-138-149 kubenswrapper[2573]: W1030 20:37:15.461551 2573 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 20:37:15.462374 ip-10-0-138-149 kubenswrapper[2573]: W1030 20:37:15.461555 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 20:37:15.462374 ip-10-0-138-149 kubenswrapper[2573]: W1030 20:37:15.461558 2573 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 20:37:15.462374 ip-10-0-138-149 kubenswrapper[2573]: W1030 20:37:15.461573 2573 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 20:37:15.462374 ip-10-0-138-149 kubenswrapper[2573]: W1030 20:37:15.461577 2573 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 20:37:15.462374 ip-10-0-138-149 kubenswrapper[2573]: W1030 20:37:15.461580 2573 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 20:37:15.462374 ip-10-0-138-149 kubenswrapper[2573]: W1030 20:37:15.461583 2573 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 20:37:15.462374 ip-10-0-138-149 kubenswrapper[2573]: W1030 20:37:15.461586 2573 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 20:37:15.462374 ip-10-0-138-149 kubenswrapper[2573]: W1030 20:37:15.461588 2573 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 20:37:15.462877 ip-10-0-138-149 kubenswrapper[2573]: W1030 20:37:15.461591 2573 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 20:37:15.462877 ip-10-0-138-149 kubenswrapper[2573]: W1030 20:37:15.461593 2573 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 20:37:15.462877 ip-10-0-138-149 kubenswrapper[2573]: W1030 20:37:15.461596 2573 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 20:37:15.462877 ip-10-0-138-149 kubenswrapper[2573]: W1030 20:37:15.461599 2573 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 20:37:15.462877 ip-10-0-138-149 kubenswrapper[2573]: W1030 20:37:15.461601 2573 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 20:37:15.462877 ip-10-0-138-149 kubenswrapper[2573]: W1030 20:37:15.461604 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 20:37:15.462877 ip-10-0-138-149 kubenswrapper[2573]: W1030 20:37:15.461607 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 20:37:15.462877 ip-10-0-138-149 kubenswrapper[2573]: W1030 20:37:15.461610 2573 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 20:37:15.462877 ip-10-0-138-149 kubenswrapper[2573]: W1030 20:37:15.461613 2573 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 20:37:15.462877 ip-10-0-138-149 kubenswrapper[2573]: W1030 20:37:15.461615 2573 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 20:37:15.462877 ip-10-0-138-149 kubenswrapper[2573]: W1030 20:37:15.461618 2573 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 20:37:15.462877 ip-10-0-138-149 kubenswrapper[2573]: W1030 20:37:15.461621 2573 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 20:37:15.462877 ip-10-0-138-149 kubenswrapper[2573]: W1030 20:37:15.461623 2573 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 20:37:15.462877 ip-10-0-138-149 kubenswrapper[2573]: W1030 20:37:15.461626 2573 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 20:37:15.462877 ip-10-0-138-149 kubenswrapper[2573]: W1030 20:37:15.461629 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 20:37:15.462877 ip-10-0-138-149 kubenswrapper[2573]: W1030 20:37:15.461631 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 20:37:15.462877 ip-10-0-138-149 kubenswrapper[2573]: W1030 20:37:15.461634 2573 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 20:37:15.462877 ip-10-0-138-149 kubenswrapper[2573]: W1030 20:37:15.461636 2573 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 20:37:15.462877 ip-10-0-138-149 kubenswrapper[2573]: W1030 20:37:15.461639 2573 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 20:37:15.463323 ip-10-0-138-149 kubenswrapper[2573]: W1030 20:37:15.461642 2573 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 20:37:15.463323 ip-10-0-138-149 kubenswrapper[2573]: W1030 20:37:15.461645 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 20:37:15.463323 ip-10-0-138-149 kubenswrapper[2573]: W1030 20:37:15.461647 2573 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 20:37:15.463323 ip-10-0-138-149 kubenswrapper[2573]: W1030 20:37:15.461650 2573 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 20:37:15.463323 ip-10-0-138-149 kubenswrapper[2573]: W1030 20:37:15.461653 2573 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 20:37:15.463323 ip-10-0-138-149 kubenswrapper[2573]: W1030 20:37:15.461655 2573 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 20:37:15.463323 ip-10-0-138-149 kubenswrapper[2573]: W1030 20:37:15.461658 2573 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 20:37:15.463323 ip-10-0-138-149 kubenswrapper[2573]: W1030 20:37:15.461661 2573 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 20:37:15.463323 ip-10-0-138-149 kubenswrapper[2573]: W1030 20:37:15.461664 2573 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 20:37:15.463323 ip-10-0-138-149 kubenswrapper[2573]: W1030 20:37:15.461667 2573 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 20:37:15.463323 ip-10-0-138-149 kubenswrapper[2573]: W1030 20:37:15.461669 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 20:37:15.463323 ip-10-0-138-149 kubenswrapper[2573]: W1030 20:37:15.461672 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 20:37:15.463323 ip-10-0-138-149 kubenswrapper[2573]: W1030 20:37:15.461674 2573 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 20:37:15.463323 ip-10-0-138-149 kubenswrapper[2573]: W1030 20:37:15.461677 2573 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 20:37:15.463323 ip-10-0-138-149 kubenswrapper[2573]: W1030 20:37:15.461679 2573 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 20:37:15.463323 ip-10-0-138-149 kubenswrapper[2573]: W1030 20:37:15.461682 2573 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 20:37:15.463323 ip-10-0-138-149 kubenswrapper[2573]: W1030 20:37:15.461685 2573 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 20:37:15.463323 ip-10-0-138-149 kubenswrapper[2573]: W1030 20:37:15.461688 2573 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 20:37:15.463323 ip-10-0-138-149 kubenswrapper[2573]: W1030 20:37:15.461690 2573 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 20:37:15.463323 ip-10-0-138-149 kubenswrapper[2573]: W1030 20:37:15.461693 2573 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 20:37:15.463323 ip-10-0-138-149 kubenswrapper[2573]: W1030 20:37:15.461696 2573 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 20:37:15.463897 ip-10-0-138-149 kubenswrapper[2573]: W1030 20:37:15.461699 2573 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 20:37:15.463897 ip-10-0-138-149 kubenswrapper[2573]: W1030 20:37:15.461702 2573 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 20:37:15.463897 ip-10-0-138-149 kubenswrapper[2573]: W1030 20:37:15.461705 2573 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 20:37:15.463897 ip-10-0-138-149 kubenswrapper[2573]: W1030 20:37:15.461707 2573 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 20:37:15.463897 ip-10-0-138-149 kubenswrapper[2573]: W1030 20:37:15.461710 2573 feature_gate.go:386] unrecognized feature gate: Example Oct 30 20:37:15.463897 ip-10-0-138-149 kubenswrapper[2573]: W1030 20:37:15.461712 2573 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 20:37:15.463897 ip-10-0-138-149 kubenswrapper[2573]: W1030 20:37:15.461715 2573 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 20:37:15.463897 ip-10-0-138-149 kubenswrapper[2573]: W1030 20:37:15.462458 2573 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 20:37:15.463897 ip-10-0-138-149 kubenswrapper[2573]: W1030 20:37:15.462465 2573 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 20:37:15.463897 ip-10-0-138-149 kubenswrapper[2573]: W1030 20:37:15.462468 2573 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 20:37:15.463897 ip-10-0-138-149 kubenswrapper[2573]: W1030 20:37:15.462471 2573 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 20:37:15.463897 ip-10-0-138-149 kubenswrapper[2573]: W1030 20:37:15.462474 2573 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 20:37:15.463897 ip-10-0-138-149 kubenswrapper[2573]: W1030 20:37:15.462477 2573 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 20:37:15.463897 ip-10-0-138-149 kubenswrapper[2573]: W1030 20:37:15.462480 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 20:37:15.463897 ip-10-0-138-149 kubenswrapper[2573]: W1030 20:37:15.462482 2573 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 20:37:15.463897 ip-10-0-138-149 kubenswrapper[2573]: W1030 20:37:15.462485 2573 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 20:37:15.463897 ip-10-0-138-149 kubenswrapper[2573]: W1030 20:37:15.462489 2573 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 20:37:15.463897 ip-10-0-138-149 kubenswrapper[2573]: W1030 20:37:15.462492 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 20:37:15.463897 ip-10-0-138-149 kubenswrapper[2573]: W1030 20:37:15.462494 2573 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 20:37:15.463897 ip-10-0-138-149 kubenswrapper[2573]: W1030 20:37:15.462497 2573 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 20:37:15.464381 ip-10-0-138-149 kubenswrapper[2573]: W1030 20:37:15.462500 2573 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 20:37:15.464381 ip-10-0-138-149 kubenswrapper[2573]: W1030 20:37:15.462503 2573 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 20:37:15.464381 ip-10-0-138-149 kubenswrapper[2573]: W1030 20:37:15.462506 2573 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 20:37:15.464381 ip-10-0-138-149 kubenswrapper[2573]: W1030 20:37:15.462508 2573 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 20:37:15.464381 ip-10-0-138-149 kubenswrapper[2573]: W1030 20:37:15.462511 2573 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 20:37:15.464381 ip-10-0-138-149 kubenswrapper[2573]: W1030 20:37:15.462514 2573 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 20:37:15.464381 ip-10-0-138-149 kubenswrapper[2573]: W1030 20:37:15.462516 2573 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 20:37:15.464381 ip-10-0-138-149 kubenswrapper[2573]: W1030 20:37:15.462519 2573 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 20:37:15.464381 ip-10-0-138-149 kubenswrapper[2573]: W1030 20:37:15.462521 2573 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 20:37:15.464381 ip-10-0-138-149 kubenswrapper[2573]: W1030 20:37:15.462524 2573 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 20:37:15.464381 ip-10-0-138-149 kubenswrapper[2573]: W1030 20:37:15.462526 2573 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 20:37:15.464381 ip-10-0-138-149 kubenswrapper[2573]: W1030 20:37:15.462529 2573 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 20:37:15.464381 ip-10-0-138-149 kubenswrapper[2573]: W1030 20:37:15.462532 2573 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 20:37:15.464381 ip-10-0-138-149 kubenswrapper[2573]: W1030 20:37:15.462535 2573 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 20:37:15.464381 ip-10-0-138-149 kubenswrapper[2573]: W1030 20:37:15.462538 2573 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 20:37:15.464381 ip-10-0-138-149 kubenswrapper[2573]: W1030 20:37:15.462541 2573 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 20:37:15.464381 ip-10-0-138-149 kubenswrapper[2573]: W1030 20:37:15.462543 2573 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 20:37:15.464381 ip-10-0-138-149 kubenswrapper[2573]: W1030 20:37:15.462545 2573 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 20:37:15.464381 ip-10-0-138-149 kubenswrapper[2573]: W1030 20:37:15.462548 2573 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 20:37:15.464381 ip-10-0-138-149 kubenswrapper[2573]: W1030 20:37:15.462551 2573 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 20:37:15.464880 ip-10-0-138-149 kubenswrapper[2573]: W1030 20:37:15.462553 2573 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 20:37:15.464880 ip-10-0-138-149 kubenswrapper[2573]: W1030 20:37:15.462556 2573 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 20:37:15.464880 ip-10-0-138-149 kubenswrapper[2573]: W1030 20:37:15.462558 2573 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 20:37:15.464880 ip-10-0-138-149 kubenswrapper[2573]: W1030 20:37:15.462578 2573 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 20:37:15.464880 ip-10-0-138-149 kubenswrapper[2573]: W1030 20:37:15.462583 2573 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 20:37:15.464880 ip-10-0-138-149 kubenswrapper[2573]: W1030 20:37:15.462587 2573 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 20:37:15.464880 ip-10-0-138-149 kubenswrapper[2573]: W1030 20:37:15.462591 2573 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 20:37:15.464880 ip-10-0-138-149 kubenswrapper[2573]: W1030 20:37:15.462594 2573 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 20:37:15.464880 ip-10-0-138-149 kubenswrapper[2573]: W1030 20:37:15.462598 2573 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 20:37:15.464880 ip-10-0-138-149 kubenswrapper[2573]: W1030 20:37:15.462601 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 20:37:15.464880 ip-10-0-138-149 kubenswrapper[2573]: W1030 20:37:15.462604 2573 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 20:37:15.464880 ip-10-0-138-149 kubenswrapper[2573]: W1030 20:37:15.462608 2573 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 20:37:15.464880 ip-10-0-138-149 kubenswrapper[2573]: W1030 20:37:15.462610 2573 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 20:37:15.464880 ip-10-0-138-149 kubenswrapper[2573]: W1030 20:37:15.462615 2573 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 20:37:15.464880 ip-10-0-138-149 kubenswrapper[2573]: W1030 20:37:15.462618 2573 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 20:37:15.464880 ip-10-0-138-149 kubenswrapper[2573]: W1030 20:37:15.462621 2573 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 20:37:15.464880 ip-10-0-138-149 kubenswrapper[2573]: W1030 20:37:15.462623 2573 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 20:37:15.464880 ip-10-0-138-149 kubenswrapper[2573]: W1030 20:37:15.462626 2573 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 20:37:15.464880 ip-10-0-138-149 kubenswrapper[2573]: W1030 20:37:15.462629 2573 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 20:37:15.464880 ip-10-0-138-149 kubenswrapper[2573]: W1030 20:37:15.462632 2573 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 20:37:15.465358 ip-10-0-138-149 kubenswrapper[2573]: W1030 20:37:15.462634 2573 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 20:37:15.465358 ip-10-0-138-149 kubenswrapper[2573]: W1030 20:37:15.462637 2573 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 20:37:15.465358 ip-10-0-138-149 kubenswrapper[2573]: W1030 20:37:15.462639 2573 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 20:37:15.465358 ip-10-0-138-149 kubenswrapper[2573]: W1030 20:37:15.462642 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 20:37:15.465358 ip-10-0-138-149 kubenswrapper[2573]: W1030 20:37:15.462645 2573 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 20:37:15.465358 ip-10-0-138-149 kubenswrapper[2573]: W1030 20:37:15.462648 2573 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 20:37:15.465358 ip-10-0-138-149 kubenswrapper[2573]: W1030 20:37:15.462651 2573 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 20:37:15.465358 ip-10-0-138-149 kubenswrapper[2573]: W1030 20:37:15.462653 2573 feature_gate.go:386] unrecognized feature gate: Example Oct 30 20:37:15.465358 ip-10-0-138-149 kubenswrapper[2573]: W1030 20:37:15.462656 2573 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 20:37:15.465358 ip-10-0-138-149 kubenswrapper[2573]: W1030 20:37:15.462659 2573 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 20:37:15.465358 ip-10-0-138-149 kubenswrapper[2573]: W1030 20:37:15.462661 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 20:37:15.465358 ip-10-0-138-149 kubenswrapper[2573]: W1030 20:37:15.462664 2573 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 20:37:15.465358 ip-10-0-138-149 kubenswrapper[2573]: W1030 20:37:15.462667 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 20:37:15.465358 ip-10-0-138-149 kubenswrapper[2573]: W1030 20:37:15.462669 2573 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 20:37:15.465358 ip-10-0-138-149 kubenswrapper[2573]: W1030 20:37:15.462671 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 20:37:15.465358 ip-10-0-138-149 kubenswrapper[2573]: W1030 20:37:15.462674 2573 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 20:37:15.465358 ip-10-0-138-149 kubenswrapper[2573]: W1030 20:37:15.462677 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 20:37:15.465358 ip-10-0-138-149 kubenswrapper[2573]: W1030 20:37:15.462680 2573 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 20:37:15.465358 ip-10-0-138-149 kubenswrapper[2573]: W1030 20:37:15.462682 2573 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 20:37:15.465829 ip-10-0-138-149 kubenswrapper[2573]: W1030 20:37:15.462687 2573 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 20:37:15.465829 ip-10-0-138-149 kubenswrapper[2573]: W1030 20:37:15.462691 2573 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 20:37:15.465829 ip-10-0-138-149 kubenswrapper[2573]: W1030 20:37:15.462694 2573 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 20:37:15.465829 ip-10-0-138-149 kubenswrapper[2573]: W1030 20:37:15.462697 2573 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 20:37:15.465829 ip-10-0-138-149 kubenswrapper[2573]: W1030 20:37:15.462700 2573 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 20:37:15.465829 ip-10-0-138-149 kubenswrapper[2573]: W1030 20:37:15.462703 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 20:37:15.465829 ip-10-0-138-149 kubenswrapper[2573]: W1030 20:37:15.462706 2573 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 20:37:15.465829 ip-10-0-138-149 kubenswrapper[2573]: W1030 20:37:15.462708 2573 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 20:37:15.465829 ip-10-0-138-149 kubenswrapper[2573]: W1030 20:37:15.462711 2573 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 20:37:15.465829 ip-10-0-138-149 kubenswrapper[2573]: W1030 20:37:15.462714 2573 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 20:37:15.465829 ip-10-0-138-149 kubenswrapper[2573]: W1030 20:37:15.462716 2573 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 20:37:15.465829 ip-10-0-138-149 kubenswrapper[2573]: W1030 20:37:15.462719 2573 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 20:37:15.465829 ip-10-0-138-149 kubenswrapper[2573]: W1030 20:37:15.462721 2573 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 20:37:15.465829 ip-10-0-138-149 kubenswrapper[2573]: W1030 20:37:15.462724 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 20:37:15.465829 ip-10-0-138-149 kubenswrapper[2573]: W1030 20:37:15.462727 2573 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 20:37:15.465829 ip-10-0-138-149 kubenswrapper[2573]: W1030 20:37:15.462729 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 20:37:15.465829 ip-10-0-138-149 kubenswrapper[2573]: W1030 20:37:15.462732 2573 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 20:37:15.465829 ip-10-0-138-149 kubenswrapper[2573]: W1030 20:37:15.462734 2573 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 20:37:15.465829 ip-10-0-138-149 kubenswrapper[2573]: W1030 20:37:15.462738 2573 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 20:37:15.466274 ip-10-0-138-149 kubenswrapper[2573]: W1030 20:37:15.462742 2573 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 20:37:15.466274 ip-10-0-138-149 kubenswrapper[2573]: W1030 20:37:15.462746 2573 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 20:37:15.466274 ip-10-0-138-149 kubenswrapper[2573]: W1030 20:37:15.462749 2573 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 20:37:15.466274 ip-10-0-138-149 kubenswrapper[2573]: W1030 20:37:15.462752 2573 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 20:37:15.466274 ip-10-0-138-149 kubenswrapper[2573]: W1030 20:37:15.462754 2573 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 20:37:15.466274 ip-10-0-138-149 kubenswrapper[2573]: W1030 20:37:15.462758 2573 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 20:37:15.466274 ip-10-0-138-149 kubenswrapper[2573]: W1030 20:37:15.462761 2573 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 20:37:15.466274 ip-10-0-138-149 kubenswrapper[2573]: W1030 20:37:15.462763 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 20:37:15.466274 ip-10-0-138-149 kubenswrapper[2573]: W1030 20:37:15.462766 2573 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 20:37:15.466274 ip-10-0-138-149 kubenswrapper[2573]: W1030 20:37:15.462768 2573 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 20:37:15.466274 ip-10-0-138-149 kubenswrapper[2573]: W1030 20:37:15.462771 2573 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 20:37:15.466274 ip-10-0-138-149 kubenswrapper[2573]: W1030 20:37:15.462774 2573 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 20:37:15.466274 ip-10-0-138-149 kubenswrapper[2573]: W1030 20:37:15.462777 2573 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 20:37:15.466274 ip-10-0-138-149 kubenswrapper[2573]: W1030 20:37:15.462780 2573 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 20:37:15.466274 ip-10-0-138-149 kubenswrapper[2573]: W1030 20:37:15.462782 2573 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 20:37:15.466274 ip-10-0-138-149 kubenswrapper[2573]: W1030 20:37:15.463263 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 20:37:15.466274 ip-10-0-138-149 kubenswrapper[2573]: W1030 20:37:15.463268 2573 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 20:37:15.466274 ip-10-0-138-149 kubenswrapper[2573]: W1030 20:37:15.463271 2573 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 20:37:15.466274 ip-10-0-138-149 kubenswrapper[2573]: W1030 20:37:15.463274 2573 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 20:37:15.466730 ip-10-0-138-149 kubenswrapper[2573]: W1030 20:37:15.463277 2573 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 20:37:15.466730 ip-10-0-138-149 kubenswrapper[2573]: W1030 20:37:15.463280 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 20:37:15.466730 ip-10-0-138-149 kubenswrapper[2573]: W1030 20:37:15.463283 2573 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 20:37:15.466730 ip-10-0-138-149 kubenswrapper[2573]: W1030 20:37:15.463285 2573 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 20:37:15.466730 ip-10-0-138-149 kubenswrapper[2573]: W1030 20:37:15.463288 2573 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 20:37:15.466730 ip-10-0-138-149 kubenswrapper[2573]: W1030 20:37:15.463290 2573 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 20:37:15.466730 ip-10-0-138-149 kubenswrapper[2573]: W1030 20:37:15.463293 2573 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 20:37:15.466730 ip-10-0-138-149 kubenswrapper[2573]: W1030 20:37:15.463295 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 20:37:15.466730 ip-10-0-138-149 kubenswrapper[2573]: W1030 20:37:15.463298 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 20:37:15.466730 ip-10-0-138-149 kubenswrapper[2573]: W1030 20:37:15.463301 2573 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 20:37:15.466730 ip-10-0-138-149 kubenswrapper[2573]: W1030 20:37:15.463303 2573 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 20:37:15.466730 ip-10-0-138-149 kubenswrapper[2573]: W1030 20:37:15.463306 2573 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 20:37:15.466730 ip-10-0-138-149 kubenswrapper[2573]: W1030 20:37:15.463309 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 20:37:15.466730 ip-10-0-138-149 kubenswrapper[2573]: W1030 20:37:15.463312 2573 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 20:37:15.466730 ip-10-0-138-149 kubenswrapper[2573]: W1030 20:37:15.463314 2573 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 20:37:15.466730 ip-10-0-138-149 kubenswrapper[2573]: W1030 20:37:15.463318 2573 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 20:37:15.466730 ip-10-0-138-149 kubenswrapper[2573]: W1030 20:37:15.463320 2573 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 20:37:15.466730 ip-10-0-138-149 kubenswrapper[2573]: W1030 20:37:15.463323 2573 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 20:37:15.466730 ip-10-0-138-149 kubenswrapper[2573]: W1030 20:37:15.463326 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 20:37:15.467174 ip-10-0-138-149 kubenswrapper[2573]: W1030 20:37:15.463329 2573 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 20:37:15.467174 ip-10-0-138-149 kubenswrapper[2573]: W1030 20:37:15.463332 2573 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 20:37:15.467174 ip-10-0-138-149 kubenswrapper[2573]: W1030 20:37:15.463334 2573 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 20:37:15.467174 ip-10-0-138-149 kubenswrapper[2573]: W1030 20:37:15.463337 2573 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 20:37:15.467174 ip-10-0-138-149 kubenswrapper[2573]: W1030 20:37:15.463340 2573 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 20:37:15.467174 ip-10-0-138-149 kubenswrapper[2573]: W1030 20:37:15.463343 2573 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 20:37:15.467174 ip-10-0-138-149 kubenswrapper[2573]: W1030 20:37:15.463345 2573 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 20:37:15.467174 ip-10-0-138-149 kubenswrapper[2573]: W1030 20:37:15.463348 2573 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 20:37:15.467174 ip-10-0-138-149 kubenswrapper[2573]: W1030 20:37:15.463351 2573 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 20:37:15.467174 ip-10-0-138-149 kubenswrapper[2573]: W1030 20:37:15.463353 2573 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 20:37:15.467174 ip-10-0-138-149 kubenswrapper[2573]: W1030 20:37:15.463356 2573 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 20:37:15.467174 ip-10-0-138-149 kubenswrapper[2573]: W1030 20:37:15.463359 2573 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 20:37:15.467174 ip-10-0-138-149 kubenswrapper[2573]: W1030 20:37:15.463361 2573 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 20:37:15.467174 ip-10-0-138-149 kubenswrapper[2573]: W1030 20:37:15.463364 2573 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 20:37:15.467174 ip-10-0-138-149 kubenswrapper[2573]: W1030 20:37:15.463367 2573 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 20:37:15.467174 ip-10-0-138-149 kubenswrapper[2573]: W1030 20:37:15.463369 2573 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 20:37:15.467174 ip-10-0-138-149 kubenswrapper[2573]: W1030 20:37:15.463372 2573 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 20:37:15.467174 ip-10-0-138-149 kubenswrapper[2573]: W1030 20:37:15.463374 2573 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 20:37:15.467174 ip-10-0-138-149 kubenswrapper[2573]: W1030 20:37:15.463377 2573 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 20:37:15.467626 ip-10-0-138-149 kubenswrapper[2573]: W1030 20:37:15.463379 2573 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 20:37:15.467626 ip-10-0-138-149 kubenswrapper[2573]: W1030 20:37:15.463382 2573 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 20:37:15.467626 ip-10-0-138-149 kubenswrapper[2573]: W1030 20:37:15.463384 2573 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 20:37:15.467626 ip-10-0-138-149 kubenswrapper[2573]: W1030 20:37:15.463387 2573 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 20:37:15.467626 ip-10-0-138-149 kubenswrapper[2573]: W1030 20:37:15.463389 2573 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 20:37:15.467626 ip-10-0-138-149 kubenswrapper[2573]: W1030 20:37:15.463392 2573 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 20:37:15.467626 ip-10-0-138-149 kubenswrapper[2573]: W1030 20:37:15.463394 2573 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 20:37:15.467626 ip-10-0-138-149 kubenswrapper[2573]: W1030 20:37:15.463397 2573 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 20:37:15.467626 ip-10-0-138-149 kubenswrapper[2573]: W1030 20:37:15.463400 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 20:37:15.467626 ip-10-0-138-149 kubenswrapper[2573]: W1030 20:37:15.463402 2573 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 20:37:15.467626 ip-10-0-138-149 kubenswrapper[2573]: W1030 20:37:15.463405 2573 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 20:37:15.467626 ip-10-0-138-149 kubenswrapper[2573]: W1030 20:37:15.463408 2573 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 20:37:15.467626 ip-10-0-138-149 kubenswrapper[2573]: W1030 20:37:15.463410 2573 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 20:37:15.467626 ip-10-0-138-149 kubenswrapper[2573]: W1030 20:37:15.463413 2573 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 20:37:15.467626 ip-10-0-138-149 kubenswrapper[2573]: W1030 20:37:15.463418 2573 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 20:37:15.467626 ip-10-0-138-149 kubenswrapper[2573]: W1030 20:37:15.463421 2573 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 20:37:15.467626 ip-10-0-138-149 kubenswrapper[2573]: W1030 20:37:15.463424 2573 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 20:37:15.467626 ip-10-0-138-149 kubenswrapper[2573]: W1030 20:37:15.463426 2573 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 20:37:15.467626 ip-10-0-138-149 kubenswrapper[2573]: W1030 20:37:15.463429 2573 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 20:37:15.467626 ip-10-0-138-149 kubenswrapper[2573]: W1030 20:37:15.463431 2573 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 20:37:15.468090 ip-10-0-138-149 kubenswrapper[2573]: W1030 20:37:15.463434 2573 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 20:37:15.468090 ip-10-0-138-149 kubenswrapper[2573]: W1030 20:37:15.463436 2573 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 20:37:15.468090 ip-10-0-138-149 kubenswrapper[2573]: W1030 20:37:15.463439 2573 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 20:37:15.468090 ip-10-0-138-149 kubenswrapper[2573]: W1030 20:37:15.463442 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 20:37:15.468090 ip-10-0-138-149 kubenswrapper[2573]: W1030 20:37:15.463446 2573 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 20:37:15.468090 ip-10-0-138-149 kubenswrapper[2573]: W1030 20:37:15.463449 2573 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 20:37:15.468090 ip-10-0-138-149 kubenswrapper[2573]: W1030 20:37:15.463452 2573 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 20:37:15.468090 ip-10-0-138-149 kubenswrapper[2573]: W1030 20:37:15.463454 2573 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 20:37:15.468090 ip-10-0-138-149 kubenswrapper[2573]: W1030 20:37:15.463459 2573 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 20:37:15.468090 ip-10-0-138-149 kubenswrapper[2573]: W1030 20:37:15.463463 2573 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 20:37:15.468090 ip-10-0-138-149 kubenswrapper[2573]: W1030 20:37:15.463466 2573 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 20:37:15.468090 ip-10-0-138-149 kubenswrapper[2573]: W1030 20:37:15.463469 2573 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 20:37:15.468090 ip-10-0-138-149 kubenswrapper[2573]: W1030 20:37:15.463473 2573 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 20:37:15.468090 ip-10-0-138-149 kubenswrapper[2573]: W1030 20:37:15.463477 2573 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 20:37:15.468090 ip-10-0-138-149 kubenswrapper[2573]: W1030 20:37:15.463479 2573 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 20:37:15.468090 ip-10-0-138-149 kubenswrapper[2573]: W1030 20:37:15.463482 2573 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 20:37:15.468090 ip-10-0-138-149 kubenswrapper[2573]: W1030 20:37:15.463485 2573 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 20:37:15.468090 ip-10-0-138-149 kubenswrapper[2573]: W1030 20:37:15.463488 2573 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 20:37:15.468090 ip-10-0-138-149 kubenswrapper[2573]: W1030 20:37:15.463491 2573 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 20:37:15.468090 ip-10-0-138-149 kubenswrapper[2573]: W1030 20:37:15.463493 2573 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 20:37:15.468590 ip-10-0-138-149 kubenswrapper[2573]: W1030 20:37:15.463496 2573 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 20:37:15.468590 ip-10-0-138-149 kubenswrapper[2573]: W1030 20:37:15.463500 2573 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 20:37:15.468590 ip-10-0-138-149 kubenswrapper[2573]: W1030 20:37:15.463503 2573 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 20:37:15.468590 ip-10-0-138-149 kubenswrapper[2573]: W1030 20:37:15.463505 2573 feature_gate.go:386] unrecognized feature gate: Example Oct 30 20:37:15.468590 ip-10-0-138-149 kubenswrapper[2573]: W1030 20:37:15.463508 2573 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 20:37:15.468590 ip-10-0-138-149 kubenswrapper[2573]: W1030 20:37:15.463511 2573 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 20:37:15.468590 ip-10-0-138-149 kubenswrapper[2573]: W1030 20:37:15.463515 2573 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 20:37:15.468590 ip-10-0-138-149 kubenswrapper[2573]: W1030 20:37:15.463517 2573 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 20:37:15.468590 ip-10-0-138-149 kubenswrapper[2573]: W1030 20:37:15.463520 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 20:37:15.468590 ip-10-0-138-149 kubenswrapper[2573]: W1030 20:37:15.463523 2573 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 20:37:15.468590 ip-10-0-138-149 kubenswrapper[2573]: W1030 20:37:15.463525 2573 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 20:37:15.468590 ip-10-0-138-149 kubenswrapper[2573]: W1030 20:37:15.463528 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 20:37:15.468590 ip-10-0-138-149 kubenswrapper[2573]: W1030 20:37:15.463531 2573 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 20:37:15.468590 ip-10-0-138-149 kubenswrapper[2573]: W1030 20:37:15.463534 2573 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 20:37:15.468590 ip-10-0-138-149 kubenswrapper[2573]: W1030 20:37:15.463536 2573 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 20:37:15.468590 ip-10-0-138-149 kubenswrapper[2573]: W1030 20:37:15.463539 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 20:37:15.468590 ip-10-0-138-149 kubenswrapper[2573]: W1030 20:37:15.463543 2573 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 20:37:15.468590 ip-10-0-138-149 kubenswrapper[2573]: W1030 20:37:15.463545 2573 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 20:37:15.468590 ip-10-0-138-149 kubenswrapper[2573]: W1030 20:37:15.463548 2573 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 20:37:15.469031 ip-10-0-138-149 kubenswrapper[2573]: W1030 20:37:15.463550 2573 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 20:37:15.469031 ip-10-0-138-149 kubenswrapper[2573]: W1030 20:37:15.463553 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 20:37:15.469031 ip-10-0-138-149 kubenswrapper[2573]: W1030 20:37:15.463558 2573 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 20:37:15.469031 ip-10-0-138-149 kubenswrapper[2573]: W1030 20:37:15.463574 2573 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 20:37:15.469031 ip-10-0-138-149 kubenswrapper[2573]: W1030 20:37:15.463578 2573 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 20:37:15.469031 ip-10-0-138-149 kubenswrapper[2573]: I1030 20:37:15.465143 2573 flags.go:64] FLAG: --address="0.0.0.0" Oct 30 20:37:15.469031 ip-10-0-138-149 kubenswrapper[2573]: I1030 20:37:15.465152 2573 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Oct 30 20:37:15.469031 ip-10-0-138-149 kubenswrapper[2573]: I1030 20:37:15.465159 2573 flags.go:64] FLAG: --anonymous-auth="true" Oct 30 20:37:15.469031 ip-10-0-138-149 kubenswrapper[2573]: I1030 20:37:15.465164 2573 flags.go:64] FLAG: --application-metrics-count-limit="100" Oct 30 20:37:15.469031 ip-10-0-138-149 kubenswrapper[2573]: I1030 20:37:15.465170 2573 flags.go:64] FLAG: --authentication-token-webhook="false" Oct 30 20:37:15.469031 ip-10-0-138-149 kubenswrapper[2573]: I1030 20:37:15.465173 2573 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Oct 30 20:37:15.469031 ip-10-0-138-149 kubenswrapper[2573]: I1030 20:37:15.465177 2573 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Oct 30 20:37:15.469031 ip-10-0-138-149 kubenswrapper[2573]: I1030 20:37:15.465182 2573 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Oct 30 20:37:15.469031 ip-10-0-138-149 kubenswrapper[2573]: I1030 20:37:15.465185 2573 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Oct 30 20:37:15.469031 ip-10-0-138-149 kubenswrapper[2573]: I1030 20:37:15.465189 2573 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Oct 30 20:37:15.469031 ip-10-0-138-149 kubenswrapper[2573]: I1030 20:37:15.465193 2573 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Oct 30 20:37:15.469031 ip-10-0-138-149 kubenswrapper[2573]: I1030 20:37:15.465197 2573 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Oct 30 20:37:15.469031 ip-10-0-138-149 kubenswrapper[2573]: I1030 20:37:15.465200 2573 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Oct 30 20:37:15.469031 ip-10-0-138-149 kubenswrapper[2573]: I1030 20:37:15.465203 2573 flags.go:64] FLAG: --cgroup-root="" Oct 30 20:37:15.469031 ip-10-0-138-149 kubenswrapper[2573]: I1030 20:37:15.465207 2573 flags.go:64] FLAG: --cgroups-per-qos="true" Oct 30 20:37:15.469031 ip-10-0-138-149 kubenswrapper[2573]: I1030 20:37:15.465210 2573 flags.go:64] FLAG: --client-ca-file="" Oct 30 20:37:15.469031 ip-10-0-138-149 kubenswrapper[2573]: I1030 20:37:15.465214 2573 flags.go:64] FLAG: --cloud-provider="external" Oct 30 20:37:15.469031 ip-10-0-138-149 kubenswrapper[2573]: I1030 20:37:15.465217 2573 flags.go:64] FLAG: --cluster-dns="[]" Oct 30 20:37:15.469580 ip-10-0-138-149 kubenswrapper[2573]: I1030 20:37:15.465221 2573 flags.go:64] FLAG: --cluster-domain="" Oct 30 20:37:15.469580 ip-10-0-138-149 kubenswrapper[2573]: I1030 20:37:15.465224 2573 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Oct 30 20:37:15.469580 ip-10-0-138-149 kubenswrapper[2573]: I1030 20:37:15.465227 2573 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Oct 30 20:37:15.469580 ip-10-0-138-149 kubenswrapper[2573]: I1030 20:37:15.465230 2573 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Oct 30 20:37:15.469580 ip-10-0-138-149 kubenswrapper[2573]: I1030 20:37:15.465234 2573 flags.go:64] FLAG: --container-log-max-files="5" Oct 30 20:37:15.469580 ip-10-0-138-149 kubenswrapper[2573]: I1030 20:37:15.465238 2573 flags.go:64] FLAG: --container-log-max-size="10Mi" Oct 30 20:37:15.469580 ip-10-0-138-149 kubenswrapper[2573]: I1030 20:37:15.465241 2573 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Oct 30 20:37:15.469580 ip-10-0-138-149 kubenswrapper[2573]: I1030 20:37:15.465245 2573 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Oct 30 20:37:15.469580 ip-10-0-138-149 kubenswrapper[2573]: I1030 20:37:15.465248 2573 flags.go:64] FLAG: --containerd-namespace="k8s.io" Oct 30 20:37:15.469580 ip-10-0-138-149 kubenswrapper[2573]: I1030 20:37:15.465251 2573 flags.go:64] FLAG: --contention-profiling="false" Oct 30 20:37:15.469580 ip-10-0-138-149 kubenswrapper[2573]: I1030 20:37:15.465254 2573 flags.go:64] FLAG: --cpu-cfs-quota="true" Oct 30 20:37:15.469580 ip-10-0-138-149 kubenswrapper[2573]: I1030 20:37:15.465257 2573 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Oct 30 20:37:15.469580 ip-10-0-138-149 kubenswrapper[2573]: I1030 20:37:15.465260 2573 flags.go:64] FLAG: --cpu-manager-policy="none" Oct 30 20:37:15.469580 ip-10-0-138-149 kubenswrapper[2573]: I1030 20:37:15.465263 2573 flags.go:64] FLAG: --cpu-manager-policy-options="" Oct 30 20:37:15.469580 ip-10-0-138-149 kubenswrapper[2573]: I1030 20:37:15.465267 2573 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Oct 30 20:37:15.469580 ip-10-0-138-149 kubenswrapper[2573]: I1030 20:37:15.465270 2573 flags.go:64] FLAG: --enable-controller-attach-detach="true" Oct 30 20:37:15.469580 ip-10-0-138-149 kubenswrapper[2573]: I1030 20:37:15.465273 2573 flags.go:64] FLAG: --enable-debugging-handlers="true" Oct 30 20:37:15.469580 ip-10-0-138-149 kubenswrapper[2573]: I1030 20:37:15.465276 2573 flags.go:64] FLAG: --enable-load-reader="false" Oct 30 20:37:15.469580 ip-10-0-138-149 kubenswrapper[2573]: I1030 20:37:15.465279 2573 flags.go:64] FLAG: --enable-server="true" Oct 30 20:37:15.469580 ip-10-0-138-149 kubenswrapper[2573]: I1030 20:37:15.465282 2573 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Oct 30 20:37:15.469580 ip-10-0-138-149 kubenswrapper[2573]: I1030 20:37:15.465287 2573 flags.go:64] FLAG: --event-burst="100" Oct 30 20:37:15.469580 ip-10-0-138-149 kubenswrapper[2573]: I1030 20:37:15.465290 2573 flags.go:64] FLAG: --event-qps="50" Oct 30 20:37:15.469580 ip-10-0-138-149 kubenswrapper[2573]: I1030 20:37:15.465293 2573 flags.go:64] FLAG: --event-storage-age-limit="default=0" Oct 30 20:37:15.469580 ip-10-0-138-149 kubenswrapper[2573]: I1030 20:37:15.465297 2573 flags.go:64] FLAG: --event-storage-event-limit="default=0" Oct 30 20:37:15.470172 ip-10-0-138-149 kubenswrapper[2573]: I1030 20:37:15.465300 2573 flags.go:64] FLAG: --eviction-hard="" Oct 30 20:37:15.470172 ip-10-0-138-149 kubenswrapper[2573]: I1030 20:37:15.465305 2573 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Oct 30 20:37:15.470172 ip-10-0-138-149 kubenswrapper[2573]: I1030 20:37:15.465308 2573 flags.go:64] FLAG: --eviction-minimum-reclaim="" Oct 30 20:37:15.470172 ip-10-0-138-149 kubenswrapper[2573]: I1030 20:37:15.465311 2573 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Oct 30 20:37:15.470172 ip-10-0-138-149 kubenswrapper[2573]: I1030 20:37:15.465316 2573 flags.go:64] FLAG: --eviction-soft="" Oct 30 20:37:15.470172 ip-10-0-138-149 kubenswrapper[2573]: I1030 20:37:15.465319 2573 flags.go:64] FLAG: --eviction-soft-grace-period="" Oct 30 20:37:15.470172 ip-10-0-138-149 kubenswrapper[2573]: I1030 20:37:15.465322 2573 flags.go:64] FLAG: --exit-on-lock-contention="false" Oct 30 20:37:15.470172 ip-10-0-138-149 kubenswrapper[2573]: I1030 20:37:15.465325 2573 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Oct 30 20:37:15.470172 ip-10-0-138-149 kubenswrapper[2573]: I1030 20:37:15.465328 2573 flags.go:64] FLAG: --experimental-mounter-path="" Oct 30 20:37:15.470172 ip-10-0-138-149 kubenswrapper[2573]: I1030 20:37:15.465331 2573 flags.go:64] FLAG: --fail-cgroupv1="true" Oct 30 20:37:15.470172 ip-10-0-138-149 kubenswrapper[2573]: I1030 20:37:15.465334 2573 flags.go:64] FLAG: --fail-swap-on="true" Oct 30 20:37:15.470172 ip-10-0-138-149 kubenswrapper[2573]: I1030 20:37:15.465337 2573 flags.go:64] FLAG: --feature-gates="" Oct 30 20:37:15.470172 ip-10-0-138-149 kubenswrapper[2573]: I1030 20:37:15.465341 2573 flags.go:64] FLAG: --file-check-frequency="20s" Oct 30 20:37:15.470172 ip-10-0-138-149 kubenswrapper[2573]: I1030 20:37:15.465344 2573 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Oct 30 20:37:15.470172 ip-10-0-138-149 kubenswrapper[2573]: I1030 20:37:15.465348 2573 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Oct 30 20:37:15.470172 ip-10-0-138-149 kubenswrapper[2573]: I1030 20:37:15.465351 2573 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Oct 30 20:37:15.470172 ip-10-0-138-149 kubenswrapper[2573]: I1030 20:37:15.465354 2573 flags.go:64] FLAG: --healthz-port="10248" Oct 30 20:37:15.470172 ip-10-0-138-149 kubenswrapper[2573]: I1030 20:37:15.465357 2573 flags.go:64] FLAG: --help="false" Oct 30 20:37:15.470172 ip-10-0-138-149 kubenswrapper[2573]: I1030 20:37:15.465360 2573 flags.go:64] FLAG: --hostname-override="ip-10-0-138-149.us-west-1.compute.internal" Oct 30 20:37:15.470172 ip-10-0-138-149 kubenswrapper[2573]: I1030 20:37:15.465364 2573 flags.go:64] FLAG: --housekeeping-interval="10s" Oct 30 20:37:15.470172 ip-10-0-138-149 kubenswrapper[2573]: I1030 20:37:15.465367 2573 flags.go:64] FLAG: --http-check-frequency="20s" Oct 30 20:37:15.470172 ip-10-0-138-149 kubenswrapper[2573]: I1030 20:37:15.465370 2573 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Oct 30 20:37:15.470172 ip-10-0-138-149 kubenswrapper[2573]: I1030 20:37:15.465374 2573 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Oct 30 20:37:15.470757 ip-10-0-138-149 kubenswrapper[2573]: I1030 20:37:15.465377 2573 flags.go:64] FLAG: --image-gc-high-threshold="85" Oct 30 20:37:15.470757 ip-10-0-138-149 kubenswrapper[2573]: I1030 20:37:15.465380 2573 flags.go:64] FLAG: --image-gc-low-threshold="80" Oct 30 20:37:15.470757 ip-10-0-138-149 kubenswrapper[2573]: I1030 20:37:15.465383 2573 flags.go:64] FLAG: --image-service-endpoint="" Oct 30 20:37:15.470757 ip-10-0-138-149 kubenswrapper[2573]: I1030 20:37:15.465386 2573 flags.go:64] FLAG: --kernel-memcg-notification="false" Oct 30 20:37:15.470757 ip-10-0-138-149 kubenswrapper[2573]: I1030 20:37:15.465389 2573 flags.go:64] FLAG: --kube-api-burst="100" Oct 30 20:37:15.470757 ip-10-0-138-149 kubenswrapper[2573]: I1030 20:37:15.465392 2573 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Oct 30 20:37:15.470757 ip-10-0-138-149 kubenswrapper[2573]: I1030 20:37:15.465395 2573 flags.go:64] FLAG: --kube-api-qps="50" Oct 30 20:37:15.470757 ip-10-0-138-149 kubenswrapper[2573]: I1030 20:37:15.465398 2573 flags.go:64] FLAG: --kube-reserved="" Oct 30 20:37:15.470757 ip-10-0-138-149 kubenswrapper[2573]: I1030 20:37:15.465401 2573 flags.go:64] FLAG: --kube-reserved-cgroup="" Oct 30 20:37:15.470757 ip-10-0-138-149 kubenswrapper[2573]: I1030 20:37:15.465404 2573 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Oct 30 20:37:15.470757 ip-10-0-138-149 kubenswrapper[2573]: I1030 20:37:15.465407 2573 flags.go:64] FLAG: --kubelet-cgroups="" Oct 30 20:37:15.470757 ip-10-0-138-149 kubenswrapper[2573]: I1030 20:37:15.465410 2573 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Oct 30 20:37:15.470757 ip-10-0-138-149 kubenswrapper[2573]: I1030 20:37:15.465413 2573 flags.go:64] FLAG: --lock-file="" Oct 30 20:37:15.470757 ip-10-0-138-149 kubenswrapper[2573]: I1030 20:37:15.465417 2573 flags.go:64] FLAG: --log-cadvisor-usage="false" Oct 30 20:37:15.470757 ip-10-0-138-149 kubenswrapper[2573]: I1030 20:37:15.465420 2573 flags.go:64] FLAG: --log-flush-frequency="5s" Oct 30 20:37:15.470757 ip-10-0-138-149 kubenswrapper[2573]: I1030 20:37:15.465423 2573 flags.go:64] FLAG: --log-json-info-buffer-size="0" Oct 30 20:37:15.470757 ip-10-0-138-149 kubenswrapper[2573]: I1030 20:37:15.465428 2573 flags.go:64] FLAG: --log-json-split-stream="false" Oct 30 20:37:15.470757 ip-10-0-138-149 kubenswrapper[2573]: I1030 20:37:15.465431 2573 flags.go:64] FLAG: --log-text-info-buffer-size="0" Oct 30 20:37:15.470757 ip-10-0-138-149 kubenswrapper[2573]: I1030 20:37:15.465434 2573 flags.go:64] FLAG: --log-text-split-stream="false" Oct 30 20:37:15.470757 ip-10-0-138-149 kubenswrapper[2573]: I1030 20:37:15.465437 2573 flags.go:64] FLAG: --logging-format="text" Oct 30 20:37:15.470757 ip-10-0-138-149 kubenswrapper[2573]: I1030 20:37:15.465440 2573 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Oct 30 20:37:15.470757 ip-10-0-138-149 kubenswrapper[2573]: I1030 20:37:15.465443 2573 flags.go:64] FLAG: --make-iptables-util-chains="true" Oct 30 20:37:15.470757 ip-10-0-138-149 kubenswrapper[2573]: I1030 20:37:15.465446 2573 flags.go:64] FLAG: --manifest-url="" Oct 30 20:37:15.470757 ip-10-0-138-149 kubenswrapper[2573]: I1030 20:37:15.465449 2573 flags.go:64] FLAG: --manifest-url-header="" Oct 30 20:37:15.470757 ip-10-0-138-149 kubenswrapper[2573]: I1030 20:37:15.465454 2573 flags.go:64] FLAG: --max-housekeeping-interval="15s" Oct 30 20:37:15.471401 ip-10-0-138-149 kubenswrapper[2573]: I1030 20:37:15.465457 2573 flags.go:64] FLAG: --max-open-files="1000000" Oct 30 20:37:15.471401 ip-10-0-138-149 kubenswrapper[2573]: I1030 20:37:15.465460 2573 flags.go:64] FLAG: --max-pods="110" Oct 30 20:37:15.471401 ip-10-0-138-149 kubenswrapper[2573]: I1030 20:37:15.465463 2573 flags.go:64] FLAG: --maximum-dead-containers="-1" Oct 30 20:37:15.471401 ip-10-0-138-149 kubenswrapper[2573]: I1030 20:37:15.465466 2573 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Oct 30 20:37:15.471401 ip-10-0-138-149 kubenswrapper[2573]: I1030 20:37:15.465469 2573 flags.go:64] FLAG: --memory-manager-policy="None" Oct 30 20:37:15.471401 ip-10-0-138-149 kubenswrapper[2573]: I1030 20:37:15.465472 2573 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Oct 30 20:37:15.471401 ip-10-0-138-149 kubenswrapper[2573]: I1030 20:37:15.465475 2573 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Oct 30 20:37:15.471401 ip-10-0-138-149 kubenswrapper[2573]: I1030 20:37:15.465478 2573 flags.go:64] FLAG: --node-ip="0.0.0.0" Oct 30 20:37:15.471401 ip-10-0-138-149 kubenswrapper[2573]: I1030 20:37:15.465481 2573 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Oct 30 20:37:15.471401 ip-10-0-138-149 kubenswrapper[2573]: I1030 20:37:15.465488 2573 flags.go:64] FLAG: --node-status-max-images="50" Oct 30 20:37:15.471401 ip-10-0-138-149 kubenswrapper[2573]: I1030 20:37:15.465491 2573 flags.go:64] FLAG: --node-status-update-frequency="10s" Oct 30 20:37:15.471401 ip-10-0-138-149 kubenswrapper[2573]: I1030 20:37:15.465494 2573 flags.go:64] FLAG: --oom-score-adj="-999" Oct 30 20:37:15.471401 ip-10-0-138-149 kubenswrapper[2573]: I1030 20:37:15.465497 2573 flags.go:64] FLAG: --pod-cidr="" Oct 30 20:37:15.471401 ip-10-0-138-149 kubenswrapper[2573]: I1030 20:37:15.465500 2573 flags.go:64] FLAG: --pod-manifest-path="" Oct 30 20:37:15.471401 ip-10-0-138-149 kubenswrapper[2573]: I1030 20:37:15.465502 2573 flags.go:64] FLAG: --pod-max-pids="-1" Oct 30 20:37:15.471401 ip-10-0-138-149 kubenswrapper[2573]: I1030 20:37:15.465505 2573 flags.go:64] FLAG: --pods-per-core="0" Oct 30 20:37:15.471401 ip-10-0-138-149 kubenswrapper[2573]: I1030 20:37:15.465509 2573 flags.go:64] FLAG: --port="10250" Oct 30 20:37:15.471401 ip-10-0-138-149 kubenswrapper[2573]: I1030 20:37:15.465512 2573 flags.go:64] FLAG: --protect-kernel-defaults="false" Oct 30 20:37:15.471401 ip-10-0-138-149 kubenswrapper[2573]: I1030 20:37:15.465514 2573 flags.go:64] FLAG: --provider-id="aws:///us-west-1a/i-0668a941f40a9593e" Oct 30 20:37:15.471401 ip-10-0-138-149 kubenswrapper[2573]: I1030 20:37:15.465518 2573 flags.go:64] FLAG: --qos-reserved="" Oct 30 20:37:15.471401 ip-10-0-138-149 kubenswrapper[2573]: I1030 20:37:15.465521 2573 flags.go:64] FLAG: --read-only-port="10255" Oct 30 20:37:15.471401 ip-10-0-138-149 kubenswrapper[2573]: I1030 20:37:15.465524 2573 flags.go:64] FLAG: --register-node="true" Oct 30 20:37:15.471401 ip-10-0-138-149 kubenswrapper[2573]: I1030 20:37:15.465527 2573 flags.go:64] FLAG: --register-with-taints="" Oct 30 20:37:15.471401 ip-10-0-138-149 kubenswrapper[2573]: I1030 20:37:15.465531 2573 flags.go:64] FLAG: --registry-burst="10" Oct 30 20:37:15.471401 ip-10-0-138-149 kubenswrapper[2573]: I1030 20:37:15.465534 2573 flags.go:64] FLAG: --registry-qps="5" Oct 30 20:37:15.471401 ip-10-0-138-149 kubenswrapper[2573]: I1030 20:37:15.465537 2573 flags.go:64] FLAG: --reserved-cpus="" Oct 30 20:37:15.472093 ip-10-0-138-149 kubenswrapper[2573]: I1030 20:37:15.465540 2573 flags.go:64] FLAG: --reserved-memory="" Oct 30 20:37:15.472093 ip-10-0-138-149 kubenswrapper[2573]: I1030 20:37:15.465544 2573 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Oct 30 20:37:15.472093 ip-10-0-138-149 kubenswrapper[2573]: I1030 20:37:15.465547 2573 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Oct 30 20:37:15.472093 ip-10-0-138-149 kubenswrapper[2573]: I1030 20:37:15.465550 2573 flags.go:64] FLAG: --rotate-certificates="false" Oct 30 20:37:15.472093 ip-10-0-138-149 kubenswrapper[2573]: I1030 20:37:15.465553 2573 flags.go:64] FLAG: --rotate-server-certificates="false" Oct 30 20:37:15.472093 ip-10-0-138-149 kubenswrapper[2573]: I1030 20:37:15.465556 2573 flags.go:64] FLAG: --runonce="false" Oct 30 20:37:15.472093 ip-10-0-138-149 kubenswrapper[2573]: I1030 20:37:15.465559 2573 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Oct 30 20:37:15.472093 ip-10-0-138-149 kubenswrapper[2573]: I1030 20:37:15.465574 2573 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Oct 30 20:37:15.472093 ip-10-0-138-149 kubenswrapper[2573]: I1030 20:37:15.465577 2573 flags.go:64] FLAG: --seccomp-default="false" Oct 30 20:37:15.472093 ip-10-0-138-149 kubenswrapper[2573]: I1030 20:37:15.465581 2573 flags.go:64] FLAG: --serialize-image-pulls="true" Oct 30 20:37:15.472093 ip-10-0-138-149 kubenswrapper[2573]: I1030 20:37:15.465584 2573 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Oct 30 20:37:15.472093 ip-10-0-138-149 kubenswrapper[2573]: I1030 20:37:15.465587 2573 flags.go:64] FLAG: --storage-driver-db="cadvisor" Oct 30 20:37:15.472093 ip-10-0-138-149 kubenswrapper[2573]: I1030 20:37:15.465590 2573 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Oct 30 20:37:15.472093 ip-10-0-138-149 kubenswrapper[2573]: I1030 20:37:15.465593 2573 flags.go:64] FLAG: --storage-driver-password="root" Oct 30 20:37:15.472093 ip-10-0-138-149 kubenswrapper[2573]: I1030 20:37:15.465596 2573 flags.go:64] FLAG: --storage-driver-secure="false" Oct 30 20:37:15.472093 ip-10-0-138-149 kubenswrapper[2573]: I1030 20:37:15.465599 2573 flags.go:64] FLAG: --storage-driver-table="stats" Oct 30 20:37:15.472093 ip-10-0-138-149 kubenswrapper[2573]: I1030 20:37:15.465602 2573 flags.go:64] FLAG: --storage-driver-user="root" Oct 30 20:37:15.472093 ip-10-0-138-149 kubenswrapper[2573]: I1030 20:37:15.465604 2573 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Oct 30 20:37:15.472093 ip-10-0-138-149 kubenswrapper[2573]: I1030 20:37:15.465608 2573 flags.go:64] FLAG: --sync-frequency="1m0s" Oct 30 20:37:15.472093 ip-10-0-138-149 kubenswrapper[2573]: I1030 20:37:15.465611 2573 flags.go:64] FLAG: --system-cgroups="" Oct 30 20:37:15.472093 ip-10-0-138-149 kubenswrapper[2573]: I1030 20:37:15.465613 2573 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Oct 30 20:37:15.472093 ip-10-0-138-149 kubenswrapper[2573]: I1030 20:37:15.465619 2573 flags.go:64] FLAG: --system-reserved-cgroup="" Oct 30 20:37:15.472093 ip-10-0-138-149 kubenswrapper[2573]: I1030 20:37:15.465622 2573 flags.go:64] FLAG: --tls-cert-file="" Oct 30 20:37:15.472093 ip-10-0-138-149 kubenswrapper[2573]: I1030 20:37:15.465625 2573 flags.go:64] FLAG: --tls-cipher-suites="[]" Oct 30 20:37:15.472093 ip-10-0-138-149 kubenswrapper[2573]: I1030 20:37:15.465629 2573 flags.go:64] FLAG: --tls-min-version="" Oct 30 20:37:15.472843 ip-10-0-138-149 kubenswrapper[2573]: I1030 20:37:15.465632 2573 flags.go:64] FLAG: --tls-private-key-file="" Oct 30 20:37:15.472843 ip-10-0-138-149 kubenswrapper[2573]: I1030 20:37:15.465635 2573 flags.go:64] FLAG: --topology-manager-policy="none" Oct 30 20:37:15.472843 ip-10-0-138-149 kubenswrapper[2573]: I1030 20:37:15.465638 2573 flags.go:64] FLAG: --topology-manager-policy-options="" Oct 30 20:37:15.472843 ip-10-0-138-149 kubenswrapper[2573]: I1030 20:37:15.465641 2573 flags.go:64] FLAG: --topology-manager-scope="container" Oct 30 20:37:15.472843 ip-10-0-138-149 kubenswrapper[2573]: I1030 20:37:15.465644 2573 flags.go:64] FLAG: --v="2" Oct 30 20:37:15.472843 ip-10-0-138-149 kubenswrapper[2573]: I1030 20:37:15.465648 2573 flags.go:64] FLAG: --version="false" Oct 30 20:37:15.472843 ip-10-0-138-149 kubenswrapper[2573]: I1030 20:37:15.465652 2573 flags.go:64] FLAG: --vmodule="" Oct 30 20:37:15.472843 ip-10-0-138-149 kubenswrapper[2573]: I1030 20:37:15.465657 2573 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Oct 30 20:37:15.472843 ip-10-0-138-149 kubenswrapper[2573]: I1030 20:37:15.465660 2573 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Oct 30 20:37:15.472843 ip-10-0-138-149 kubenswrapper[2573]: W1030 20:37:15.465795 2573 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 20:37:15.472843 ip-10-0-138-149 kubenswrapper[2573]: W1030 20:37:15.465798 2573 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 20:37:15.472843 ip-10-0-138-149 kubenswrapper[2573]: W1030 20:37:15.465803 2573 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 20:37:15.472843 ip-10-0-138-149 kubenswrapper[2573]: W1030 20:37:15.465808 2573 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 20:37:15.472843 ip-10-0-138-149 kubenswrapper[2573]: W1030 20:37:15.465812 2573 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 20:37:15.472843 ip-10-0-138-149 kubenswrapper[2573]: W1030 20:37:15.465815 2573 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 20:37:15.472843 ip-10-0-138-149 kubenswrapper[2573]: W1030 20:37:15.465819 2573 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 20:37:15.472843 ip-10-0-138-149 kubenswrapper[2573]: W1030 20:37:15.465823 2573 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 20:37:15.472843 ip-10-0-138-149 kubenswrapper[2573]: W1030 20:37:15.465826 2573 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 20:37:15.472843 ip-10-0-138-149 kubenswrapper[2573]: W1030 20:37:15.465829 2573 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 20:37:15.472843 ip-10-0-138-149 kubenswrapper[2573]: W1030 20:37:15.465832 2573 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 20:37:15.472843 ip-10-0-138-149 kubenswrapper[2573]: W1030 20:37:15.465835 2573 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 20:37:15.473552 ip-10-0-138-149 kubenswrapper[2573]: W1030 20:37:15.465838 2573 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 20:37:15.473552 ip-10-0-138-149 kubenswrapper[2573]: W1030 20:37:15.465841 2573 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 20:37:15.473552 ip-10-0-138-149 kubenswrapper[2573]: W1030 20:37:15.465844 2573 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 20:37:15.473552 ip-10-0-138-149 kubenswrapper[2573]: W1030 20:37:15.465846 2573 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 20:37:15.473552 ip-10-0-138-149 kubenswrapper[2573]: W1030 20:37:15.465849 2573 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 20:37:15.473552 ip-10-0-138-149 kubenswrapper[2573]: W1030 20:37:15.465852 2573 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 20:37:15.473552 ip-10-0-138-149 kubenswrapper[2573]: W1030 20:37:15.465855 2573 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 20:37:15.473552 ip-10-0-138-149 kubenswrapper[2573]: W1030 20:37:15.465857 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 20:37:15.473552 ip-10-0-138-149 kubenswrapper[2573]: W1030 20:37:15.465860 2573 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 20:37:15.473552 ip-10-0-138-149 kubenswrapper[2573]: W1030 20:37:15.465863 2573 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 20:37:15.473552 ip-10-0-138-149 kubenswrapper[2573]: W1030 20:37:15.465865 2573 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 20:37:15.473552 ip-10-0-138-149 kubenswrapper[2573]: W1030 20:37:15.465868 2573 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 20:37:15.473552 ip-10-0-138-149 kubenswrapper[2573]: W1030 20:37:15.465870 2573 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 20:37:15.473552 ip-10-0-138-149 kubenswrapper[2573]: W1030 20:37:15.465873 2573 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 20:37:15.473552 ip-10-0-138-149 kubenswrapper[2573]: W1030 20:37:15.465876 2573 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 20:37:15.473552 ip-10-0-138-149 kubenswrapper[2573]: W1030 20:37:15.465879 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 20:37:15.473552 ip-10-0-138-149 kubenswrapper[2573]: W1030 20:37:15.465881 2573 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 20:37:15.473552 ip-10-0-138-149 kubenswrapper[2573]: W1030 20:37:15.465884 2573 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 20:37:15.473552 ip-10-0-138-149 kubenswrapper[2573]: W1030 20:37:15.465887 2573 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 20:37:15.474017 ip-10-0-138-149 kubenswrapper[2573]: W1030 20:37:15.465889 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 20:37:15.474017 ip-10-0-138-149 kubenswrapper[2573]: W1030 20:37:15.465892 2573 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 20:37:15.474017 ip-10-0-138-149 kubenswrapper[2573]: W1030 20:37:15.465894 2573 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 20:37:15.474017 ip-10-0-138-149 kubenswrapper[2573]: W1030 20:37:15.465898 2573 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 20:37:15.474017 ip-10-0-138-149 kubenswrapper[2573]: W1030 20:37:15.465901 2573 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 20:37:15.474017 ip-10-0-138-149 kubenswrapper[2573]: W1030 20:37:15.465904 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 20:37:15.474017 ip-10-0-138-149 kubenswrapper[2573]: W1030 20:37:15.465906 2573 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 20:37:15.474017 ip-10-0-138-149 kubenswrapper[2573]: W1030 20:37:15.465909 2573 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 20:37:15.474017 ip-10-0-138-149 kubenswrapper[2573]: W1030 20:37:15.465912 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 20:37:15.474017 ip-10-0-138-149 kubenswrapper[2573]: W1030 20:37:15.465914 2573 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 20:37:15.474017 ip-10-0-138-149 kubenswrapper[2573]: W1030 20:37:15.465917 2573 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 20:37:15.474017 ip-10-0-138-149 kubenswrapper[2573]: W1030 20:37:15.465919 2573 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 20:37:15.474017 ip-10-0-138-149 kubenswrapper[2573]: W1030 20:37:15.465922 2573 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 20:37:15.474017 ip-10-0-138-149 kubenswrapper[2573]: W1030 20:37:15.465924 2573 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 20:37:15.474017 ip-10-0-138-149 kubenswrapper[2573]: W1030 20:37:15.465927 2573 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 20:37:15.474017 ip-10-0-138-149 kubenswrapper[2573]: W1030 20:37:15.465930 2573 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 20:37:15.474017 ip-10-0-138-149 kubenswrapper[2573]: W1030 20:37:15.465933 2573 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 20:37:15.474017 ip-10-0-138-149 kubenswrapper[2573]: W1030 20:37:15.465935 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 20:37:15.474017 ip-10-0-138-149 kubenswrapper[2573]: W1030 20:37:15.465938 2573 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 20:37:15.474017 ip-10-0-138-149 kubenswrapper[2573]: W1030 20:37:15.465940 2573 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 20:37:15.474492 ip-10-0-138-149 kubenswrapper[2573]: W1030 20:37:15.465943 2573 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 20:37:15.474492 ip-10-0-138-149 kubenswrapper[2573]: W1030 20:37:15.465946 2573 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 20:37:15.474492 ip-10-0-138-149 kubenswrapper[2573]: W1030 20:37:15.465949 2573 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 20:37:15.474492 ip-10-0-138-149 kubenswrapper[2573]: W1030 20:37:15.465952 2573 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 20:37:15.474492 ip-10-0-138-149 kubenswrapper[2573]: W1030 20:37:15.465954 2573 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 20:37:15.474492 ip-10-0-138-149 kubenswrapper[2573]: W1030 20:37:15.465958 2573 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 20:37:15.474492 ip-10-0-138-149 kubenswrapper[2573]: W1030 20:37:15.465962 2573 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 20:37:15.474492 ip-10-0-138-149 kubenswrapper[2573]: W1030 20:37:15.465965 2573 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 20:37:15.474492 ip-10-0-138-149 kubenswrapper[2573]: W1030 20:37:15.465968 2573 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 20:37:15.474492 ip-10-0-138-149 kubenswrapper[2573]: W1030 20:37:15.465971 2573 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 20:37:15.474492 ip-10-0-138-149 kubenswrapper[2573]: W1030 20:37:15.465973 2573 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 20:37:15.474492 ip-10-0-138-149 kubenswrapper[2573]: W1030 20:37:15.465976 2573 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 20:37:15.474492 ip-10-0-138-149 kubenswrapper[2573]: W1030 20:37:15.465978 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 20:37:15.474492 ip-10-0-138-149 kubenswrapper[2573]: W1030 20:37:15.465981 2573 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 20:37:15.474492 ip-10-0-138-149 kubenswrapper[2573]: W1030 20:37:15.465983 2573 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 20:37:15.474492 ip-10-0-138-149 kubenswrapper[2573]: W1030 20:37:15.465987 2573 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 20:37:15.474492 ip-10-0-138-149 kubenswrapper[2573]: W1030 20:37:15.465990 2573 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 20:37:15.474492 ip-10-0-138-149 kubenswrapper[2573]: W1030 20:37:15.465993 2573 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 20:37:15.474492 ip-10-0-138-149 kubenswrapper[2573]: W1030 20:37:15.465996 2573 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 20:37:15.474978 ip-10-0-138-149 kubenswrapper[2573]: W1030 20:37:15.465998 2573 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 20:37:15.474978 ip-10-0-138-149 kubenswrapper[2573]: W1030 20:37:15.466001 2573 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 20:37:15.474978 ip-10-0-138-149 kubenswrapper[2573]: W1030 20:37:15.466003 2573 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 20:37:15.474978 ip-10-0-138-149 kubenswrapper[2573]: W1030 20:37:15.466006 2573 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 20:37:15.474978 ip-10-0-138-149 kubenswrapper[2573]: W1030 20:37:15.466008 2573 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 20:37:15.474978 ip-10-0-138-149 kubenswrapper[2573]: W1030 20:37:15.466011 2573 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 20:37:15.474978 ip-10-0-138-149 kubenswrapper[2573]: W1030 20:37:15.466014 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 20:37:15.474978 ip-10-0-138-149 kubenswrapper[2573]: W1030 20:37:15.466016 2573 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 20:37:15.474978 ip-10-0-138-149 kubenswrapper[2573]: W1030 20:37:15.466019 2573 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 20:37:15.474978 ip-10-0-138-149 kubenswrapper[2573]: W1030 20:37:15.466021 2573 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 20:37:15.474978 ip-10-0-138-149 kubenswrapper[2573]: W1030 20:37:15.466024 2573 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 20:37:15.474978 ip-10-0-138-149 kubenswrapper[2573]: W1030 20:37:15.466026 2573 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 20:37:15.474978 ip-10-0-138-149 kubenswrapper[2573]: W1030 20:37:15.466029 2573 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 20:37:15.474978 ip-10-0-138-149 kubenswrapper[2573]: W1030 20:37:15.466031 2573 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 20:37:15.474978 ip-10-0-138-149 kubenswrapper[2573]: W1030 20:37:15.466035 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 20:37:15.474978 ip-10-0-138-149 kubenswrapper[2573]: W1030 20:37:15.466038 2573 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 20:37:15.474978 ip-10-0-138-149 kubenswrapper[2573]: W1030 20:37:15.466040 2573 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 20:37:15.474978 ip-10-0-138-149 kubenswrapper[2573]: W1030 20:37:15.466043 2573 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 20:37:15.474978 ip-10-0-138-149 kubenswrapper[2573]: W1030 20:37:15.466046 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 20:37:15.474978 ip-10-0-138-149 kubenswrapper[2573]: W1030 20:37:15.466048 2573 feature_gate.go:386] unrecognized feature gate: Example Oct 30 20:37:15.475454 ip-10-0-138-149 kubenswrapper[2573]: W1030 20:37:15.466051 2573 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 20:37:15.475454 ip-10-0-138-149 kubenswrapper[2573]: W1030 20:37:15.466054 2573 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 20:37:15.475454 ip-10-0-138-149 kubenswrapper[2573]: W1030 20:37:15.466057 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 20:37:15.475454 ip-10-0-138-149 kubenswrapper[2573]: W1030 20:37:15.466059 2573 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 20:37:15.475454 ip-10-0-138-149 kubenswrapper[2573]: W1030 20:37:15.466062 2573 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 20:37:15.475454 ip-10-0-138-149 kubenswrapper[2573]: W1030 20:37:15.466065 2573 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 20:37:15.475454 ip-10-0-138-149 kubenswrapper[2573]: W1030 20:37:15.466067 2573 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 20:37:15.475454 ip-10-0-138-149 kubenswrapper[2573]: W1030 20:37:15.466070 2573 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 20:37:15.475454 ip-10-0-138-149 kubenswrapper[2573]: W1030 20:37:15.466074 2573 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 20:37:15.475454 ip-10-0-138-149 kubenswrapper[2573]: W1030 20:37:15.466076 2573 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 20:37:15.475454 ip-10-0-138-149 kubenswrapper[2573]: W1030 20:37:15.466079 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 20:37:15.475454 ip-10-0-138-149 kubenswrapper[2573]: W1030 20:37:15.466081 2573 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 20:37:15.475454 ip-10-0-138-149 kubenswrapper[2573]: W1030 20:37:15.466084 2573 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 20:37:15.475454 ip-10-0-138-149 kubenswrapper[2573]: W1030 20:37:15.466087 2573 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 20:37:15.475454 ip-10-0-138-149 kubenswrapper[2573]: W1030 20:37:15.466089 2573 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 20:37:15.475454 ip-10-0-138-149 kubenswrapper[2573]: W1030 20:37:15.466092 2573 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 20:37:15.475852 ip-10-0-138-149 kubenswrapper[2573]: I1030 20:37:15.466117 2573 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Oct 30 20:37:15.475852 ip-10-0-138-149 kubenswrapper[2573]: W1030 20:37:15.466153 2573 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 20:37:15.475852 ip-10-0-138-149 kubenswrapper[2573]: W1030 20:37:15.466157 2573 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 20:37:15.475852 ip-10-0-138-149 kubenswrapper[2573]: W1030 20:37:15.466161 2573 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 20:37:15.475852 ip-10-0-138-149 kubenswrapper[2573]: W1030 20:37:15.466165 2573 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 20:37:15.475852 ip-10-0-138-149 kubenswrapper[2573]: W1030 20:37:15.466169 2573 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 20:37:15.475852 ip-10-0-138-149 kubenswrapper[2573]: W1030 20:37:15.466171 2573 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 20:37:15.475852 ip-10-0-138-149 kubenswrapper[2573]: W1030 20:37:15.466174 2573 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 20:37:15.475852 ip-10-0-138-149 kubenswrapper[2573]: W1030 20:37:15.466177 2573 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 20:37:15.475852 ip-10-0-138-149 kubenswrapper[2573]: W1030 20:37:15.466180 2573 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 20:37:15.475852 ip-10-0-138-149 kubenswrapper[2573]: W1030 20:37:15.466183 2573 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 20:37:15.475852 ip-10-0-138-149 kubenswrapper[2573]: W1030 20:37:15.466187 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 20:37:15.475852 ip-10-0-138-149 kubenswrapper[2573]: W1030 20:37:15.466190 2573 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 20:37:15.475852 ip-10-0-138-149 kubenswrapper[2573]: W1030 20:37:15.466192 2573 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 20:37:15.475852 ip-10-0-138-149 kubenswrapper[2573]: W1030 20:37:15.466195 2573 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 20:37:15.476203 ip-10-0-138-149 kubenswrapper[2573]: W1030 20:37:15.466197 2573 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 20:37:15.476203 ip-10-0-138-149 kubenswrapper[2573]: W1030 20:37:15.466200 2573 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 20:37:15.476203 ip-10-0-138-149 kubenswrapper[2573]: W1030 20:37:15.466203 2573 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 20:37:15.476203 ip-10-0-138-149 kubenswrapper[2573]: W1030 20:37:15.466206 2573 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 20:37:15.476203 ip-10-0-138-149 kubenswrapper[2573]: W1030 20:37:15.466209 2573 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 20:37:15.476203 ip-10-0-138-149 kubenswrapper[2573]: W1030 20:37:15.466212 2573 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 20:37:15.476203 ip-10-0-138-149 kubenswrapper[2573]: W1030 20:37:15.466214 2573 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 20:37:15.476203 ip-10-0-138-149 kubenswrapper[2573]: W1030 20:37:15.466217 2573 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 20:37:15.476203 ip-10-0-138-149 kubenswrapper[2573]: W1030 20:37:15.466219 2573 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 20:37:15.476203 ip-10-0-138-149 kubenswrapper[2573]: W1030 20:37:15.466223 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 20:37:15.476203 ip-10-0-138-149 kubenswrapper[2573]: W1030 20:37:15.466226 2573 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 20:37:15.476203 ip-10-0-138-149 kubenswrapper[2573]: W1030 20:37:15.466228 2573 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 20:37:15.476203 ip-10-0-138-149 kubenswrapper[2573]: W1030 20:37:15.466231 2573 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 20:37:15.476203 ip-10-0-138-149 kubenswrapper[2573]: W1030 20:37:15.466234 2573 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 20:37:15.476203 ip-10-0-138-149 kubenswrapper[2573]: W1030 20:37:15.466236 2573 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 20:37:15.476203 ip-10-0-138-149 kubenswrapper[2573]: W1030 20:37:15.466239 2573 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 20:37:15.476203 ip-10-0-138-149 kubenswrapper[2573]: W1030 20:37:15.466241 2573 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 20:37:15.476203 ip-10-0-138-149 kubenswrapper[2573]: W1030 20:37:15.466244 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 20:37:15.476203 ip-10-0-138-149 kubenswrapper[2573]: W1030 20:37:15.466247 2573 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 20:37:15.476203 ip-10-0-138-149 kubenswrapper[2573]: W1030 20:37:15.466249 2573 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 20:37:15.476691 ip-10-0-138-149 kubenswrapper[2573]: W1030 20:37:15.466252 2573 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 20:37:15.476691 ip-10-0-138-149 kubenswrapper[2573]: W1030 20:37:15.466254 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 20:37:15.476691 ip-10-0-138-149 kubenswrapper[2573]: W1030 20:37:15.466257 2573 feature_gate.go:386] unrecognized feature gate: Example Oct 30 20:37:15.476691 ip-10-0-138-149 kubenswrapper[2573]: W1030 20:37:15.466260 2573 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 20:37:15.476691 ip-10-0-138-149 kubenswrapper[2573]: W1030 20:37:15.466262 2573 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 20:37:15.476691 ip-10-0-138-149 kubenswrapper[2573]: W1030 20:37:15.466265 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 20:37:15.476691 ip-10-0-138-149 kubenswrapper[2573]: W1030 20:37:15.466267 2573 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 20:37:15.476691 ip-10-0-138-149 kubenswrapper[2573]: W1030 20:37:15.466270 2573 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 20:37:15.476691 ip-10-0-138-149 kubenswrapper[2573]: W1030 20:37:15.466274 2573 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 20:37:15.476691 ip-10-0-138-149 kubenswrapper[2573]: W1030 20:37:15.466277 2573 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 20:37:15.476691 ip-10-0-138-149 kubenswrapper[2573]: W1030 20:37:15.466280 2573 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 20:37:15.476691 ip-10-0-138-149 kubenswrapper[2573]: W1030 20:37:15.466283 2573 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 20:37:15.476691 ip-10-0-138-149 kubenswrapper[2573]: W1030 20:37:15.466286 2573 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 20:37:15.476691 ip-10-0-138-149 kubenswrapper[2573]: W1030 20:37:15.466289 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 20:37:15.476691 ip-10-0-138-149 kubenswrapper[2573]: W1030 20:37:15.466291 2573 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 20:37:15.476691 ip-10-0-138-149 kubenswrapper[2573]: W1030 20:37:15.466294 2573 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 20:37:15.476691 ip-10-0-138-149 kubenswrapper[2573]: W1030 20:37:15.466297 2573 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 20:37:15.476691 ip-10-0-138-149 kubenswrapper[2573]: W1030 20:37:15.466300 2573 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 20:37:15.476691 ip-10-0-138-149 kubenswrapper[2573]: W1030 20:37:15.466302 2573 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 20:37:15.477132 ip-10-0-138-149 kubenswrapper[2573]: W1030 20:37:15.466305 2573 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 20:37:15.477132 ip-10-0-138-149 kubenswrapper[2573]: W1030 20:37:15.466307 2573 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 20:37:15.477132 ip-10-0-138-149 kubenswrapper[2573]: W1030 20:37:15.466312 2573 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 20:37:15.477132 ip-10-0-138-149 kubenswrapper[2573]: W1030 20:37:15.466316 2573 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 20:37:15.477132 ip-10-0-138-149 kubenswrapper[2573]: W1030 20:37:15.466320 2573 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 20:37:15.477132 ip-10-0-138-149 kubenswrapper[2573]: W1030 20:37:15.466323 2573 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 20:37:15.477132 ip-10-0-138-149 kubenswrapper[2573]: W1030 20:37:15.466327 2573 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 20:37:15.477132 ip-10-0-138-149 kubenswrapper[2573]: W1030 20:37:15.466330 2573 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 20:37:15.477132 ip-10-0-138-149 kubenswrapper[2573]: W1030 20:37:15.466332 2573 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 20:37:15.477132 ip-10-0-138-149 kubenswrapper[2573]: W1030 20:37:15.466335 2573 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 20:37:15.477132 ip-10-0-138-149 kubenswrapper[2573]: W1030 20:37:15.466338 2573 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 20:37:15.477132 ip-10-0-138-149 kubenswrapper[2573]: W1030 20:37:15.466342 2573 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 20:37:15.477132 ip-10-0-138-149 kubenswrapper[2573]: W1030 20:37:15.466345 2573 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 20:37:15.477132 ip-10-0-138-149 kubenswrapper[2573]: W1030 20:37:15.466347 2573 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 20:37:15.477132 ip-10-0-138-149 kubenswrapper[2573]: W1030 20:37:15.466350 2573 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 20:37:15.477132 ip-10-0-138-149 kubenswrapper[2573]: W1030 20:37:15.466352 2573 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 20:37:15.477132 ip-10-0-138-149 kubenswrapper[2573]: W1030 20:37:15.466355 2573 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 20:37:15.477132 ip-10-0-138-149 kubenswrapper[2573]: W1030 20:37:15.466358 2573 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 20:37:15.477132 ip-10-0-138-149 kubenswrapper[2573]: W1030 20:37:15.466360 2573 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 20:37:15.477592 ip-10-0-138-149 kubenswrapper[2573]: W1030 20:37:15.466363 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 20:37:15.477592 ip-10-0-138-149 kubenswrapper[2573]: W1030 20:37:15.466366 2573 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 20:37:15.477592 ip-10-0-138-149 kubenswrapper[2573]: W1030 20:37:15.466369 2573 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 20:37:15.477592 ip-10-0-138-149 kubenswrapper[2573]: W1030 20:37:15.466372 2573 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 20:37:15.477592 ip-10-0-138-149 kubenswrapper[2573]: W1030 20:37:15.466375 2573 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 20:37:15.477592 ip-10-0-138-149 kubenswrapper[2573]: W1030 20:37:15.466377 2573 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 20:37:15.477592 ip-10-0-138-149 kubenswrapper[2573]: W1030 20:37:15.466380 2573 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 20:37:15.477592 ip-10-0-138-149 kubenswrapper[2573]: W1030 20:37:15.466382 2573 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 20:37:15.477592 ip-10-0-138-149 kubenswrapper[2573]: W1030 20:37:15.466385 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 20:37:15.477592 ip-10-0-138-149 kubenswrapper[2573]: W1030 20:37:15.466388 2573 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 20:37:15.477592 ip-10-0-138-149 kubenswrapper[2573]: W1030 20:37:15.466390 2573 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 20:37:15.477592 ip-10-0-138-149 kubenswrapper[2573]: W1030 20:37:15.466393 2573 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 20:37:15.477592 ip-10-0-138-149 kubenswrapper[2573]: W1030 20:37:15.466396 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 20:37:15.477592 ip-10-0-138-149 kubenswrapper[2573]: W1030 20:37:15.466399 2573 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 20:37:15.477592 ip-10-0-138-149 kubenswrapper[2573]: W1030 20:37:15.466402 2573 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 20:37:15.477592 ip-10-0-138-149 kubenswrapper[2573]: W1030 20:37:15.466406 2573 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 20:37:15.477592 ip-10-0-138-149 kubenswrapper[2573]: W1030 20:37:15.466408 2573 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 20:37:15.477592 ip-10-0-138-149 kubenswrapper[2573]: W1030 20:37:15.466412 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 20:37:15.477592 ip-10-0-138-149 kubenswrapper[2573]: W1030 20:37:15.466415 2573 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 20:37:15.478047 ip-10-0-138-149 kubenswrapper[2573]: W1030 20:37:15.466417 2573 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 20:37:15.478047 ip-10-0-138-149 kubenswrapper[2573]: W1030 20:37:15.466420 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 20:37:15.478047 ip-10-0-138-149 kubenswrapper[2573]: W1030 20:37:15.466422 2573 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 20:37:15.478047 ip-10-0-138-149 kubenswrapper[2573]: W1030 20:37:15.466425 2573 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 20:37:15.478047 ip-10-0-138-149 kubenswrapper[2573]: W1030 20:37:15.466427 2573 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 20:37:15.478047 ip-10-0-138-149 kubenswrapper[2573]: W1030 20:37:15.466430 2573 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 20:37:15.478047 ip-10-0-138-149 kubenswrapper[2573]: W1030 20:37:15.466432 2573 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 20:37:15.478047 ip-10-0-138-149 kubenswrapper[2573]: W1030 20:37:15.466435 2573 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 20:37:15.478047 ip-10-0-138-149 kubenswrapper[2573]: W1030 20:37:15.466437 2573 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 20:37:15.478047 ip-10-0-138-149 kubenswrapper[2573]: W1030 20:37:15.466440 2573 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 20:37:15.478047 ip-10-0-138-149 kubenswrapper[2573]: W1030 20:37:15.466443 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 20:37:15.478047 ip-10-0-138-149 kubenswrapper[2573]: W1030 20:37:15.466445 2573 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 20:37:15.478047 ip-10-0-138-149 kubenswrapper[2573]: W1030 20:37:15.466448 2573 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 20:37:15.478047 ip-10-0-138-149 kubenswrapper[2573]: W1030 20:37:15.466451 2573 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 20:37:15.478047 ip-10-0-138-149 kubenswrapper[2573]: W1030 20:37:15.466453 2573 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 20:37:15.478047 ip-10-0-138-149 kubenswrapper[2573]: I1030 20:37:15.474676 2573 server.go:525] "Kubelet version" kubeletVersion="v1.35.9" Oct 30 20:37:15.478047 ip-10-0-138-149 kubenswrapper[2573]: I1030 20:37:15.474693 2573 server.go:527] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Oct 30 20:37:15.478047 ip-10-0-138-149 kubenswrapper[2573]: I1030 20:37:15.474710 2573 watchdog_linux.go:95] "Systemd watchdog is not enabled" Oct 30 20:37:15.478047 ip-10-0-138-149 kubenswrapper[2573]: I1030 20:37:15.474715 2573 watchdog_linux.go:138] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Oct 30 20:37:15.478047 ip-10-0-138-149 kubenswrapper[2573]: W1030 20:37:15.474772 2573 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 20:37:15.478508 ip-10-0-138-149 kubenswrapper[2573]: W1030 20:37:15.474776 2573 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 20:37:15.478508 ip-10-0-138-149 kubenswrapper[2573]: W1030 20:37:15.474780 2573 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 20:37:15.478508 ip-10-0-138-149 kubenswrapper[2573]: W1030 20:37:15.474783 2573 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 20:37:15.478508 ip-10-0-138-149 kubenswrapper[2573]: W1030 20:37:15.474787 2573 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 20:37:15.478508 ip-10-0-138-149 kubenswrapper[2573]: W1030 20:37:15.474791 2573 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 20:37:15.478508 ip-10-0-138-149 kubenswrapper[2573]: W1030 20:37:15.474794 2573 feature_gate.go:386] unrecognized feature gate: Example Oct 30 20:37:15.478508 ip-10-0-138-149 kubenswrapper[2573]: W1030 20:37:15.474798 2573 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 20:37:15.478508 ip-10-0-138-149 kubenswrapper[2573]: W1030 20:37:15.474801 2573 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 20:37:15.478508 ip-10-0-138-149 kubenswrapper[2573]: W1030 20:37:15.474804 2573 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 20:37:15.478508 ip-10-0-138-149 kubenswrapper[2573]: W1030 20:37:15.474808 2573 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 20:37:15.478508 ip-10-0-138-149 kubenswrapper[2573]: W1030 20:37:15.474811 2573 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 20:37:15.478508 ip-10-0-138-149 kubenswrapper[2573]: W1030 20:37:15.474814 2573 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 20:37:15.478508 ip-10-0-138-149 kubenswrapper[2573]: W1030 20:37:15.474816 2573 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 20:37:15.478508 ip-10-0-138-149 kubenswrapper[2573]: W1030 20:37:15.474819 2573 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 20:37:15.478508 ip-10-0-138-149 kubenswrapper[2573]: W1030 20:37:15.474822 2573 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 20:37:15.478508 ip-10-0-138-149 kubenswrapper[2573]: W1030 20:37:15.474825 2573 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 20:37:15.478508 ip-10-0-138-149 kubenswrapper[2573]: W1030 20:37:15.474827 2573 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 20:37:15.478508 ip-10-0-138-149 kubenswrapper[2573]: W1030 20:37:15.474830 2573 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 20:37:15.478508 ip-10-0-138-149 kubenswrapper[2573]: W1030 20:37:15.474833 2573 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 20:37:15.478973 ip-10-0-138-149 kubenswrapper[2573]: W1030 20:37:15.474835 2573 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 20:37:15.478973 ip-10-0-138-149 kubenswrapper[2573]: W1030 20:37:15.474838 2573 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 20:37:15.478973 ip-10-0-138-149 kubenswrapper[2573]: W1030 20:37:15.474841 2573 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 20:37:15.478973 ip-10-0-138-149 kubenswrapper[2573]: W1030 20:37:15.474844 2573 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 20:37:15.478973 ip-10-0-138-149 kubenswrapper[2573]: W1030 20:37:15.474847 2573 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 20:37:15.478973 ip-10-0-138-149 kubenswrapper[2573]: W1030 20:37:15.474849 2573 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 20:37:15.478973 ip-10-0-138-149 kubenswrapper[2573]: W1030 20:37:15.474852 2573 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 20:37:15.478973 ip-10-0-138-149 kubenswrapper[2573]: W1030 20:37:15.474862 2573 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 20:37:15.478973 ip-10-0-138-149 kubenswrapper[2573]: W1030 20:37:15.474866 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 20:37:15.478973 ip-10-0-138-149 kubenswrapper[2573]: W1030 20:37:15.474869 2573 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 20:37:15.478973 ip-10-0-138-149 kubenswrapper[2573]: W1030 20:37:15.474872 2573 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 20:37:15.478973 ip-10-0-138-149 kubenswrapper[2573]: W1030 20:37:15.474875 2573 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 20:37:15.478973 ip-10-0-138-149 kubenswrapper[2573]: W1030 20:37:15.474878 2573 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 20:37:15.478973 ip-10-0-138-149 kubenswrapper[2573]: W1030 20:37:15.474881 2573 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 20:37:15.478973 ip-10-0-138-149 kubenswrapper[2573]: W1030 20:37:15.474884 2573 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 20:37:15.478973 ip-10-0-138-149 kubenswrapper[2573]: W1030 20:37:15.474887 2573 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 20:37:15.478973 ip-10-0-138-149 kubenswrapper[2573]: W1030 20:37:15.474889 2573 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 20:37:15.478973 ip-10-0-138-149 kubenswrapper[2573]: W1030 20:37:15.474892 2573 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 20:37:15.478973 ip-10-0-138-149 kubenswrapper[2573]: W1030 20:37:15.474895 2573 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 20:37:15.479415 ip-10-0-138-149 kubenswrapper[2573]: W1030 20:37:15.474897 2573 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 20:37:15.479415 ip-10-0-138-149 kubenswrapper[2573]: W1030 20:37:15.474900 2573 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 20:37:15.479415 ip-10-0-138-149 kubenswrapper[2573]: W1030 20:37:15.474903 2573 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 20:37:15.479415 ip-10-0-138-149 kubenswrapper[2573]: W1030 20:37:15.474905 2573 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 20:37:15.479415 ip-10-0-138-149 kubenswrapper[2573]: W1030 20:37:15.474908 2573 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 20:37:15.479415 ip-10-0-138-149 kubenswrapper[2573]: W1030 20:37:15.474910 2573 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 20:37:15.479415 ip-10-0-138-149 kubenswrapper[2573]: W1030 20:37:15.474913 2573 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 20:37:15.479415 ip-10-0-138-149 kubenswrapper[2573]: W1030 20:37:15.474916 2573 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 20:37:15.479415 ip-10-0-138-149 kubenswrapper[2573]: W1030 20:37:15.474919 2573 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 20:37:15.479415 ip-10-0-138-149 kubenswrapper[2573]: W1030 20:37:15.474921 2573 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 20:37:15.479415 ip-10-0-138-149 kubenswrapper[2573]: W1030 20:37:15.474924 2573 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 20:37:15.479415 ip-10-0-138-149 kubenswrapper[2573]: W1030 20:37:15.474927 2573 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 20:37:15.479415 ip-10-0-138-149 kubenswrapper[2573]: W1030 20:37:15.474929 2573 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 20:37:15.479415 ip-10-0-138-149 kubenswrapper[2573]: W1030 20:37:15.474933 2573 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 20:37:15.479415 ip-10-0-138-149 kubenswrapper[2573]: W1030 20:37:15.474937 2573 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 20:37:15.479415 ip-10-0-138-149 kubenswrapper[2573]: W1030 20:37:15.474941 2573 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 20:37:15.479415 ip-10-0-138-149 kubenswrapper[2573]: W1030 20:37:15.474944 2573 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 20:37:15.479415 ip-10-0-138-149 kubenswrapper[2573]: W1030 20:37:15.474947 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 20:37:15.479415 ip-10-0-138-149 kubenswrapper[2573]: W1030 20:37:15.474950 2573 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 20:37:15.479415 ip-10-0-138-149 kubenswrapper[2573]: W1030 20:37:15.474952 2573 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 20:37:15.479895 ip-10-0-138-149 kubenswrapper[2573]: W1030 20:37:15.474955 2573 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 20:37:15.479895 ip-10-0-138-149 kubenswrapper[2573]: W1030 20:37:15.474959 2573 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 20:37:15.479895 ip-10-0-138-149 kubenswrapper[2573]: W1030 20:37:15.474961 2573 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 20:37:15.479895 ip-10-0-138-149 kubenswrapper[2573]: W1030 20:37:15.474964 2573 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 20:37:15.479895 ip-10-0-138-149 kubenswrapper[2573]: W1030 20:37:15.474967 2573 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 20:37:15.479895 ip-10-0-138-149 kubenswrapper[2573]: W1030 20:37:15.474969 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 20:37:15.479895 ip-10-0-138-149 kubenswrapper[2573]: W1030 20:37:15.474972 2573 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 20:37:15.479895 ip-10-0-138-149 kubenswrapper[2573]: W1030 20:37:15.474975 2573 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 20:37:15.479895 ip-10-0-138-149 kubenswrapper[2573]: W1030 20:37:15.474977 2573 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 20:37:15.479895 ip-10-0-138-149 kubenswrapper[2573]: W1030 20:37:15.474980 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 20:37:15.479895 ip-10-0-138-149 kubenswrapper[2573]: W1030 20:37:15.474983 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 20:37:15.479895 ip-10-0-138-149 kubenswrapper[2573]: W1030 20:37:15.474986 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 20:37:15.479895 ip-10-0-138-149 kubenswrapper[2573]: W1030 20:37:15.474989 2573 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 20:37:15.479895 ip-10-0-138-149 kubenswrapper[2573]: W1030 20:37:15.474992 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 20:37:15.479895 ip-10-0-138-149 kubenswrapper[2573]: W1030 20:37:15.474995 2573 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 20:37:15.479895 ip-10-0-138-149 kubenswrapper[2573]: W1030 20:37:15.474998 2573 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 20:37:15.479895 ip-10-0-138-149 kubenswrapper[2573]: W1030 20:37:15.475000 2573 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 20:37:15.479895 ip-10-0-138-149 kubenswrapper[2573]: W1030 20:37:15.475003 2573 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 20:37:15.479895 ip-10-0-138-149 kubenswrapper[2573]: W1030 20:37:15.475006 2573 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 20:37:15.479895 ip-10-0-138-149 kubenswrapper[2573]: W1030 20:37:15.475008 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 20:37:15.480346 ip-10-0-138-149 kubenswrapper[2573]: W1030 20:37:15.475012 2573 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 20:37:15.480346 ip-10-0-138-149 kubenswrapper[2573]: W1030 20:37:15.475015 2573 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 20:37:15.480346 ip-10-0-138-149 kubenswrapper[2573]: W1030 20:37:15.475017 2573 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 20:37:15.480346 ip-10-0-138-149 kubenswrapper[2573]: W1030 20:37:15.475020 2573 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 20:37:15.480346 ip-10-0-138-149 kubenswrapper[2573]: W1030 20:37:15.475023 2573 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 20:37:15.480346 ip-10-0-138-149 kubenswrapper[2573]: W1030 20:37:15.475025 2573 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 20:37:15.480346 ip-10-0-138-149 kubenswrapper[2573]: W1030 20:37:15.475028 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 20:37:15.480346 ip-10-0-138-149 kubenswrapper[2573]: W1030 20:37:15.475031 2573 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 20:37:15.480346 ip-10-0-138-149 kubenswrapper[2573]: W1030 20:37:15.475033 2573 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 20:37:15.480346 ip-10-0-138-149 kubenswrapper[2573]: W1030 20:37:15.475035 2573 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 20:37:15.480346 ip-10-0-138-149 kubenswrapper[2573]: W1030 20:37:15.475038 2573 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 20:37:15.480346 ip-10-0-138-149 kubenswrapper[2573]: W1030 20:37:15.475041 2573 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 20:37:15.480346 ip-10-0-138-149 kubenswrapper[2573]: W1030 20:37:15.475044 2573 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 20:37:15.480346 ip-10-0-138-149 kubenswrapper[2573]: W1030 20:37:15.475047 2573 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 20:37:15.480346 ip-10-0-138-149 kubenswrapper[2573]: W1030 20:37:15.475049 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 20:37:15.480346 ip-10-0-138-149 kubenswrapper[2573]: W1030 20:37:15.475052 2573 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 20:37:15.480346 ip-10-0-138-149 kubenswrapper[2573]: W1030 20:37:15.475054 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 20:37:15.480346 ip-10-0-138-149 kubenswrapper[2573]: W1030 20:37:15.475057 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 20:37:15.480346 ip-10-0-138-149 kubenswrapper[2573]: W1030 20:37:15.475060 2573 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 20:37:15.480842 ip-10-0-138-149 kubenswrapper[2573]: W1030 20:37:15.475063 2573 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 20:37:15.480842 ip-10-0-138-149 kubenswrapper[2573]: W1030 20:37:15.475065 2573 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 20:37:15.480842 ip-10-0-138-149 kubenswrapper[2573]: W1030 20:37:15.475068 2573 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 20:37:15.480842 ip-10-0-138-149 kubenswrapper[2573]: W1030 20:37:15.475071 2573 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 20:37:15.480842 ip-10-0-138-149 kubenswrapper[2573]: W1030 20:37:15.475073 2573 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 20:37:15.480842 ip-10-0-138-149 kubenswrapper[2573]: W1030 20:37:15.475076 2573 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 20:37:15.480842 ip-10-0-138-149 kubenswrapper[2573]: W1030 20:37:15.475078 2573 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 20:37:15.480842 ip-10-0-138-149 kubenswrapper[2573]: W1030 20:37:15.475081 2573 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 20:37:15.480842 ip-10-0-138-149 kubenswrapper[2573]: I1030 20:37:15.475109 2573 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Oct 30 20:37:15.480842 ip-10-0-138-149 kubenswrapper[2573]: W1030 20:37:15.475241 2573 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 20:37:15.480842 ip-10-0-138-149 kubenswrapper[2573]: W1030 20:37:15.475245 2573 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 20:37:15.480842 ip-10-0-138-149 kubenswrapper[2573]: W1030 20:37:15.475248 2573 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 20:37:15.480842 ip-10-0-138-149 kubenswrapper[2573]: W1030 20:37:15.475251 2573 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 20:37:15.480842 ip-10-0-138-149 kubenswrapper[2573]: W1030 20:37:15.475253 2573 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 20:37:15.480842 ip-10-0-138-149 kubenswrapper[2573]: W1030 20:37:15.475257 2573 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 20:37:15.480842 ip-10-0-138-149 kubenswrapper[2573]: W1030 20:37:15.475260 2573 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 20:37:15.481228 ip-10-0-138-149 kubenswrapper[2573]: W1030 20:37:15.475262 2573 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 20:37:15.481228 ip-10-0-138-149 kubenswrapper[2573]: W1030 20:37:15.475265 2573 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 20:37:15.481228 ip-10-0-138-149 kubenswrapper[2573]: W1030 20:37:15.475267 2573 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 20:37:15.481228 ip-10-0-138-149 kubenswrapper[2573]: W1030 20:37:15.475270 2573 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 20:37:15.481228 ip-10-0-138-149 kubenswrapper[2573]: W1030 20:37:15.475273 2573 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 20:37:15.481228 ip-10-0-138-149 kubenswrapper[2573]: W1030 20:37:15.475275 2573 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 20:37:15.481228 ip-10-0-138-149 kubenswrapper[2573]: W1030 20:37:15.475278 2573 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 20:37:15.481228 ip-10-0-138-149 kubenswrapper[2573]: W1030 20:37:15.475280 2573 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 20:37:15.481228 ip-10-0-138-149 kubenswrapper[2573]: W1030 20:37:15.475283 2573 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 20:37:15.481228 ip-10-0-138-149 kubenswrapper[2573]: W1030 20:37:15.475286 2573 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 20:37:15.481228 ip-10-0-138-149 kubenswrapper[2573]: W1030 20:37:15.475289 2573 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 20:37:15.481228 ip-10-0-138-149 kubenswrapper[2573]: W1030 20:37:15.475291 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 20:37:15.481228 ip-10-0-138-149 kubenswrapper[2573]: W1030 20:37:15.475294 2573 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 20:37:15.481228 ip-10-0-138-149 kubenswrapper[2573]: W1030 20:37:15.475297 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 20:37:15.481228 ip-10-0-138-149 kubenswrapper[2573]: W1030 20:37:15.475299 2573 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 20:37:15.481228 ip-10-0-138-149 kubenswrapper[2573]: W1030 20:37:15.475302 2573 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 20:37:15.481228 ip-10-0-138-149 kubenswrapper[2573]: W1030 20:37:15.475304 2573 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 20:37:15.481228 ip-10-0-138-149 kubenswrapper[2573]: W1030 20:37:15.475307 2573 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 20:37:15.481228 ip-10-0-138-149 kubenswrapper[2573]: W1030 20:37:15.475309 2573 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 20:37:15.481228 ip-10-0-138-149 kubenswrapper[2573]: W1030 20:37:15.475312 2573 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 20:37:15.481731 ip-10-0-138-149 kubenswrapper[2573]: W1030 20:37:15.475315 2573 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 20:37:15.481731 ip-10-0-138-149 kubenswrapper[2573]: W1030 20:37:15.475317 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 20:37:15.481731 ip-10-0-138-149 kubenswrapper[2573]: W1030 20:37:15.475320 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 20:37:15.481731 ip-10-0-138-149 kubenswrapper[2573]: W1030 20:37:15.475322 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 20:37:15.481731 ip-10-0-138-149 kubenswrapper[2573]: W1030 20:37:15.475325 2573 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 20:37:15.481731 ip-10-0-138-149 kubenswrapper[2573]: W1030 20:37:15.475328 2573 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 20:37:15.481731 ip-10-0-138-149 kubenswrapper[2573]: W1030 20:37:15.475331 2573 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 20:37:15.481731 ip-10-0-138-149 kubenswrapper[2573]: W1030 20:37:15.475333 2573 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 20:37:15.481731 ip-10-0-138-149 kubenswrapper[2573]: W1030 20:37:15.475336 2573 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 20:37:15.481731 ip-10-0-138-149 kubenswrapper[2573]: W1030 20:37:15.475340 2573 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 20:37:15.481731 ip-10-0-138-149 kubenswrapper[2573]: W1030 20:37:15.475343 2573 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 20:37:15.481731 ip-10-0-138-149 kubenswrapper[2573]: W1030 20:37:15.475346 2573 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 20:37:15.481731 ip-10-0-138-149 kubenswrapper[2573]: W1030 20:37:15.475349 2573 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 20:37:15.481731 ip-10-0-138-149 kubenswrapper[2573]: W1030 20:37:15.475351 2573 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 20:37:15.481731 ip-10-0-138-149 kubenswrapper[2573]: W1030 20:37:15.475354 2573 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 20:37:15.481731 ip-10-0-138-149 kubenswrapper[2573]: W1030 20:37:15.475357 2573 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 20:37:15.481731 ip-10-0-138-149 kubenswrapper[2573]: W1030 20:37:15.475359 2573 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 20:37:15.481731 ip-10-0-138-149 kubenswrapper[2573]: W1030 20:37:15.475362 2573 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 20:37:15.481731 ip-10-0-138-149 kubenswrapper[2573]: W1030 20:37:15.475364 2573 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 20:37:15.482189 ip-10-0-138-149 kubenswrapper[2573]: W1030 20:37:15.475367 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 20:37:15.482189 ip-10-0-138-149 kubenswrapper[2573]: W1030 20:37:15.475369 2573 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 20:37:15.482189 ip-10-0-138-149 kubenswrapper[2573]: W1030 20:37:15.475372 2573 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 20:37:15.482189 ip-10-0-138-149 kubenswrapper[2573]: W1030 20:37:15.475375 2573 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 20:37:15.482189 ip-10-0-138-149 kubenswrapper[2573]: W1030 20:37:15.475378 2573 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 20:37:15.482189 ip-10-0-138-149 kubenswrapper[2573]: W1030 20:37:15.475380 2573 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 20:37:15.482189 ip-10-0-138-149 kubenswrapper[2573]: W1030 20:37:15.475383 2573 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 20:37:15.482189 ip-10-0-138-149 kubenswrapper[2573]: W1030 20:37:15.475385 2573 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 20:37:15.482189 ip-10-0-138-149 kubenswrapper[2573]: W1030 20:37:15.475388 2573 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 20:37:15.482189 ip-10-0-138-149 kubenswrapper[2573]: W1030 20:37:15.475391 2573 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 20:37:15.482189 ip-10-0-138-149 kubenswrapper[2573]: W1030 20:37:15.475393 2573 feature_gate.go:386] unrecognized feature gate: Example Oct 30 20:37:15.482189 ip-10-0-138-149 kubenswrapper[2573]: W1030 20:37:15.475396 2573 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 20:37:15.482189 ip-10-0-138-149 kubenswrapper[2573]: W1030 20:37:15.475398 2573 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 20:37:15.482189 ip-10-0-138-149 kubenswrapper[2573]: W1030 20:37:15.475401 2573 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 20:37:15.482189 ip-10-0-138-149 kubenswrapper[2573]: W1030 20:37:15.475404 2573 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 20:37:15.482189 ip-10-0-138-149 kubenswrapper[2573]: W1030 20:37:15.475406 2573 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 20:37:15.482189 ip-10-0-138-149 kubenswrapper[2573]: W1030 20:37:15.475409 2573 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 20:37:15.482189 ip-10-0-138-149 kubenswrapper[2573]: W1030 20:37:15.475412 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 20:37:15.482189 ip-10-0-138-149 kubenswrapper[2573]: W1030 20:37:15.475414 2573 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 20:37:15.482701 ip-10-0-138-149 kubenswrapper[2573]: W1030 20:37:15.475418 2573 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 20:37:15.482701 ip-10-0-138-149 kubenswrapper[2573]: W1030 20:37:15.475422 2573 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 20:37:15.482701 ip-10-0-138-149 kubenswrapper[2573]: W1030 20:37:15.475426 2573 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 20:37:15.482701 ip-10-0-138-149 kubenswrapper[2573]: W1030 20:37:15.475429 2573 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 20:37:15.482701 ip-10-0-138-149 kubenswrapper[2573]: W1030 20:37:15.475432 2573 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 20:37:15.482701 ip-10-0-138-149 kubenswrapper[2573]: W1030 20:37:15.475435 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 20:37:15.482701 ip-10-0-138-149 kubenswrapper[2573]: W1030 20:37:15.475438 2573 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 20:37:15.482701 ip-10-0-138-149 kubenswrapper[2573]: W1030 20:37:15.475441 2573 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 20:37:15.482701 ip-10-0-138-149 kubenswrapper[2573]: W1030 20:37:15.475444 2573 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 20:37:15.482701 ip-10-0-138-149 kubenswrapper[2573]: W1030 20:37:15.475446 2573 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 20:37:15.482701 ip-10-0-138-149 kubenswrapper[2573]: W1030 20:37:15.475449 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 20:37:15.482701 ip-10-0-138-149 kubenswrapper[2573]: W1030 20:37:15.475451 2573 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 20:37:15.482701 ip-10-0-138-149 kubenswrapper[2573]: W1030 20:37:15.475454 2573 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 20:37:15.482701 ip-10-0-138-149 kubenswrapper[2573]: W1030 20:37:15.475457 2573 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 20:37:15.482701 ip-10-0-138-149 kubenswrapper[2573]: W1030 20:37:15.475460 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 20:37:15.482701 ip-10-0-138-149 kubenswrapper[2573]: W1030 20:37:15.475462 2573 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 20:37:15.482701 ip-10-0-138-149 kubenswrapper[2573]: W1030 20:37:15.475465 2573 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 20:37:15.482701 ip-10-0-138-149 kubenswrapper[2573]: W1030 20:37:15.475468 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 20:37:15.482701 ip-10-0-138-149 kubenswrapper[2573]: W1030 20:37:15.475471 2573 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 20:37:15.482701 ip-10-0-138-149 kubenswrapper[2573]: W1030 20:37:15.475473 2573 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 20:37:15.483175 ip-10-0-138-149 kubenswrapper[2573]: W1030 20:37:15.475476 2573 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 20:37:15.483175 ip-10-0-138-149 kubenswrapper[2573]: W1030 20:37:15.475478 2573 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 20:37:15.483175 ip-10-0-138-149 kubenswrapper[2573]: W1030 20:37:15.475481 2573 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 20:37:15.483175 ip-10-0-138-149 kubenswrapper[2573]: W1030 20:37:15.475484 2573 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 20:37:15.483175 ip-10-0-138-149 kubenswrapper[2573]: W1030 20:37:15.475486 2573 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 20:37:15.483175 ip-10-0-138-149 kubenswrapper[2573]: W1030 20:37:15.475489 2573 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 20:37:15.483175 ip-10-0-138-149 kubenswrapper[2573]: W1030 20:37:15.475492 2573 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 20:37:15.483175 ip-10-0-138-149 kubenswrapper[2573]: W1030 20:37:15.475494 2573 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 20:37:15.483175 ip-10-0-138-149 kubenswrapper[2573]: W1030 20:37:15.475497 2573 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 20:37:15.483175 ip-10-0-138-149 kubenswrapper[2573]: W1030 20:37:15.475500 2573 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 20:37:15.483175 ip-10-0-138-149 kubenswrapper[2573]: W1030 20:37:15.475502 2573 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 20:37:15.483175 ip-10-0-138-149 kubenswrapper[2573]: W1030 20:37:15.475505 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 20:37:15.483175 ip-10-0-138-149 kubenswrapper[2573]: W1030 20:37:15.475507 2573 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 20:37:15.483175 ip-10-0-138-149 kubenswrapper[2573]: W1030 20:37:15.475510 2573 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 20:37:15.483175 ip-10-0-138-149 kubenswrapper[2573]: W1030 20:37:15.475513 2573 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 20:37:15.483175 ip-10-0-138-149 kubenswrapper[2573]: W1030 20:37:15.475515 2573 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 20:37:15.483175 ip-10-0-138-149 kubenswrapper[2573]: W1030 20:37:15.475518 2573 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 20:37:15.483175 ip-10-0-138-149 kubenswrapper[2573]: W1030 20:37:15.475520 2573 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 20:37:15.483175 ip-10-0-138-149 kubenswrapper[2573]: W1030 20:37:15.475523 2573 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 20:37:15.483654 ip-10-0-138-149 kubenswrapper[2573]: W1030 20:37:15.475526 2573 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 20:37:15.483654 ip-10-0-138-149 kubenswrapper[2573]: W1030 20:37:15.475528 2573 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 20:37:15.483654 ip-10-0-138-149 kubenswrapper[2573]: I1030 20:37:15.475551 2573 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Oct 30 20:37:15.483654 ip-10-0-138-149 kubenswrapper[2573]: W1030 20:37:15.475594 2573 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 20:37:15.483654 ip-10-0-138-149 kubenswrapper[2573]: W1030 20:37:15.475598 2573 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 20:37:15.483654 ip-10-0-138-149 kubenswrapper[2573]: W1030 20:37:15.475602 2573 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 20:37:15.483654 ip-10-0-138-149 kubenswrapper[2573]: W1030 20:37:15.475604 2573 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 20:37:15.483654 ip-10-0-138-149 kubenswrapper[2573]: W1030 20:37:15.475607 2573 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 20:37:15.483654 ip-10-0-138-149 kubenswrapper[2573]: W1030 20:37:15.475610 2573 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 20:37:15.483654 ip-10-0-138-149 kubenswrapper[2573]: W1030 20:37:15.475613 2573 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 20:37:15.483654 ip-10-0-138-149 kubenswrapper[2573]: W1030 20:37:15.475616 2573 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 20:37:15.483654 ip-10-0-138-149 kubenswrapper[2573]: W1030 20:37:15.475618 2573 feature_gate.go:386] unrecognized feature gate: Example Oct 30 20:37:15.483654 ip-10-0-138-149 kubenswrapper[2573]: W1030 20:37:15.475621 2573 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 20:37:15.483654 ip-10-0-138-149 kubenswrapper[2573]: W1030 20:37:15.475623 2573 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 20:37:15.483654 ip-10-0-138-149 kubenswrapper[2573]: W1030 20:37:15.475626 2573 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 20:37:15.483654 ip-10-0-138-149 kubenswrapper[2573]: W1030 20:37:15.475629 2573 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 20:37:15.484052 ip-10-0-138-149 kubenswrapper[2573]: W1030 20:37:15.475631 2573 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 20:37:15.484052 ip-10-0-138-149 kubenswrapper[2573]: W1030 20:37:15.475634 2573 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 20:37:15.484052 ip-10-0-138-149 kubenswrapper[2573]: W1030 20:37:15.475636 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 20:37:15.484052 ip-10-0-138-149 kubenswrapper[2573]: W1030 20:37:15.475639 2573 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 20:37:15.484052 ip-10-0-138-149 kubenswrapper[2573]: W1030 20:37:15.475643 2573 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 20:37:15.484052 ip-10-0-138-149 kubenswrapper[2573]: W1030 20:37:15.475647 2573 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 20:37:15.484052 ip-10-0-138-149 kubenswrapper[2573]: W1030 20:37:15.475650 2573 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 20:37:15.484052 ip-10-0-138-149 kubenswrapper[2573]: W1030 20:37:15.475652 2573 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 20:37:15.484052 ip-10-0-138-149 kubenswrapper[2573]: W1030 20:37:15.475655 2573 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 20:37:15.484052 ip-10-0-138-149 kubenswrapper[2573]: W1030 20:37:15.475658 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 20:37:15.484052 ip-10-0-138-149 kubenswrapper[2573]: W1030 20:37:15.475660 2573 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 20:37:15.484052 ip-10-0-138-149 kubenswrapper[2573]: W1030 20:37:15.475663 2573 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 20:37:15.484052 ip-10-0-138-149 kubenswrapper[2573]: W1030 20:37:15.475666 2573 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 20:37:15.484052 ip-10-0-138-149 kubenswrapper[2573]: W1030 20:37:15.475668 2573 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 20:37:15.484052 ip-10-0-138-149 kubenswrapper[2573]: W1030 20:37:15.475671 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 20:37:15.484052 ip-10-0-138-149 kubenswrapper[2573]: W1030 20:37:15.475673 2573 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 20:37:15.484052 ip-10-0-138-149 kubenswrapper[2573]: W1030 20:37:15.475676 2573 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 20:37:15.484052 ip-10-0-138-149 kubenswrapper[2573]: W1030 20:37:15.475679 2573 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 20:37:15.484052 ip-10-0-138-149 kubenswrapper[2573]: W1030 20:37:15.475682 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 20:37:15.484492 ip-10-0-138-149 kubenswrapper[2573]: W1030 20:37:15.475684 2573 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 20:37:15.484492 ip-10-0-138-149 kubenswrapper[2573]: W1030 20:37:15.475687 2573 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 20:37:15.484492 ip-10-0-138-149 kubenswrapper[2573]: W1030 20:37:15.475690 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 20:37:15.484492 ip-10-0-138-149 kubenswrapper[2573]: W1030 20:37:15.475692 2573 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 20:37:15.484492 ip-10-0-138-149 kubenswrapper[2573]: W1030 20:37:15.475695 2573 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 20:37:15.484492 ip-10-0-138-149 kubenswrapper[2573]: W1030 20:37:15.475697 2573 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 20:37:15.484492 ip-10-0-138-149 kubenswrapper[2573]: W1030 20:37:15.475702 2573 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 20:37:15.484492 ip-10-0-138-149 kubenswrapper[2573]: W1030 20:37:15.475705 2573 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 20:37:15.484492 ip-10-0-138-149 kubenswrapper[2573]: W1030 20:37:15.475707 2573 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 20:37:15.484492 ip-10-0-138-149 kubenswrapper[2573]: W1030 20:37:15.475710 2573 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 20:37:15.484492 ip-10-0-138-149 kubenswrapper[2573]: W1030 20:37:15.475713 2573 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 20:37:15.484492 ip-10-0-138-149 kubenswrapper[2573]: W1030 20:37:15.475715 2573 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 20:37:15.484492 ip-10-0-138-149 kubenswrapper[2573]: W1030 20:37:15.475719 2573 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 20:37:15.484492 ip-10-0-138-149 kubenswrapper[2573]: W1030 20:37:15.475721 2573 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 20:37:15.484492 ip-10-0-138-149 kubenswrapper[2573]: W1030 20:37:15.475724 2573 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 20:37:15.484492 ip-10-0-138-149 kubenswrapper[2573]: W1030 20:37:15.475726 2573 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 20:37:15.484492 ip-10-0-138-149 kubenswrapper[2573]: W1030 20:37:15.475729 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 20:37:15.484492 ip-10-0-138-149 kubenswrapper[2573]: W1030 20:37:15.475731 2573 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 20:37:15.484492 ip-10-0-138-149 kubenswrapper[2573]: W1030 20:37:15.475734 2573 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 20:37:15.484961 ip-10-0-138-149 kubenswrapper[2573]: W1030 20:37:15.475736 2573 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 20:37:15.484961 ip-10-0-138-149 kubenswrapper[2573]: W1030 20:37:15.475741 2573 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 20:37:15.484961 ip-10-0-138-149 kubenswrapper[2573]: W1030 20:37:15.475743 2573 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 20:37:15.484961 ip-10-0-138-149 kubenswrapper[2573]: W1030 20:37:15.475746 2573 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 20:37:15.484961 ip-10-0-138-149 kubenswrapper[2573]: W1030 20:37:15.475749 2573 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 20:37:15.484961 ip-10-0-138-149 kubenswrapper[2573]: W1030 20:37:15.475751 2573 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 20:37:15.484961 ip-10-0-138-149 kubenswrapper[2573]: W1030 20:37:15.475754 2573 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 20:37:15.484961 ip-10-0-138-149 kubenswrapper[2573]: W1030 20:37:15.475757 2573 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 20:37:15.484961 ip-10-0-138-149 kubenswrapper[2573]: W1030 20:37:15.475759 2573 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 20:37:15.484961 ip-10-0-138-149 kubenswrapper[2573]: W1030 20:37:15.475762 2573 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 20:37:15.484961 ip-10-0-138-149 kubenswrapper[2573]: W1030 20:37:15.475764 2573 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 20:37:15.484961 ip-10-0-138-149 kubenswrapper[2573]: W1030 20:37:15.475767 2573 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 20:37:15.484961 ip-10-0-138-149 kubenswrapper[2573]: W1030 20:37:15.475770 2573 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 20:37:15.484961 ip-10-0-138-149 kubenswrapper[2573]: W1030 20:37:15.475773 2573 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 20:37:15.484961 ip-10-0-138-149 kubenswrapper[2573]: W1030 20:37:15.475776 2573 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 20:37:15.484961 ip-10-0-138-149 kubenswrapper[2573]: W1030 20:37:15.475779 2573 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 20:37:15.484961 ip-10-0-138-149 kubenswrapper[2573]: W1030 20:37:15.475782 2573 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 20:37:15.484961 ip-10-0-138-149 kubenswrapper[2573]: W1030 20:37:15.475784 2573 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 20:37:15.484961 ip-10-0-138-149 kubenswrapper[2573]: W1030 20:37:15.475787 2573 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 20:37:15.484961 ip-10-0-138-149 kubenswrapper[2573]: W1030 20:37:15.475791 2573 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 20:37:15.485467 ip-10-0-138-149 kubenswrapper[2573]: W1030 20:37:15.475793 2573 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 20:37:15.485467 ip-10-0-138-149 kubenswrapper[2573]: W1030 20:37:15.475796 2573 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 20:37:15.485467 ip-10-0-138-149 kubenswrapper[2573]: W1030 20:37:15.475798 2573 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 20:37:15.485467 ip-10-0-138-149 kubenswrapper[2573]: W1030 20:37:15.475801 2573 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 20:37:15.485467 ip-10-0-138-149 kubenswrapper[2573]: W1030 20:37:15.475804 2573 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 20:37:15.485467 ip-10-0-138-149 kubenswrapper[2573]: W1030 20:37:15.475806 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 20:37:15.485467 ip-10-0-138-149 kubenswrapper[2573]: W1030 20:37:15.475809 2573 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 20:37:15.485467 ip-10-0-138-149 kubenswrapper[2573]: W1030 20:37:15.475811 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 20:37:15.485467 ip-10-0-138-149 kubenswrapper[2573]: W1030 20:37:15.475814 2573 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 20:37:15.485467 ip-10-0-138-149 kubenswrapper[2573]: W1030 20:37:15.475816 2573 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 20:37:15.485467 ip-10-0-138-149 kubenswrapper[2573]: W1030 20:37:15.475819 2573 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 20:37:15.485467 ip-10-0-138-149 kubenswrapper[2573]: W1030 20:37:15.475822 2573 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 20:37:15.485467 ip-10-0-138-149 kubenswrapper[2573]: W1030 20:37:15.475824 2573 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 20:37:15.485467 ip-10-0-138-149 kubenswrapper[2573]: W1030 20:37:15.475828 2573 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 20:37:15.485467 ip-10-0-138-149 kubenswrapper[2573]: W1030 20:37:15.475831 2573 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 20:37:15.485467 ip-10-0-138-149 kubenswrapper[2573]: W1030 20:37:15.475834 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 20:37:15.485467 ip-10-0-138-149 kubenswrapper[2573]: W1030 20:37:15.475836 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 20:37:15.485467 ip-10-0-138-149 kubenswrapper[2573]: W1030 20:37:15.475839 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 20:37:15.485467 ip-10-0-138-149 kubenswrapper[2573]: W1030 20:37:15.475842 2573 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 20:37:15.485467 ip-10-0-138-149 kubenswrapper[2573]: W1030 20:37:15.475845 2573 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 20:37:15.485950 ip-10-0-138-149 kubenswrapper[2573]: W1030 20:37:15.475847 2573 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 20:37:15.485950 ip-10-0-138-149 kubenswrapper[2573]: W1030 20:37:15.475850 2573 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 20:37:15.485950 ip-10-0-138-149 kubenswrapper[2573]: W1030 20:37:15.475853 2573 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 20:37:15.485950 ip-10-0-138-149 kubenswrapper[2573]: W1030 20:37:15.475857 2573 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 20:37:15.485950 ip-10-0-138-149 kubenswrapper[2573]: W1030 20:37:15.475860 2573 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 20:37:15.485950 ip-10-0-138-149 kubenswrapper[2573]: W1030 20:37:15.475864 2573 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 20:37:15.485950 ip-10-0-138-149 kubenswrapper[2573]: W1030 20:37:15.475867 2573 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 20:37:15.485950 ip-10-0-138-149 kubenswrapper[2573]: W1030 20:37:15.475869 2573 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 20:37:15.485950 ip-10-0-138-149 kubenswrapper[2573]: W1030 20:37:15.475873 2573 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 20:37:15.485950 ip-10-0-138-149 kubenswrapper[2573]: W1030 20:37:15.475875 2573 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 20:37:15.485950 ip-10-0-138-149 kubenswrapper[2573]: W1030 20:37:15.475878 2573 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 20:37:15.485950 ip-10-0-138-149 kubenswrapper[2573]: W1030 20:37:15.475882 2573 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 20:37:15.485950 ip-10-0-138-149 kubenswrapper[2573]: W1030 20:37:15.475884 2573 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 20:37:15.485950 ip-10-0-138-149 kubenswrapper[2573]: W1030 20:37:15.475887 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 20:37:15.485950 ip-10-0-138-149 kubenswrapper[2573]: W1030 20:37:15.475890 2573 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 20:37:15.485950 ip-10-0-138-149 kubenswrapper[2573]: I1030 20:37:15.475947 2573 server.go:957] "Client rotation is on, will bootstrap in background" Oct 30 20:37:15.485950 ip-10-0-138-149 kubenswrapper[2573]: I1030 20:37:15.478119 2573 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir" Oct 30 20:37:15.485950 ip-10-0-138-149 kubenswrapper[2573]: I1030 20:37:15.479133 2573 server.go:1014] "Starting client certificate rotation" Oct 30 20:37:15.485950 ip-10-0-138-149 kubenswrapper[2573]: I1030 20:37:15.479223 2573 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Oct 30 20:37:15.486391 ip-10-0-138-149 kubenswrapper[2573]: I1030 20:37:15.479263 2573 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet" Oct 30 20:37:15.486391 ip-10-0-138-149 kubenswrapper[2573]: I1030 20:37:15.481232 2573 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Oct 30 20:37:15.486391 ip-10-0-138-149 kubenswrapper[2573]: I1030 20:37:15.482493 2573 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Oct 30 20:37:15.495992 ip-10-0-138-149 kubenswrapper[2573]: I1030 20:37:15.495975 2573 log.go:25] "Validated CRI v1 runtime API" Oct 30 20:37:15.499874 ip-10-0-138-149 kubenswrapper[2573]: I1030 20:37:15.499860 2573 log.go:25] "Validated CRI v1 image API" Oct 30 20:37:15.500718 ip-10-0-138-149 kubenswrapper[2573]: I1030 20:37:15.500698 2573 server.go:1424] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Oct 30 20:37:15.504860 ip-10-0-138-149 kubenswrapper[2573]: I1030 20:37:15.504840 2573 fs.go:135] Filesystem UUIDs: map[285ee138-af8f-418c-af93-c68f9ce708c2:/dev/nvme0n1p3 7B77-95E7:/dev/nvme0n1p2 8c3538b6-0559-401f-b1e6-c7c0dbc0ba6b:/dev/nvme0n1p4] Oct 30 20:37:15.504929 ip-10-0-138-149 kubenswrapper[2573]: I1030 20:37:15.504858 2573 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:24 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:26 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:32 fsType:tmpfs blockSize:0} composefs_0-34:{mountpoint:/ major:0 minor:34 fsType:overlay blockSize:0}] Oct 30 20:37:15.510802 ip-10-0-138-149 kubenswrapper[2573]: I1030 20:37:15.510690 2573 manager.go:217] Machine: {Timestamp:2026-10-30 20:37:15.509257367 +0000 UTC m=+0.321708665 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3096819 MemoryCapacity:33164341248 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2c7df77bcd86752b2f17ccecc031b6 SystemUUID:ec2c7df7-7bcd-8675-2b2f-17ccecc031b6 BootID:e1c502c8-2a55-4c24-b41a-b750355263d4 Filesystems:[{Device:/dev/shm DeviceMajor:0 DeviceMinor:24 Capacity:16582168576 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:26 Capacity:6632869888 Type:vfs Inodes:819200 HasInodes:true} {Device:composefs_0-34 DeviceMajor:0 DeviceMinor:34 Capacity:6422528 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:32 Capacity:16582172672 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:3f:92:bd:4e:f1 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:3f:92:bd:4e:f1 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:da:25:e1:58:1d:73 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164341248 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Oct 30 20:37:15.510802 ip-10-0-138-149 kubenswrapper[2573]: I1030 20:37:15.510800 2573 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Oct 30 20:37:15.510917 ip-10-0-138-149 kubenswrapper[2573]: I1030 20:37:15.510879 2573 manager.go:233] Version: {KernelVersion:5.14.0-687.60.1.el9_8.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.8.20261023-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Oct 30 20:37:15.515784 ip-10-0-138-149 kubenswrapper[2573]: I1030 20:37:15.515623 2573 container_manager_linux.go:275] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Oct 30 20:37:15.515927 ip-10-0-138-149 kubenswrapper[2573]: I1030 20:37:15.515788 2573 container_manager_linux.go:280] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-138-149.us-west-1.compute.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"/system.slice","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{},"system-reserved-compressible":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Oct 30 20:37:15.515972 ip-10-0-138-149 kubenswrapper[2573]: I1030 20:37:15.515929 2573 topology_manager.go:143] "Creating topology manager with none policy" Oct 30 20:37:15.515972 ip-10-0-138-149 kubenswrapper[2573]: I1030 20:37:15.515937 2573 container_manager_linux.go:311] "Creating device plugin manager" Oct 30 20:37:15.516038 ip-10-0-138-149 kubenswrapper[2573]: I1030 20:37:15.515986 2573 manager.go:144] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Oct 30 20:37:15.516038 ip-10-0-138-149 kubenswrapper[2573]: I1030 20:37:15.516005 2573 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Oct 30 20:37:15.517105 ip-10-0-138-149 kubenswrapper[2573]: I1030 20:37:15.517095 2573 container_manager_linux.go:320] "Creating Dynamic Resource Allocation (DRA) manager" Oct 30 20:37:15.521096 ip-10-0-138-149 kubenswrapper[2573]: I1030 20:37:15.521084 2573 state_mem.go:41] "Initialized" logger="CPUManager state memory" Oct 30 20:37:15.521207 ip-10-0-138-149 kubenswrapper[2573]: I1030 20:37:15.521198 2573 server.go:1235] "Using root directory" path="/var/lib/kubelet" Oct 30 20:37:15.522043 ip-10-0-138-149 kubenswrapper[2573]: I1030 20:37:15.522035 2573 kubelet.go:493] "Attempting to sync node with API server" Oct 30 20:37:15.522080 ip-10-0-138-149 kubenswrapper[2573]: I1030 20:37:15.522047 2573 kubelet.go:394] "Adding static pod path" path="/etc/kubernetes/manifests" Oct 30 20:37:15.522080 ip-10-0-138-149 kubenswrapper[2573]: I1030 20:37:15.522057 2573 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Oct 30 20:37:15.522080 ip-10-0-138-149 kubenswrapper[2573]: I1030 20:37:15.522067 2573 kubelet.go:405] "Adding apiserver pod source" Oct 30 20:37:15.522080 ip-10-0-138-149 kubenswrapper[2573]: I1030 20:37:15.522075 2573 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Oct 30 20:37:15.522437 ip-10-0-138-149 kubenswrapper[2573]: I1030 20:37:15.522426 2573 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Oct 30 20:37:15.524800 ip-10-0-138-149 kubenswrapper[2573]: I1030 20:37:15.524781 2573 kuberuntime_manager.go:304] "Container runtime initialized" containerRuntime="cri-o" version="1.35.10-3.rhaos4.22.git9383502.el9" apiVersion="v1" Oct 30 20:37:15.531430 ip-10-0-138-149 kubenswrapper[2573]: I1030 20:37:15.531414 2573 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Oct 30 20:37:15.532360 ip-10-0-138-149 kubenswrapper[2573]: I1030 20:37:15.532346 2573 kubelet.go:963] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Oct 30 20:37:15.532434 ip-10-0-138-149 kubenswrapper[2573]: I1030 20:37:15.532386 2573 kubelet.go:990] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Oct 30 20:37:15.532592 ip-10-0-138-149 kubenswrapper[2573]: I1030 20:37:15.532580 2573 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Oct 30 20:37:15.532651 ip-10-0-138-149 kubenswrapper[2573]: I1030 20:37:15.532600 2573 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Oct 30 20:37:15.532651 ip-10-0-138-149 kubenswrapper[2573]: I1030 20:37:15.532609 2573 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Oct 30 20:37:15.532651 ip-10-0-138-149 kubenswrapper[2573]: I1030 20:37:15.532620 2573 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Oct 30 20:37:15.532651 ip-10-0-138-149 kubenswrapper[2573]: I1030 20:37:15.532629 2573 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Oct 30 20:37:15.532651 ip-10-0-138-149 kubenswrapper[2573]: I1030 20:37:15.532638 2573 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Oct 30 20:37:15.533286 ip-10-0-138-149 kubenswrapper[2573]: I1030 20:37:15.533276 2573 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Oct 30 20:37:15.533332 ip-10-0-138-149 kubenswrapper[2573]: I1030 20:37:15.533290 2573 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Oct 30 20:37:15.533332 ip-10-0-138-149 kubenswrapper[2573]: I1030 20:37:15.533301 2573 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Oct 30 20:37:15.533332 ip-10-0-138-149 kubenswrapper[2573]: I1030 20:37:15.533312 2573 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Oct 30 20:37:15.533332 ip-10-0-138-149 kubenswrapper[2573]: I1030 20:37:15.533325 2573 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Oct 30 20:37:15.533470 ip-10-0-138-149 kubenswrapper[2573]: I1030 20:37:15.533340 2573 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Oct 30 20:37:15.534746 ip-10-0-138-149 kubenswrapper[2573]: I1030 20:37:15.534736 2573 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Oct 30 20:37:15.534810 ip-10-0-138-149 kubenswrapper[2573]: I1030 20:37:15.534751 2573 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Oct 30 20:37:15.538580 ip-10-0-138-149 kubenswrapper[2573]: I1030 20:37:15.538550 2573 server.go:1263] "Started kubelet" Oct 30 20:37:15.538686 ip-10-0-138-149 kubenswrapper[2573]: I1030 20:37:15.538638 2573 server.go:182] "Starting to listen" address="0.0.0.0" port=10250 Oct 30 20:37:15.538742 ip-10-0-138-149 kubenswrapper[2573]: I1030 20:37:15.538647 2573 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Oct 30 20:37:15.538742 ip-10-0-138-149 kubenswrapper[2573]: I1030 20:37:15.538716 2573 server_v1.go:49] "podresources" method="list" useActivePods=true Oct 30 20:37:15.539419 ip-10-0-138-149 systemd[1]: Started Kubernetes Kubelet. Oct 30 20:37:15.541262 ip-10-0-138-149 kubenswrapper[2573]: I1030 20:37:15.541246 2573 server.go:254] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Oct 30 20:37:15.541929 ip-10-0-138-149 kubenswrapper[2573]: I1030 20:37:15.541914 2573 server.go:317] "Adding debug handlers to kubelet server" Oct 30 20:37:15.544994 ip-10-0-138-149 kubenswrapper[2573]: E1030 20:37:15.544970 2573 kubelet.go:1679] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache" Oct 30 20:37:15.547212 ip-10-0-138-149 kubenswrapper[2573]: I1030 20:37:15.547197 2573 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Oct 30 20:37:15.548507 ip-10-0-138-149 kubenswrapper[2573]: I1030 20:37:15.548493 2573 fs_resource_analyzer.go:69] "Starting FS ResourceAnalyzer" Oct 30 20:37:15.548707 ip-10-0-138-149 kubenswrapper[2573]: E1030 20:37:15.548693 2573 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-138-149.us-west-1.compute.internal\" not found" Oct 30 20:37:15.549332 ip-10-0-138-149 kubenswrapper[2573]: I1030 20:37:15.549313 2573 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Oct 30 20:37:15.549494 ip-10-0-138-149 kubenswrapper[2573]: I1030 20:37:15.549319 2573 volume_manager.go:309] "The desired_state_of_world populator starts" Oct 30 20:37:15.549494 ip-10-0-138-149 kubenswrapper[2573]: I1030 20:37:15.549489 2573 volume_manager.go:311] "Starting Kubelet Volume Manager" Oct 30 20:37:15.549658 ip-10-0-138-149 kubenswrapper[2573]: I1030 20:37:15.549438 2573 factory.go:55] Registering systemd factory Oct 30 20:37:15.549658 ip-10-0-138-149 kubenswrapper[2573]: I1030 20:37:15.549527 2573 factory.go:223] Registration of the systemd container factory successfully Oct 30 20:37:15.549658 ip-10-0-138-149 kubenswrapper[2573]: I1030 20:37:15.549579 2573 reconstruct.go:97] "Volume reconstruction finished" Oct 30 20:37:15.549658 ip-10-0-138-149 kubenswrapper[2573]: I1030 20:37:15.549590 2573 reconciler.go:29] "Reconciler: start to sync state" Oct 30 20:37:15.549836 ip-10-0-138-149 kubenswrapper[2573]: I1030 20:37:15.549823 2573 factory.go:153] Registering CRI-O factory Oct 30 20:37:15.549868 ip-10-0-138-149 kubenswrapper[2573]: I1030 20:37:15.549841 2573 factory.go:223] Registration of the crio container factory successfully Oct 30 20:37:15.549895 ip-10-0-138-149 kubenswrapper[2573]: I1030 20:37:15.549889 2573 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Oct 30 20:37:15.549924 ip-10-0-138-149 kubenswrapper[2573]: I1030 20:37:15.549912 2573 factory.go:103] Registering Raw factory Oct 30 20:37:15.549951 ip-10-0-138-149 kubenswrapper[2573]: I1030 20:37:15.549928 2573 manager.go:1199] Started watching for new ooms in manager Oct 30 20:37:15.551054 ip-10-0-138-149 kubenswrapper[2573]: I1030 20:37:15.551039 2573 manager.go:319] Starting recovery of all containers Oct 30 20:37:15.556687 ip-10-0-138-149 kubenswrapper[2573]: I1030 20:37:15.556649 2573 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Oct 30 20:37:15.560521 ip-10-0-138-149 kubenswrapper[2573]: I1030 20:37:15.560506 2573 manager.go:324] Recovery completed Oct 30 20:37:15.564850 ip-10-0-138-149 kubenswrapper[2573]: I1030 20:37:15.564834 2573 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 20:37:15.566827 ip-10-0-138-149 kubenswrapper[2573]: I1030 20:37:15.566811 2573 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-138-149.us-west-1.compute.internal" event="NodeHasSufficientMemory" Oct 30 20:37:15.566910 ip-10-0-138-149 kubenswrapper[2573]: I1030 20:37:15.566838 2573 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-138-149.us-west-1.compute.internal" event="NodeHasNoDiskPressure" Oct 30 20:37:15.566910 ip-10-0-138-149 kubenswrapper[2573]: I1030 20:37:15.566848 2573 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-138-149.us-west-1.compute.internal" event="NodeHasSufficientPID" Oct 30 20:37:15.567366 ip-10-0-138-149 kubenswrapper[2573]: I1030 20:37:15.567354 2573 cpu_manager.go:226] "Starting" policy="none" Oct 30 20:37:15.567440 ip-10-0-138-149 kubenswrapper[2573]: I1030 20:37:15.567366 2573 cpu_manager.go:227] "Reconciling" reconcilePeriod="10s" Oct 30 20:37:15.567440 ip-10-0-138-149 kubenswrapper[2573]: I1030 20:37:15.567358 2573 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 20:37:15.567440 ip-10-0-138-149 kubenswrapper[2573]: I1030 20:37:15.567383 2573 state_mem.go:41] "Initialized" logger="CPUManager state checkpoint.CPUManager state memory" Oct 30 20:37:15.568244 ip-10-0-138-149 kubenswrapper[2573]: I1030 20:37:15.568228 2573 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-138-149.us-west-1.compute.internal" event="NodeHasSufficientMemory" Oct 30 20:37:15.568297 ip-10-0-138-149 kubenswrapper[2573]: I1030 20:37:15.568254 2573 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-138-149.us-west-1.compute.internal" event="NodeHasNoDiskPressure" Oct 30 20:37:15.568297 ip-10-0-138-149 kubenswrapper[2573]: I1030 20:37:15.568264 2573 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-138-149.us-west-1.compute.internal" event="NodeHasSufficientPID" Oct 30 20:37:15.569613 ip-10-0-138-149 kubenswrapper[2573]: I1030 20:37:15.569598 2573 policy_none.go:50] "Start" Oct 30 20:37:15.569689 ip-10-0-138-149 kubenswrapper[2573]: I1030 20:37:15.569616 2573 memory_manager.go:187] "Starting memorymanager" policy="None" Oct 30 20:37:15.569689 ip-10-0-138-149 kubenswrapper[2573]: I1030 20:37:15.569629 2573 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Oct 30 20:37:15.571702 ip-10-0-138-149 kubenswrapper[2573]: I1030 20:37:15.571685 2573 policy_none.go:44] "Start" Oct 30 20:37:15.719948 ip-10-0-138-149 kubenswrapper[2573]: I1030 20:37:15.605793 2573 node_container_manager_linux.go:130] "Enforcing system reserved compressible on cgroup" cgroupName="/system.slice" limits={"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"} Oct 30 20:37:15.719948 ip-10-0-138-149 kubenswrapper[2573]: I1030 20:37:15.609076 2573 manager.go:341] "Starting Device Plugin manager" Oct 30 20:37:15.719948 ip-10-0-138-149 kubenswrapper[2573]: E1030 20:37:15.609103 2573 manager.go:525] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Oct 30 20:37:15.719948 ip-10-0-138-149 kubenswrapper[2573]: I1030 20:37:15.609114 2573 server.go:87] "Starting device plugin registration server" Oct 30 20:37:15.719948 ip-10-0-138-149 kubenswrapper[2573]: I1030 20:37:15.609308 2573 eviction_manager.go:194] "Eviction manager: starting control loop" Oct 30 20:37:15.719948 ip-10-0-138-149 kubenswrapper[2573]: I1030 20:37:15.609317 2573 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Oct 30 20:37:15.719948 ip-10-0-138-149 kubenswrapper[2573]: I1030 20:37:15.609434 2573 plugin_watcher.go:53] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Oct 30 20:37:15.719948 ip-10-0-138-149 kubenswrapper[2573]: I1030 20:37:15.609515 2573 plugin_manager.go:119] "The desired_state_of_world populator (plugin watcher) starts" Oct 30 20:37:15.719948 ip-10-0-138-149 kubenswrapper[2573]: I1030 20:37:15.609523 2573 plugin_manager.go:121] "Starting Kubelet Plugin Manager" Oct 30 20:37:15.719948 ip-10-0-138-149 kubenswrapper[2573]: E1030 20:37:15.610025 2573 eviction_manager.go:272] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Oct 30 20:37:15.719948 ip-10-0-138-149 kubenswrapper[2573]: E1030 20:37:15.610076 2573 eviction_manager.go:297] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-138-149.us-west-1.compute.internal\" not found" Oct 30 20:37:15.719948 ip-10-0-138-149 kubenswrapper[2573]: I1030 20:37:15.681458 2573 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-138-149.us-west-1.compute.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Oct 30 20:37:15.719948 ip-10-0-138-149 kubenswrapper[2573]: E1030 20:37:15.681470 2573 controller.go:201] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-138-149.us-west-1.compute.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms" Oct 30 20:37:15.719948 ip-10-0-138-149 kubenswrapper[2573]: E1030 20:37:15.681470 2573 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-138-149.us-west-1.compute.internal.18e36919a813384c default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-138-149.us-west-1.compute.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-138-149.us-west-1.compute.internal,},FirstTimestamp:2026-10-30 20:37:15.53852014 +0000 UTC m=+0.350971439,LastTimestamp:2026-10-30 20:37:15.53852014 +0000 UTC m=+0.350971439,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-138-149.us-west-1.compute.internal,}" Oct 30 20:37:15.719948 ip-10-0-138-149 kubenswrapper[2573]: I1030 20:37:15.709728 2573 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 20:37:15.719948 ip-10-0-138-149 kubenswrapper[2573]: I1030 20:37:15.710635 2573 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-138-149.us-west-1.compute.internal" event="NodeHasSufficientMemory" Oct 30 20:37:15.719948 ip-10-0-138-149 kubenswrapper[2573]: I1030 20:37:15.710667 2573 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-138-149.us-west-1.compute.internal" event="NodeHasNoDiskPressure" Oct 30 20:37:15.719948 ip-10-0-138-149 kubenswrapper[2573]: I1030 20:37:15.710682 2573 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-138-149.us-west-1.compute.internal" event="NodeHasSufficientPID" Oct 30 20:37:15.719948 ip-10-0-138-149 kubenswrapper[2573]: I1030 20:37:15.710714 2573 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-138-149.us-west-1.compute.internal" Oct 30 20:37:15.728629 ip-10-0-138-149 kubenswrapper[2573]: I1030 20:37:15.728608 2573 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Oct 30 20:37:15.728629 ip-10-0-138-149 kubenswrapper[2573]: I1030 20:37:15.728632 2573 status_manager.go:249] "Starting to sync pod status with apiserver" Oct 30 20:37:15.728789 ip-10-0-138-149 kubenswrapper[2573]: I1030 20:37:15.728655 2573 kubelet.go:2521] "Starting kubelet main sync loop" Oct 30 20:37:15.728789 ip-10-0-138-149 kubenswrapper[2573]: E1030 20:37:15.728692 2573 kubelet.go:2545] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful" Oct 30 20:37:15.800315 ip-10-0-138-149 kubenswrapper[2573]: I1030 20:37:15.800296 2573 reflector.go:446] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Oct 30 20:37:15.800427 ip-10-0-138-149 kubenswrapper[2573]: I1030 20:37:15.800335 2573 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-zhsrp" Oct 30 20:37:15.828928 ip-10-0-138-149 kubenswrapper[2573]: I1030 20:37:15.828908 2573 kubelet.go:2608] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-149.us-west-1.compute.internal","kube-system/kube-apiserver-proxy-ip-10-0-138-149.us-west-1.compute.internal"] Oct 30 20:37:15.828985 ip-10-0-138-149 kubenswrapper[2573]: I1030 20:37:15.828962 2573 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 20:37:15.829808 ip-10-0-138-149 kubenswrapper[2573]: I1030 20:37:15.829791 2573 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-138-149.us-west-1.compute.internal" event="NodeHasSufficientMemory" Oct 30 20:37:15.829900 ip-10-0-138-149 kubenswrapper[2573]: I1030 20:37:15.829825 2573 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-138-149.us-west-1.compute.internal" event="NodeHasNoDiskPressure" Oct 30 20:37:15.829900 ip-10-0-138-149 kubenswrapper[2573]: I1030 20:37:15.829839 2573 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-138-149.us-west-1.compute.internal" event="NodeHasSufficientPID" Oct 30 20:37:15.832191 ip-10-0-138-149 kubenswrapper[2573]: I1030 20:37:15.832173 2573 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 20:37:15.832325 ip-10-0-138-149 kubenswrapper[2573]: I1030 20:37:15.832311 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-149.us-west-1.compute.internal" Oct 30 20:37:15.832370 ip-10-0-138-149 kubenswrapper[2573]: I1030 20:37:15.832340 2573 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 20:37:15.832900 ip-10-0-138-149 kubenswrapper[2573]: I1030 20:37:15.832886 2573 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-138-149.us-west-1.compute.internal" event="NodeHasSufficientMemory" Oct 30 20:37:15.832963 ip-10-0-138-149 kubenswrapper[2573]: I1030 20:37:15.832925 2573 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-138-149.us-west-1.compute.internal" event="NodeHasNoDiskPressure" Oct 30 20:37:15.832963 ip-10-0-138-149 kubenswrapper[2573]: I1030 20:37:15.832935 2573 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-138-149.us-west-1.compute.internal" event="NodeHasSufficientPID" Oct 30 20:37:15.833025 ip-10-0-138-149 kubenswrapper[2573]: I1030 20:37:15.832887 2573 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-138-149.us-west-1.compute.internal" event="NodeHasSufficientMemory" Oct 30 20:37:15.833025 ip-10-0-138-149 kubenswrapper[2573]: I1030 20:37:15.832989 2573 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-138-149.us-west-1.compute.internal" event="NodeHasNoDiskPressure" Oct 30 20:37:15.833025 ip-10-0-138-149 kubenswrapper[2573]: I1030 20:37:15.833001 2573 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-138-149.us-west-1.compute.internal" event="NodeHasSufficientPID" Oct 30 20:37:15.835138 ip-10-0-138-149 kubenswrapper[2573]: I1030 20:37:15.835122 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-138-149.us-west-1.compute.internal" Oct 30 20:37:15.835236 ip-10-0-138-149 kubenswrapper[2573]: I1030 20:37:15.835158 2573 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 20:37:15.835848 ip-10-0-138-149 kubenswrapper[2573]: I1030 20:37:15.835828 2573 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-138-149.us-west-1.compute.internal" event="NodeHasSufficientMemory" Oct 30 20:37:15.835945 ip-10-0-138-149 kubenswrapper[2573]: I1030 20:37:15.835857 2573 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-138-149.us-west-1.compute.internal" event="NodeHasNoDiskPressure" Oct 30 20:37:15.835945 ip-10-0-138-149 kubenswrapper[2573]: I1030 20:37:15.835866 2573 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-138-149.us-west-1.compute.internal" event="NodeHasSufficientPID" Oct 30 20:37:15.839171 ip-10-0-138-149 kubenswrapper[2573]: E1030 20:37:15.839019 2573 reflector.go:204] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-138-149.us-west-1.compute.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.Node" Oct 30 20:37:15.839171 ip-10-0-138-149 kubenswrapper[2573]: E1030 20:37:15.839118 2573 reflector.go:204] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.Service" Oct 30 20:37:15.839171 ip-10-0-138-149 kubenswrapper[2573]: E1030 20:37:15.839063 2573 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-138-149.us-west-1.compute.internal.18e36919a9c31ef9 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-138-149.us-west-1.compute.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-138-149.us-west-1.compute.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-138-149.us-west-1.compute.internal,},FirstTimestamp:2026-10-30 20:37:15.566825209 +0000 UTC m=+0.379276507,LastTimestamp:2026-10-30 20:37:15.566825209 +0000 UTC m=+0.379276507,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-138-149.us-west-1.compute.internal,}" Oct 30 20:37:15.839339 ip-10-0-138-149 kubenswrapper[2573]: E1030 20:37:15.839148 2573 reflector.go:204] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.CSIDriver" Oct 30 20:37:15.851879 ip-10-0-138-149 kubenswrapper[2573]: I1030 20:37:15.851858 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/60d05e7d5fab914587e076383d73f1df-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-138-149.us-west-1.compute.internal\" (UID: \"60d05e7d5fab914587e076383d73f1df\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-149.us-west-1.compute.internal" Oct 30 20:37:15.851929 ip-10-0-138-149 kubenswrapper[2573]: I1030 20:37:15.851885 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/60d05e7d5fab914587e076383d73f1df-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-138-149.us-west-1.compute.internal\" (UID: \"60d05e7d5fab914587e076383d73f1df\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-149.us-west-1.compute.internal" Oct 30 20:37:15.851929 ip-10-0-138-149 kubenswrapper[2573]: I1030 20:37:15.851901 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/0d2e0d42ed810f19933d4cd655966ed4-config\") pod \"kube-apiserver-proxy-ip-10-0-138-149.us-west-1.compute.internal\" (UID: \"0d2e0d42ed810f19933d4cd655966ed4\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-138-149.us-west-1.compute.internal" Oct 30 20:37:15.863187 ip-10-0-138-149 kubenswrapper[2573]: E1030 20:37:15.863170 2573 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-138-149.us-west-1.compute.internal\" not found" node="ip-10-0-138-149.us-west-1.compute.internal" Oct 30 20:37:15.865607 ip-10-0-138-149 kubenswrapper[2573]: E1030 20:37:15.865592 2573 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-138-149.us-west-1.compute.internal\" not found" node="ip-10-0-138-149.us-west-1.compute.internal" Oct 30 20:37:15.952613 ip-10-0-138-149 kubenswrapper[2573]: I1030 20:37:15.952585 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/60d05e7d5fab914587e076383d73f1df-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-138-149.us-west-1.compute.internal\" (UID: \"60d05e7d5fab914587e076383d73f1df\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-149.us-west-1.compute.internal" Oct 30 20:37:15.952613 ip-10-0-138-149 kubenswrapper[2573]: I1030 20:37:15.952614 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/60d05e7d5fab914587e076383d73f1df-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-138-149.us-west-1.compute.internal\" (UID: \"60d05e7d5fab914587e076383d73f1df\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-149.us-west-1.compute.internal" Oct 30 20:37:15.952787 ip-10-0-138-149 kubenswrapper[2573]: I1030 20:37:15.952632 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/0d2e0d42ed810f19933d4cd655966ed4-config\") pod \"kube-apiserver-proxy-ip-10-0-138-149.us-west-1.compute.internal\" (UID: \"0d2e0d42ed810f19933d4cd655966ed4\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-138-149.us-west-1.compute.internal" Oct 30 20:37:15.952787 ip-10-0-138-149 kubenswrapper[2573]: I1030 20:37:15.952670 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/60d05e7d5fab914587e076383d73f1df-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-138-149.us-west-1.compute.internal\" (UID: \"60d05e7d5fab914587e076383d73f1df\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-149.us-west-1.compute.internal" Oct 30 20:37:15.952787 ip-10-0-138-149 kubenswrapper[2573]: I1030 20:37:15.952691 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/0d2e0d42ed810f19933d4cd655966ed4-config\") pod \"kube-apiserver-proxy-ip-10-0-138-149.us-west-1.compute.internal\" (UID: \"0d2e0d42ed810f19933d4cd655966ed4\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-138-149.us-west-1.compute.internal" Oct 30 20:37:15.952787 ip-10-0-138-149 kubenswrapper[2573]: I1030 20:37:15.952694 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/60d05e7d5fab914587e076383d73f1df-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-138-149.us-west-1.compute.internal\" (UID: \"60d05e7d5fab914587e076383d73f1df\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-149.us-west-1.compute.internal" Oct 30 20:37:15.997481 ip-10-0-138-149 kubenswrapper[2573]: I1030 20:37:15.997436 2573 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 20:37:15.998060 ip-10-0-138-149 kubenswrapper[2573]: E1030 20:37:15.998044 2573 kubelet_node_status.go:115] "Unable to register node with API server, error getting existing node" err="nodes \"ip-10-0-138-149.us-west-1.compute.internal\" not found" node="ip-10-0-138-149.us-west-1.compute.internal" Oct 30 20:37:16.051346 ip-10-0-138-149 kubenswrapper[2573]: E1030 20:37:16.051328 2573 nodelease.go:50] "Failed to get node when trying to set owner ref to the node lease" err="nodes \"ip-10-0-138-149.us-west-1.compute.internal\" not found" node="ip-10-0-138-149.us-west-1.compute.internal" Oct 30 20:37:16.168743 ip-10-0-138-149 kubenswrapper[2573]: I1030 20:37:16.168723 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-149.us-west-1.compute.internal" Oct 30 20:37:16.170916 ip-10-0-138-149 kubenswrapper[2573]: I1030 20:37:16.170902 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-138-149.us-west-1.compute.internal" Oct 30 20:37:16.198733 ip-10-0-138-149 kubenswrapper[2573]: I1030 20:37:16.198715 2573 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 20:37:16.199538 ip-10-0-138-149 kubenswrapper[2573]: I1030 20:37:16.199523 2573 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-138-149.us-west-1.compute.internal" event="NodeHasSufficientMemory" Oct 30 20:37:16.199625 ip-10-0-138-149 kubenswrapper[2573]: I1030 20:37:16.199553 2573 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-138-149.us-west-1.compute.internal" event="NodeHasNoDiskPressure" Oct 30 20:37:16.199625 ip-10-0-138-149 kubenswrapper[2573]: I1030 20:37:16.199582 2573 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-138-149.us-west-1.compute.internal" event="NodeHasSufficientPID" Oct 30 20:37:16.199625 ip-10-0-138-149 kubenswrapper[2573]: I1030 20:37:16.199612 2573 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-138-149.us-west-1.compute.internal" Oct 30 20:37:16.258385 ip-10-0-138-149 kubenswrapper[2573]: I1030 20:37:16.258339 2573 kubelet_node_status.go:80] "Successfully registered node" node="ip-10-0-138-149.us-west-1.compute.internal" Oct 30 20:37:16.258385 ip-10-0-138-149 kubenswrapper[2573]: E1030 20:37:16.258359 2573 kubelet_node_status.go:524] "Error updating node status, will retry" err="error getting node \"ip-10-0-138-149.us-west-1.compute.internal\": node \"ip-10-0-138-149.us-west-1.compute.internal\" not found" Oct 30 20:37:16.387263 ip-10-0-138-149 kubenswrapper[2573]: E1030 20:37:16.387229 2573 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-138-149.us-west-1.compute.internal\" not found" Oct 30 20:37:16.478448 ip-10-0-138-149 kubenswrapper[2573]: I1030 20:37:16.478423 2573 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials" Oct 30 20:37:16.478889 ip-10-0-138-149 kubenswrapper[2573]: I1030 20:37:16.478545 2573 reflector.go:578] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:161: Unexpected watch close - watch lasted less than a second and no items received" Oct 30 20:37:16.478889 ip-10-0-138-149 kubenswrapper[2573]: E1030 20:37:16.478622 2573 event.go:368] "Unable to write event (may retry after sleeping)" err="Patch \"https://a3138b275a5484ea19a8d9971322f756-b97bd68a7b677e68.elb.us-east-2.amazonaws.com:6443/api/v1/namespaces/default/events/ip-10-0-138-149.us-west-1.compute.internal.18e36919a9c31ef9\": read tcp 10.0.138.149:55034->13.184.201.241:6443: use of closed network connection" event="&Event{ObjectMeta:{ip-10-0-138-149.us-west-1.compute.internal.18e36919a9c31ef9 default 5125 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-138-149.us-west-1.compute.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-138-149.us-west-1.compute.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-138-149.us-west-1.compute.internal,},FirstTimestamp:2026-10-30 20:37:15 +0000 UTC,LastTimestamp:2026-10-30 20:37:15.710647654 +0000 UTC m=+0.523098956,Count:3,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-138-149.us-west-1.compute.internal,}" Oct 30 20:37:16.487711 ip-10-0-138-149 kubenswrapper[2573]: E1030 20:37:16.487690 2573 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-138-149.us-west-1.compute.internal\" not found" Oct 30 20:37:16.588452 ip-10-0-138-149 kubenswrapper[2573]: E1030 20:37:16.588387 2573 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-138-149.us-west-1.compute.internal\" not found" Oct 30 20:37:16.689275 ip-10-0-138-149 kubenswrapper[2573]: E1030 20:37:16.689247 2573 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-138-149.us-west-1.compute.internal\" not found" Oct 30 20:37:16.791160 ip-10-0-138-149 kubenswrapper[2573]: E1030 20:37:16.791127 2573 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-138-149.us-west-1.compute.internal\" not found" Oct 30 20:37:16.806051 ip-10-0-138-149 kubenswrapper[2573]: I1030 20:37:16.806023 2573 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-10-29 20:32:16 +0000 UTC" deadline="2028-07-25 01:40:41.439719308 +0000 UTC" Oct 30 20:37:16.806051 ip-10-0-138-149 kubenswrapper[2573]: I1030 20:37:16.806046 2573 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="15197h3m24.633677226s" Oct 30 20:37:16.819133 ip-10-0-138-149 kubenswrapper[2573]: I1030 20:37:16.819111 2573 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 20:37:16.820125 ip-10-0-138-149 kubenswrapper[2573]: I1030 20:37:16.820109 2573 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-138-149.us-west-1.compute.internal" event="NodeHasSufficientMemory" Oct 30 20:37:16.820227 ip-10-0-138-149 kubenswrapper[2573]: I1030 20:37:16.820140 2573 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-138-149.us-west-1.compute.internal" event="NodeHasNoDiskPressure" Oct 30 20:37:16.820227 ip-10-0-138-149 kubenswrapper[2573]: I1030 20:37:16.820151 2573 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-138-149.us-west-1.compute.internal" event="NodeHasSufficientPID" Oct 30 20:37:16.892392 ip-10-0-138-149 kubenswrapper[2573]: E1030 20:37:16.892322 2573 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-138-149.us-west-1.compute.internal\" not found" Oct 30 20:37:16.993523 ip-10-0-138-149 kubenswrapper[2573]: E1030 20:37:16.993493 2573 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-138-149.us-west-1.compute.internal\" not found" Oct 30 20:37:17.059923 ip-10-0-138-149 kubenswrapper[2573]: I1030 20:37:17.059901 2573 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 20:37:17.095025 ip-10-0-138-149 kubenswrapper[2573]: E1030 20:37:17.094996 2573 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-138-149.us-west-1.compute.internal\" not found" Oct 30 20:37:17.180330 ip-10-0-138-149 kubenswrapper[2573]: I1030 20:37:17.180255 2573 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 20:37:17.249092 ip-10-0-138-149 kubenswrapper[2573]: I1030 20:37:17.249059 2573 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-149.us-west-1.compute.internal" Oct 30 20:37:17.323337 ip-10-0-138-149 kubenswrapper[2573]: I1030 20:37:17.323316 2573 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Oct 30 20:37:17.324387 ip-10-0-138-149 kubenswrapper[2573]: I1030 20:37:17.324371 2573 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-138-149.us-west-1.compute.internal" Oct 30 20:37:17.331120 ip-10-0-138-149 kubenswrapper[2573]: W1030 20:37:17.331094 2573 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0d2e0d42ed810f19933d4cd655966ed4.slice/crio-da4df417dfa91620c245546301b2dbee88814652f2989ebdddb126fbdfc28a98 WatchSource:0}: Error finding container da4df417dfa91620c245546301b2dbee88814652f2989ebdddb126fbdfc28a98: Status 404 returned error can't find the container with id da4df417dfa91620c245546301b2dbee88814652f2989ebdddb126fbdfc28a98 Oct 30 20:37:17.331596 ip-10-0-138-149 kubenswrapper[2573]: W1030 20:37:17.331579 2573 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod60d05e7d5fab914587e076383d73f1df.slice/crio-3ac9f12e7f98d1036fcc0c39f3aa3ab3c15fc903458208bef56f5c1969e7f9e0 WatchSource:0}: Error finding container 3ac9f12e7f98d1036fcc0c39f3aa3ab3c15fc903458208bef56f5c1969e7f9e0: Status 404 returned error can't find the container with id 3ac9f12e7f98d1036fcc0c39f3aa3ab3c15fc903458208bef56f5c1969e7f9e0 Oct 30 20:37:17.337416 ip-10-0-138-149 kubenswrapper[2573]: I1030 20:37:17.337400 2573 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 30 20:37:17.391977 ip-10-0-138-149 kubenswrapper[2573]: I1030 20:37:17.391954 2573 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Oct 30 20:37:17.424741 ip-10-0-138-149 kubenswrapper[2573]: I1030 20:37:17.424721 2573 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 20:37:17.523806 ip-10-0-138-149 kubenswrapper[2573]: I1030 20:37:17.523753 2573 apiserver.go:52] "Watching apiserver" Oct 30 20:37:17.549887 ip-10-0-138-149 kubenswrapper[2573]: I1030 20:37:17.549865 2573 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving" Oct 30 20:37:17.634703 ip-10-0-138-149 kubenswrapper[2573]: I1030 20:37:17.634683 2573 reflector.go:446] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Oct 30 20:37:17.635023 ip-10-0-138-149 kubenswrapper[2573]: I1030 20:37:17.635003 2573 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-multus/multus-additional-cni-plugins-blccx","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-svf4h","openshift-multus/multus-dzbhx","openshift-network-diagnostics/network-check-target-ws655","kube-system/konnectivity-agent-cjrkj","openshift-cluster-node-tuning-operator/tuned-w9npq","openshift-dns/node-resolver-nv2rk","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-149.us-west-1.compute.internal","openshift-multus/network-metrics-daemon-nhs9k","openshift-network-operator/iptables-alerter-kvq2l","openshift-ovn-kubernetes/ovnkube-node-7x9jg","kube-system/global-pull-secret-syncer-bcmwc","kube-system/kube-apiserver-proxy-ip-10-0-138-149.us-west-1.compute.internal","openshift-image-registry/node-ca-tlch5"] Oct 30 20:37:17.638842 ip-10-0-138-149 kubenswrapper[2573]: I1030 20:37:17.638827 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-nhs9k" Oct 30 20:37:17.638918 ip-10-0-138-149 kubenswrapper[2573]: E1030 20:37:17.638901 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-nhs9k" podUID="e886897a-c6a3-414f-875f-aa112541b608" Oct 30 20:37:17.640937 ip-10-0-138-149 kubenswrapper[2573]: I1030 20:37:17.640922 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-tlch5" Oct 30 20:37:17.643180 ip-10-0-138-149 kubenswrapper[2573]: I1030 20:37:17.643162 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-dzbhx" Oct 30 20:37:17.645448 ip-10-0-138-149 kubenswrapper[2573]: I1030 20:37:17.645431 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ws655" Oct 30 20:37:17.645521 ip-10-0-138-149 kubenswrapper[2573]: E1030 20:37:17.645493 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-ws655" podUID="567a72f6-035d-4c61-acaf-f46008029168" Oct 30 20:37:17.648920 ip-10-0-138-149 kubenswrapper[2573]: I1030 20:37:17.648896 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-7x9jg" Oct 30 20:37:17.651289 ip-10-0-138-149 kubenswrapper[2573]: I1030 20:37:17.651267 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-w9npq" Oct 30 20:37:17.653758 ip-10-0-138-149 kubenswrapper[2573]: I1030 20:37:17.653740 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-blccx" Oct 30 20:37:17.656043 ip-10-0-138-149 kubenswrapper[2573]: I1030 20:37:17.656025 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-nv2rk" Oct 30 20:37:17.658416 ip-10-0-138-149 kubenswrapper[2573]: I1030 20:37:17.658400 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-kvq2l" Oct 30 20:37:17.663239 ip-10-0-138-149 kubenswrapper[2573]: I1030 20:37:17.663213 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/c2ff359c-cf92-4a5b-8610-f6eb41acbe73-host-cni-netd\") pod \"ovnkube-node-7x9jg\" (UID: \"c2ff359c-cf92-4a5b-8610-f6eb41acbe73\") " pod="openshift-ovn-kubernetes/ovnkube-node-7x9jg" Oct 30 20:37:17.663344 ip-10-0-138-149 kubenswrapper[2573]: I1030 20:37:17.663253 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/c2ff359c-cf92-4a5b-8610-f6eb41acbe73-ovnkube-config\") pod \"ovnkube-node-7x9jg\" (UID: \"c2ff359c-cf92-4a5b-8610-f6eb41acbe73\") " pod="openshift-ovn-kubernetes/ovnkube-node-7x9jg" Oct 30 20:37:17.663344 ip-10-0-138-149 kubenswrapper[2573]: I1030 20:37:17.663279 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/23527d33-4e73-4c36-ba42-7f0a5b7f62c6-etc-modprobe-d\") pod \"tuned-w9npq\" (UID: \"23527d33-4e73-4c36-ba42-7f0a5b7f62c6\") " pod="openshift-cluster-node-tuning-operator/tuned-w9npq" Oct 30 20:37:17.663451 ip-10-0-138-149 kubenswrapper[2573]: I1030 20:37:17.663384 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/23527d33-4e73-4c36-ba42-7f0a5b7f62c6-run\") pod \"tuned-w9npq\" (UID: \"23527d33-4e73-4c36-ba42-7f0a5b7f62c6\") " pod="openshift-cluster-node-tuning-operator/tuned-w9npq" Oct 30 20:37:17.663451 ip-10-0-138-149 kubenswrapper[2573]: I1030 20:37:17.663417 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/23527d33-4e73-4c36-ba42-7f0a5b7f62c6-etc-tuned\") pod \"tuned-w9npq\" (UID: \"23527d33-4e73-4c36-ba42-7f0a5b7f62c6\") " pod="openshift-cluster-node-tuning-operator/tuned-w9npq" Oct 30 20:37:17.663451 ip-10-0-138-149 kubenswrapper[2573]: I1030 20:37:17.663442 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/c2ff359c-cf92-4a5b-8610-f6eb41acbe73-node-log\") pod \"ovnkube-node-7x9jg\" (UID: \"c2ff359c-cf92-4a5b-8610-f6eb41acbe73\") " pod="openshift-ovn-kubernetes/ovnkube-node-7x9jg" Oct 30 20:37:17.663606 ip-10-0-138-149 kubenswrapper[2573]: I1030 20:37:17.663470 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/c2ff359c-cf92-4a5b-8610-f6eb41acbe73-log-socket\") pod \"ovnkube-node-7x9jg\" (UID: \"c2ff359c-cf92-4a5b-8610-f6eb41acbe73\") " pod="openshift-ovn-kubernetes/ovnkube-node-7x9jg" Oct 30 20:37:17.663606 ip-10-0-138-149 kubenswrapper[2573]: I1030 20:37:17.663442 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-cjrkj" Oct 30 20:37:17.663606 ip-10-0-138-149 kubenswrapper[2573]: I1030 20:37:17.663536 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/b7fa45b4-283a-4439-82b7-778abf2e9deb-serviceca\") pod \"node-ca-tlch5\" (UID: \"b7fa45b4-283a-4439-82b7-778abf2e9deb\") " pod="openshift-image-registry/node-ca-tlch5" Oct 30 20:37:17.663606 ip-10-0-138-149 kubenswrapper[2573]: I1030 20:37:17.663589 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/c2ff359c-cf92-4a5b-8610-f6eb41acbe73-ovn-node-metrics-cert\") pod \"ovnkube-node-7x9jg\" (UID: \"c2ff359c-cf92-4a5b-8610-f6eb41acbe73\") " pod="openshift-ovn-kubernetes/ovnkube-node-7x9jg" Oct 30 20:37:17.663785 ip-10-0-138-149 kubenswrapper[2573]: I1030 20:37:17.663618 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/c2ff359c-cf92-4a5b-8610-f6eb41acbe73-run-systemd\") pod \"ovnkube-node-7x9jg\" (UID: \"c2ff359c-cf92-4a5b-8610-f6eb41acbe73\") " pod="openshift-ovn-kubernetes/ovnkube-node-7x9jg" Oct 30 20:37:17.663785 ip-10-0-138-149 kubenswrapper[2573]: I1030 20:37:17.663642 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/c2ff359c-cf92-4a5b-8610-f6eb41acbe73-host-cni-bin\") pod \"ovnkube-node-7x9jg\" (UID: \"c2ff359c-cf92-4a5b-8610-f6eb41acbe73\") " pod="openshift-ovn-kubernetes/ovnkube-node-7x9jg" Oct 30 20:37:17.663785 ip-10-0-138-149 kubenswrapper[2573]: I1030 20:37:17.663669 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/022c4cf0-9797-4428-8fa3-9f3112f2ccc5-cnibin\") pod \"multus-dzbhx\" (UID: \"022c4cf0-9797-4428-8fa3-9f3112f2ccc5\") " pod="openshift-multus/multus-dzbhx" Oct 30 20:37:17.663785 ip-10-0-138-149 kubenswrapper[2573]: I1030 20:37:17.663695 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/23527d33-4e73-4c36-ba42-7f0a5b7f62c6-host\") pod \"tuned-w9npq\" (UID: \"23527d33-4e73-4c36-ba42-7f0a5b7f62c6\") " pod="openshift-cluster-node-tuning-operator/tuned-w9npq" Oct 30 20:37:17.663785 ip-10-0-138-149 kubenswrapper[2573]: I1030 20:37:17.663719 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/23527d33-4e73-4c36-ba42-7f0a5b7f62c6-tmp\") pod \"tuned-w9npq\" (UID: \"23527d33-4e73-4c36-ba42-7f0a5b7f62c6\") " pod="openshift-cluster-node-tuning-operator/tuned-w9npq" Oct 30 20:37:17.663785 ip-10-0-138-149 kubenswrapper[2573]: I1030 20:37:17.663741 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/022c4cf0-9797-4428-8fa3-9f3112f2ccc5-hostroot\") pod \"multus-dzbhx\" (UID: \"022c4cf0-9797-4428-8fa3-9f3112f2ccc5\") " pod="openshift-multus/multus-dzbhx" Oct 30 20:37:17.663785 ip-10-0-138-149 kubenswrapper[2573]: I1030 20:37:17.663766 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/022c4cf0-9797-4428-8fa3-9f3112f2ccc5-host-run-multus-certs\") pod \"multus-dzbhx\" (UID: \"022c4cf0-9797-4428-8fa3-9f3112f2ccc5\") " pod="openshift-multus/multus-dzbhx" Oct 30 20:37:17.664111 ip-10-0-138-149 kubenswrapper[2573]: I1030 20:37:17.663848 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/022c4cf0-9797-4428-8fa3-9f3112f2ccc5-multus-daemon-config\") pod \"multus-dzbhx\" (UID: \"022c4cf0-9797-4428-8fa3-9f3112f2ccc5\") " pod="openshift-multus/multus-dzbhx" Oct 30 20:37:17.664111 ip-10-0-138-149 kubenswrapper[2573]: I1030 20:37:17.663886 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/c2ff359c-cf92-4a5b-8610-f6eb41acbe73-ovnkube-script-lib\") pod \"ovnkube-node-7x9jg\" (UID: \"c2ff359c-cf92-4a5b-8610-f6eb41acbe73\") " pod="openshift-ovn-kubernetes/ovnkube-node-7x9jg" Oct 30 20:37:17.664111 ip-10-0-138-149 kubenswrapper[2573]: I1030 20:37:17.663918 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/23527d33-4e73-4c36-ba42-7f0a5b7f62c6-etc-systemd\") pod \"tuned-w9npq\" (UID: \"23527d33-4e73-4c36-ba42-7f0a5b7f62c6\") " pod="openshift-cluster-node-tuning-operator/tuned-w9npq" Oct 30 20:37:17.664111 ip-10-0-138-149 kubenswrapper[2573]: I1030 20:37:17.663954 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/022c4cf0-9797-4428-8fa3-9f3112f2ccc5-system-cni-dir\") pod \"multus-dzbhx\" (UID: \"022c4cf0-9797-4428-8fa3-9f3112f2ccc5\") " pod="openshift-multus/multus-dzbhx" Oct 30 20:37:17.664111 ip-10-0-138-149 kubenswrapper[2573]: I1030 20:37:17.663985 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/022c4cf0-9797-4428-8fa3-9f3112f2ccc5-multus-socket-dir-parent\") pod \"multus-dzbhx\" (UID: \"022c4cf0-9797-4428-8fa3-9f3112f2ccc5\") " pod="openshift-multus/multus-dzbhx" Oct 30 20:37:17.664111 ip-10-0-138-149 kubenswrapper[2573]: I1030 20:37:17.664015 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/022c4cf0-9797-4428-8fa3-9f3112f2ccc5-host-var-lib-kubelet\") pod \"multus-dzbhx\" (UID: \"022c4cf0-9797-4428-8fa3-9f3112f2ccc5\") " pod="openshift-multus/multus-dzbhx" Oct 30 20:37:17.664412 ip-10-0-138-149 kubenswrapper[2573]: I1030 20:37:17.664110 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-24qwx\" (UniqueName: \"kubernetes.io/projected/c2ff359c-cf92-4a5b-8610-f6eb41acbe73-kube-api-access-24qwx\") pod \"ovnkube-node-7x9jg\" (UID: \"c2ff359c-cf92-4a5b-8610-f6eb41acbe73\") " pod="openshift-ovn-kubernetes/ovnkube-node-7x9jg" Oct 30 20:37:17.664412 ip-10-0-138-149 kubenswrapper[2573]: I1030 20:37:17.664133 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/c2ff359c-cf92-4a5b-8610-f6eb41acbe73-host-run-netns\") pod \"ovnkube-node-7x9jg\" (UID: \"c2ff359c-cf92-4a5b-8610-f6eb41acbe73\") " pod="openshift-ovn-kubernetes/ovnkube-node-7x9jg" Oct 30 20:37:17.664412 ip-10-0-138-149 kubenswrapper[2573]: I1030 20:37:17.664151 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c2ff359c-cf92-4a5b-8610-f6eb41acbe73-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-7x9jg\" (UID: \"c2ff359c-cf92-4a5b-8610-f6eb41acbe73\") " pod="openshift-ovn-kubernetes/ovnkube-node-7x9jg" Oct 30 20:37:17.664412 ip-10-0-138-149 kubenswrapper[2573]: I1030 20:37:17.664167 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/c2ff359c-cf92-4a5b-8610-f6eb41acbe73-env-overrides\") pod \"ovnkube-node-7x9jg\" (UID: \"c2ff359c-cf92-4a5b-8610-f6eb41acbe73\") " pod="openshift-ovn-kubernetes/ovnkube-node-7x9jg" Oct 30 20:37:17.664412 ip-10-0-138-149 kubenswrapper[2573]: I1030 20:37:17.664187 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/022c4cf0-9797-4428-8fa3-9f3112f2ccc5-multus-cni-dir\") pod \"multus-dzbhx\" (UID: \"022c4cf0-9797-4428-8fa3-9f3112f2ccc5\") " pod="openshift-multus/multus-dzbhx" Oct 30 20:37:17.664412 ip-10-0-138-149 kubenswrapper[2573]: I1030 20:37:17.664224 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/022c4cf0-9797-4428-8fa3-9f3112f2ccc5-host-var-lib-cni-multus\") pod \"multus-dzbhx\" (UID: \"022c4cf0-9797-4428-8fa3-9f3112f2ccc5\") " pod="openshift-multus/multus-dzbhx" Oct 30 20:37:17.664412 ip-10-0-138-149 kubenswrapper[2573]: I1030 20:37:17.664255 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/c2ff359c-cf92-4a5b-8610-f6eb41acbe73-run-ovn\") pod \"ovnkube-node-7x9jg\" (UID: \"c2ff359c-cf92-4a5b-8610-f6eb41acbe73\") " pod="openshift-ovn-kubernetes/ovnkube-node-7x9jg" Oct 30 20:37:17.664412 ip-10-0-138-149 kubenswrapper[2573]: I1030 20:37:17.664306 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/c2ff359c-cf92-4a5b-8610-f6eb41acbe73-var-lib-openvswitch\") pod \"ovnkube-node-7x9jg\" (UID: \"c2ff359c-cf92-4a5b-8610-f6eb41acbe73\") " pod="openshift-ovn-kubernetes/ovnkube-node-7x9jg" Oct 30 20:37:17.664412 ip-10-0-138-149 kubenswrapper[2573]: I1030 20:37:17.664332 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/23527d33-4e73-4c36-ba42-7f0a5b7f62c6-etc-sysconfig\") pod \"tuned-w9npq\" (UID: \"23527d33-4e73-4c36-ba42-7f0a5b7f62c6\") " pod="openshift-cluster-node-tuning-operator/tuned-w9npq" Oct 30 20:37:17.664412 ip-10-0-138-149 kubenswrapper[2573]: I1030 20:37:17.664361 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/022c4cf0-9797-4428-8fa3-9f3112f2ccc5-cni-binary-copy\") pod \"multus-dzbhx\" (UID: \"022c4cf0-9797-4428-8fa3-9f3112f2ccc5\") " pod="openshift-multus/multus-dzbhx" Oct 30 20:37:17.664412 ip-10-0-138-149 kubenswrapper[2573]: I1030 20:37:17.664383 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bc9kz\" (UniqueName: \"kubernetes.io/projected/567a72f6-035d-4c61-acaf-f46008029168-kube-api-access-bc9kz\") pod \"network-check-target-ws655\" (UID: \"567a72f6-035d-4c61-acaf-f46008029168\") " pod="openshift-network-diagnostics/network-check-target-ws655" Oct 30 20:37:17.664843 ip-10-0-138-149 kubenswrapper[2573]: I1030 20:37:17.664433 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qjn9k\" (UniqueName: \"kubernetes.io/projected/022c4cf0-9797-4428-8fa3-9f3112f2ccc5-kube-api-access-qjn9k\") pod \"multus-dzbhx\" (UID: \"022c4cf0-9797-4428-8fa3-9f3112f2ccc5\") " pod="openshift-multus/multus-dzbhx" Oct 30 20:37:17.664843 ip-10-0-138-149 kubenswrapper[2573]: I1030 20:37:17.664465 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e886897a-c6a3-414f-875f-aa112541b608-metrics-certs\") pod \"network-metrics-daemon-nhs9k\" (UID: \"e886897a-c6a3-414f-875f-aa112541b608\") " pod="openshift-multus/network-metrics-daemon-nhs9k" Oct 30 20:37:17.664843 ip-10-0-138-149 kubenswrapper[2573]: I1030 20:37:17.664485 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/b7fa45b4-283a-4439-82b7-778abf2e9deb-host\") pod \"node-ca-tlch5\" (UID: \"b7fa45b4-283a-4439-82b7-778abf2e9deb\") " pod="openshift-image-registry/node-ca-tlch5" Oct 30 20:37:17.664843 ip-10-0-138-149 kubenswrapper[2573]: I1030 20:37:17.664501 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/022c4cf0-9797-4428-8fa3-9f3112f2ccc5-host-run-k8s-cni-cncf-io\") pod \"multus-dzbhx\" (UID: \"022c4cf0-9797-4428-8fa3-9f3112f2ccc5\") " pod="openshift-multus/multus-dzbhx" Oct 30 20:37:17.664843 ip-10-0-138-149 kubenswrapper[2573]: I1030 20:37:17.664518 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/022c4cf0-9797-4428-8fa3-9f3112f2ccc5-etc-kubernetes\") pod \"multus-dzbhx\" (UID: \"022c4cf0-9797-4428-8fa3-9f3112f2ccc5\") " pod="openshift-multus/multus-dzbhx" Oct 30 20:37:17.664843 ip-10-0-138-149 kubenswrapper[2573]: I1030 20:37:17.664587 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/c2ff359c-cf92-4a5b-8610-f6eb41acbe73-host-slash\") pod \"ovnkube-node-7x9jg\" (UID: \"c2ff359c-cf92-4a5b-8610-f6eb41acbe73\") " pod="openshift-ovn-kubernetes/ovnkube-node-7x9jg" Oct 30 20:37:17.664843 ip-10-0-138-149 kubenswrapper[2573]: I1030 20:37:17.664626 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/23527d33-4e73-4c36-ba42-7f0a5b7f62c6-etc-kubernetes\") pod \"tuned-w9npq\" (UID: \"23527d33-4e73-4c36-ba42-7f0a5b7f62c6\") " pod="openshift-cluster-node-tuning-operator/tuned-w9npq" Oct 30 20:37:17.664843 ip-10-0-138-149 kubenswrapper[2573]: I1030 20:37:17.664655 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/022c4cf0-9797-4428-8fa3-9f3112f2ccc5-multus-conf-dir\") pod \"multus-dzbhx\" (UID: \"022c4cf0-9797-4428-8fa3-9f3112f2ccc5\") " pod="openshift-multus/multus-dzbhx" Oct 30 20:37:17.664843 ip-10-0-138-149 kubenswrapper[2573]: I1030 20:37:17.664683 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/c2ff359c-cf92-4a5b-8610-f6eb41acbe73-etc-openvswitch\") pod \"ovnkube-node-7x9jg\" (UID: \"c2ff359c-cf92-4a5b-8610-f6eb41acbe73\") " pod="openshift-ovn-kubernetes/ovnkube-node-7x9jg" Oct 30 20:37:17.664843 ip-10-0-138-149 kubenswrapper[2573]: I1030 20:37:17.664706 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/23527d33-4e73-4c36-ba42-7f0a5b7f62c6-etc-sysctl-d\") pod \"tuned-w9npq\" (UID: \"23527d33-4e73-4c36-ba42-7f0a5b7f62c6\") " pod="openshift-cluster-node-tuning-operator/tuned-w9npq" Oct 30 20:37:17.664843 ip-10-0-138-149 kubenswrapper[2573]: I1030 20:37:17.664730 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/23527d33-4e73-4c36-ba42-7f0a5b7f62c6-sys\") pod \"tuned-w9npq\" (UID: \"23527d33-4e73-4c36-ba42-7f0a5b7f62c6\") " pod="openshift-cluster-node-tuning-operator/tuned-w9npq" Oct 30 20:37:17.664843 ip-10-0-138-149 kubenswrapper[2573]: I1030 20:37:17.664746 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/022c4cf0-9797-4428-8fa3-9f3112f2ccc5-host-var-lib-cni-bin\") pod \"multus-dzbhx\" (UID: \"022c4cf0-9797-4428-8fa3-9f3112f2ccc5\") " pod="openshift-multus/multus-dzbhx" Oct 30 20:37:17.664843 ip-10-0-138-149 kubenswrapper[2573]: I1030 20:37:17.664762 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/c2ff359c-cf92-4a5b-8610-f6eb41acbe73-run-openvswitch\") pod \"ovnkube-node-7x9jg\" (UID: \"c2ff359c-cf92-4a5b-8610-f6eb41acbe73\") " pod="openshift-ovn-kubernetes/ovnkube-node-7x9jg" Oct 30 20:37:17.664843 ip-10-0-138-149 kubenswrapper[2573]: I1030 20:37:17.664777 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c2ff359c-cf92-4a5b-8610-f6eb41acbe73-host-run-ovn-kubernetes\") pod \"ovnkube-node-7x9jg\" (UID: \"c2ff359c-cf92-4a5b-8610-f6eb41acbe73\") " pod="openshift-ovn-kubernetes/ovnkube-node-7x9jg" Oct 30 20:37:17.664843 ip-10-0-138-149 kubenswrapper[2573]: I1030 20:37:17.664793 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/23527d33-4e73-4c36-ba42-7f0a5b7f62c6-etc-sysctl-conf\") pod \"tuned-w9npq\" (UID: \"23527d33-4e73-4c36-ba42-7f0a5b7f62c6\") " pod="openshift-cluster-node-tuning-operator/tuned-w9npq" Oct 30 20:37:17.664843 ip-10-0-138-149 kubenswrapper[2573]: I1030 20:37:17.664823 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/23527d33-4e73-4c36-ba42-7f0a5b7f62c6-lib-modules\") pod \"tuned-w9npq\" (UID: \"23527d33-4e73-4c36-ba42-7f0a5b7f62c6\") " pod="openshift-cluster-node-tuning-operator/tuned-w9npq" Oct 30 20:37:17.665355 ip-10-0-138-149 kubenswrapper[2573]: I1030 20:37:17.664872 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/022c4cf0-9797-4428-8fa3-9f3112f2ccc5-host-run-netns\") pod \"multus-dzbhx\" (UID: \"022c4cf0-9797-4428-8fa3-9f3112f2ccc5\") " pod="openshift-multus/multus-dzbhx" Oct 30 20:37:17.665355 ip-10-0-138-149 kubenswrapper[2573]: I1030 20:37:17.664889 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/c2ff359c-cf92-4a5b-8610-f6eb41acbe73-host-kubelet\") pod \"ovnkube-node-7x9jg\" (UID: \"c2ff359c-cf92-4a5b-8610-f6eb41acbe73\") " pod="openshift-ovn-kubernetes/ovnkube-node-7x9jg" Oct 30 20:37:17.665355 ip-10-0-138-149 kubenswrapper[2573]: I1030 20:37:17.664908 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/c2ff359c-cf92-4a5b-8610-f6eb41acbe73-systemd-units\") pod \"ovnkube-node-7x9jg\" (UID: \"c2ff359c-cf92-4a5b-8610-f6eb41acbe73\") " pod="openshift-ovn-kubernetes/ovnkube-node-7x9jg" Oct 30 20:37:17.665355 ip-10-0-138-149 kubenswrapper[2573]: I1030 20:37:17.664924 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-28zhr\" (UniqueName: \"kubernetes.io/projected/e886897a-c6a3-414f-875f-aa112541b608-kube-api-access-28zhr\") pod \"network-metrics-daemon-nhs9k\" (UID: \"e886897a-c6a3-414f-875f-aa112541b608\") " pod="openshift-multus/network-metrics-daemon-nhs9k" Oct 30 20:37:17.665355 ip-10-0-138-149 kubenswrapper[2573]: I1030 20:37:17.664941 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/23527d33-4e73-4c36-ba42-7f0a5b7f62c6-var-lib-kubelet\") pod \"tuned-w9npq\" (UID: \"23527d33-4e73-4c36-ba42-7f0a5b7f62c6\") " pod="openshift-cluster-node-tuning-operator/tuned-w9npq" Oct 30 20:37:17.665355 ip-10-0-138-149 kubenswrapper[2573]: I1030 20:37:17.665009 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xcs6c\" (UniqueName: \"kubernetes.io/projected/23527d33-4e73-4c36-ba42-7f0a5b7f62c6-kube-api-access-xcs6c\") pod \"tuned-w9npq\" (UID: \"23527d33-4e73-4c36-ba42-7f0a5b7f62c6\") " pod="openshift-cluster-node-tuning-operator/tuned-w9npq" Oct 30 20:37:17.665355 ip-10-0-138-149 kubenswrapper[2573]: I1030 20:37:17.665041 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-crg8q\" (UniqueName: \"kubernetes.io/projected/b7fa45b4-283a-4439-82b7-778abf2e9deb-kube-api-access-crg8q\") pod \"node-ca-tlch5\" (UID: \"b7fa45b4-283a-4439-82b7-778abf2e9deb\") " pod="openshift-image-registry/node-ca-tlch5" Oct 30 20:37:17.666003 ip-10-0-138-149 kubenswrapper[2573]: I1030 20:37:17.665984 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-bcmwc" Oct 30 20:37:17.666101 ip-10-0-138-149 kubenswrapper[2573]: E1030 20:37:17.666035 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-bcmwc" podUID="0d6769f4-dc38-45ac-863b-c75fa672866a" Oct 30 20:37:17.668982 ip-10-0-138-149 kubenswrapper[2573]: I1030 20:37:17.668953 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-svf4h" Oct 30 20:37:17.696631 ip-10-0-138-149 kubenswrapper[2573]: I1030 20:37:17.696611 2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Oct 30 20:37:17.696738 ip-10-0-138-149 kubenswrapper[2573]: I1030 20:37:17.696610 2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Oct 30 20:37:17.696738 ip-10-0-138-149 kubenswrapper[2573]: I1030 20:37:17.696679 2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Oct 30 20:37:17.696823 ip-10-0-138-149 kubenswrapper[2573]: I1030 20:37:17.696784 2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-sr85n\"" Oct 30 20:37:17.698143 ip-10-0-138-149 kubenswrapper[2573]: I1030 20:37:17.698119 2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Oct 30 20:37:17.698434 ip-10-0-138-149 kubenswrapper[2573]: I1030 20:37:17.698249 2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Oct 30 20:37:17.698434 ip-10-0-138-149 kubenswrapper[2573]: I1030 20:37:17.698308 2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Oct 30 20:37:17.698434 ip-10-0-138-149 kubenswrapper[2573]: I1030 20:37:17.698310 2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Oct 30 20:37:17.698434 ip-10-0-138-149 kubenswrapper[2573]: I1030 20:37:17.698405 2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-node-identity-dockercfg-rv75s\"" Oct 30 20:37:17.704657 ip-10-0-138-149 kubenswrapper[2573]: I1030 20:37:17.704622 2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Oct 30 20:37:17.704657 ip-10-0-138-149 kubenswrapper[2573]: I1030 20:37:17.704641 2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Oct 30 20:37:17.704839 ip-10-0-138-149 kubenswrapper[2573]: I1030 20:37:17.704805 2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Oct 30 20:37:17.704839 ip-10-0-138-149 kubenswrapper[2573]: I1030 20:37:17.704830 2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-g2pwr\"" Oct 30 20:37:17.704975 ip-10-0-138-149 kubenswrapper[2573]: I1030 20:37:17.704864 2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Oct 30 20:37:17.704975 ip-10-0-138-149 kubenswrapper[2573]: I1030 20:37:17.704872 2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Oct 30 20:37:17.704975 ip-10-0-138-149 kubenswrapper[2573]: I1030 20:37:17.704870 2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Oct 30 20:37:17.706442 ip-10-0-138-149 kubenswrapper[2573]: I1030 20:37:17.706429 2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Oct 30 20:37:17.706586 ip-10-0-138-149 kubenswrapper[2573]: I1030 20:37:17.706570 2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Oct 30 20:37:17.706690 ip-10-0-138-149 kubenswrapper[2573]: I1030 20:37:17.706674 2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-86vxf\"" Oct 30 20:37:17.708691 ip-10-0-138-149 kubenswrapper[2573]: I1030 20:37:17.708676 2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-jwfqc\"" Oct 30 20:37:17.713378 ip-10-0-138-149 kubenswrapper[2573]: I1030 20:37:17.713364 2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Oct 30 20:37:17.713430 ip-10-0-138-149 kubenswrapper[2573]: I1030 20:37:17.713371 2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Oct 30 20:37:17.713430 ip-10-0-138-149 kubenswrapper[2573]: I1030 20:37:17.713405 2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Oct 30 20:37:17.734782 ip-10-0-138-149 kubenswrapper[2573]: I1030 20:37:17.734742 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-149.us-west-1.compute.internal" event={"ID":"60d05e7d5fab914587e076383d73f1df","Type":"ContainerStarted","Data":"3ac9f12e7f98d1036fcc0c39f3aa3ab3c15fc903458208bef56f5c1969e7f9e0"} Oct 30 20:37:17.735853 ip-10-0-138-149 kubenswrapper[2573]: I1030 20:37:17.735828 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-138-149.us-west-1.compute.internal" event={"ID":"0d2e0d42ed810f19933d4cd655966ed4","Type":"ContainerStarted","Data":"da4df417dfa91620c245546301b2dbee88814652f2989ebdddb126fbdfc28a98"} Oct 30 20:37:17.737128 ip-10-0-138-149 kubenswrapper[2573]: I1030 20:37:17.737107 2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Oct 30 20:37:17.737233 ip-10-0-138-149 kubenswrapper[2573]: I1030 20:37:17.737141 2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Oct 30 20:37:17.737233 ip-10-0-138-149 kubenswrapper[2573]: I1030 20:37:17.737158 2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Oct 30 20:37:17.737353 ip-10-0-138-149 kubenswrapper[2573]: I1030 20:37:17.737265 2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Oct 30 20:37:17.737353 ip-10-0-138-149 kubenswrapper[2573]: I1030 20:37:17.737327 2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Oct 30 20:37:17.737353 ip-10-0-138-149 kubenswrapper[2573]: I1030 20:37:17.737335 2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Oct 30 20:37:17.737353 ip-10-0-138-149 kubenswrapper[2573]: I1030 20:37:17.737337 2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent-dockercfg-c5chh\"" Oct 30 20:37:17.737512 ip-10-0-138-149 kubenswrapper[2573]: I1030 20:37:17.737361 2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Oct 30 20:37:17.737512 ip-10-0-138-149 kubenswrapper[2573]: I1030 20:37:17.737418 2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Oct 30 20:37:17.744209 ip-10-0-138-149 kubenswrapper[2573]: I1030 20:37:17.744193 2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-94s75\"" Oct 30 20:37:17.744267 ip-10-0-138-149 kubenswrapper[2573]: I1030 20:37:17.744199 2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Oct 30 20:37:17.744267 ip-10-0-138-149 kubenswrapper[2573]: I1030 20:37:17.744222 2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-nvzwj\"" Oct 30 20:37:17.744267 ip-10-0-138-149 kubenswrapper[2573]: I1030 20:37:17.744236 2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-bmpds\"" Oct 30 20:37:17.746528 ip-10-0-138-149 kubenswrapper[2573]: I1030 20:37:17.746511 2573 reflector.go:446] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Oct 30 20:37:17.746585 ip-10-0-138-149 kubenswrapper[2573]: I1030 20:37:17.746559 2573 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-tth8k" Oct 30 20:37:17.750600 ip-10-0-138-149 kubenswrapper[2573]: I1030 20:37:17.750581 2573 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Oct 30 20:37:17.765950 ip-10-0-138-149 kubenswrapper[2573]: I1030 20:37:17.765928 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c2ff359c-cf92-4a5b-8610-f6eb41acbe73-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-7x9jg\" (UID: \"c2ff359c-cf92-4a5b-8610-f6eb41acbe73\") " pod="openshift-ovn-kubernetes/ovnkube-node-7x9jg" Oct 30 20:37:17.766029 ip-10-0-138-149 kubenswrapper[2573]: I1030 20:37:17.765953 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/c2ff359c-cf92-4a5b-8610-f6eb41acbe73-env-overrides\") pod \"ovnkube-node-7x9jg\" (UID: \"c2ff359c-cf92-4a5b-8610-f6eb41acbe73\") " pod="openshift-ovn-kubernetes/ovnkube-node-7x9jg" Oct 30 20:37:17.766029 ip-10-0-138-149 kubenswrapper[2573]: I1030 20:37:17.765971 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/022c4cf0-9797-4428-8fa3-9f3112f2ccc5-multus-cni-dir\") pod \"multus-dzbhx\" (UID: \"022c4cf0-9797-4428-8fa3-9f3112f2ccc5\") " pod="openshift-multus/multus-dzbhx" Oct 30 20:37:17.766029 ip-10-0-138-149 kubenswrapper[2573]: I1030 20:37:17.765986 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/022c4cf0-9797-4428-8fa3-9f3112f2ccc5-host-var-lib-cni-multus\") pod \"multus-dzbhx\" (UID: \"022c4cf0-9797-4428-8fa3-9f3112f2ccc5\") " pod="openshift-multus/multus-dzbhx" Oct 30 20:37:17.766139 ip-10-0-138-149 kubenswrapper[2573]: I1030 20:37:17.766038 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c2ff359c-cf92-4a5b-8610-f6eb41acbe73-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-7x9jg\" (UID: \"c2ff359c-cf92-4a5b-8610-f6eb41acbe73\") " pod="openshift-ovn-kubernetes/ovnkube-node-7x9jg" Oct 30 20:37:17.766139 ip-10-0-138-149 kubenswrapper[2573]: I1030 20:37:17.766077 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/662e676d-c357-4ac8-a813-d34096051aff-etc-selinux\") pod \"aws-ebs-csi-driver-node-svf4h\" (UID: \"662e676d-c357-4ac8-a813-d34096051aff\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-svf4h" Oct 30 20:37:17.766139 ip-10-0-138-149 kubenswrapper[2573]: I1030 20:37:17.766105 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/910c0d9f-c79b-4839-9a1b-f2d278d11fdb-tuning-conf-dir\") pod \"multus-additional-cni-plugins-blccx\" (UID: \"910c0d9f-c79b-4839-9a1b-f2d278d11fdb\") " pod="openshift-multus/multus-additional-cni-plugins-blccx" Oct 30 20:37:17.766139 ip-10-0-138-149 kubenswrapper[2573]: I1030 20:37:17.766127 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/022c4cf0-9797-4428-8fa3-9f3112f2ccc5-multus-cni-dir\") pod \"multus-dzbhx\" (UID: \"022c4cf0-9797-4428-8fa3-9f3112f2ccc5\") " pod="openshift-multus/multus-dzbhx" Oct 30 20:37:17.766287 ip-10-0-138-149 kubenswrapper[2573]: I1030 20:37:17.766136 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/c2ff359c-cf92-4a5b-8610-f6eb41acbe73-run-ovn\") pod \"ovnkube-node-7x9jg\" (UID: \"c2ff359c-cf92-4a5b-8610-f6eb41acbe73\") " pod="openshift-ovn-kubernetes/ovnkube-node-7x9jg" Oct 30 20:37:17.766287 ip-10-0-138-149 kubenswrapper[2573]: I1030 20:37:17.766128 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/022c4cf0-9797-4428-8fa3-9f3112f2ccc5-host-var-lib-cni-multus\") pod \"multus-dzbhx\" (UID: \"022c4cf0-9797-4428-8fa3-9f3112f2ccc5\") " pod="openshift-multus/multus-dzbhx" Oct 30 20:37:17.766287 ip-10-0-138-149 kubenswrapper[2573]: I1030 20:37:17.766199 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/910c0d9f-c79b-4839-9a1b-f2d278d11fdb-cni-binary-copy\") pod \"multus-additional-cni-plugins-blccx\" (UID: \"910c0d9f-c79b-4839-9a1b-f2d278d11fdb\") " pod="openshift-multus/multus-additional-cni-plugins-blccx" Oct 30 20:37:17.766287 ip-10-0-138-149 kubenswrapper[2573]: I1030 20:37:17.766227 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/c2ff359c-cf92-4a5b-8610-f6eb41acbe73-run-ovn\") pod \"ovnkube-node-7x9jg\" (UID: \"c2ff359c-cf92-4a5b-8610-f6eb41acbe73\") " pod="openshift-ovn-kubernetes/ovnkube-node-7x9jg" Oct 30 20:37:17.766287 ip-10-0-138-149 kubenswrapper[2573]: I1030 20:37:17.766233 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qs9cm\" (UniqueName: \"kubernetes.io/projected/910c0d9f-c79b-4839-9a1b-f2d278d11fdb-kube-api-access-qs9cm\") pod \"multus-additional-cni-plugins-blccx\" (UID: \"910c0d9f-c79b-4839-9a1b-f2d278d11fdb\") " pod="openshift-multus/multus-additional-cni-plugins-blccx" Oct 30 20:37:17.766287 ip-10-0-138-149 kubenswrapper[2573]: I1030 20:37:17.766270 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/c2ff359c-cf92-4a5b-8610-f6eb41acbe73-var-lib-openvswitch\") pod \"ovnkube-node-7x9jg\" (UID: \"c2ff359c-cf92-4a5b-8610-f6eb41acbe73\") " pod="openshift-ovn-kubernetes/ovnkube-node-7x9jg" Oct 30 20:37:17.766547 ip-10-0-138-149 kubenswrapper[2573]: I1030 20:37:17.766300 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/c2ff359c-cf92-4a5b-8610-f6eb41acbe73-var-lib-openvswitch\") pod \"ovnkube-node-7x9jg\" (UID: \"c2ff359c-cf92-4a5b-8610-f6eb41acbe73\") " pod="openshift-ovn-kubernetes/ovnkube-node-7x9jg" Oct 30 20:37:17.766547 ip-10-0-138-149 kubenswrapper[2573]: I1030 20:37:17.766312 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/23527d33-4e73-4c36-ba42-7f0a5b7f62c6-etc-sysconfig\") pod \"tuned-w9npq\" (UID: \"23527d33-4e73-4c36-ba42-7f0a5b7f62c6\") " pod="openshift-cluster-node-tuning-operator/tuned-w9npq" Oct 30 20:37:17.766547 ip-10-0-138-149 kubenswrapper[2573]: I1030 20:37:17.766345 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/022c4cf0-9797-4428-8fa3-9f3112f2ccc5-cni-binary-copy\") pod \"multus-dzbhx\" (UID: \"022c4cf0-9797-4428-8fa3-9f3112f2ccc5\") " pod="openshift-multus/multus-dzbhx" Oct 30 20:37:17.766547 ip-10-0-138-149 kubenswrapper[2573]: I1030 20:37:17.766373 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bc9kz\" (UniqueName: \"kubernetes.io/projected/567a72f6-035d-4c61-acaf-f46008029168-kube-api-access-bc9kz\") pod \"network-check-target-ws655\" (UID: \"567a72f6-035d-4c61-acaf-f46008029168\") " pod="openshift-network-diagnostics/network-check-target-ws655" Oct 30 20:37:17.766547 ip-10-0-138-149 kubenswrapper[2573]: I1030 20:37:17.766400 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/0d6769f4-dc38-45ac-863b-c75fa672866a-dbus\") pod \"global-pull-secret-syncer-bcmwc\" (UID: \"0d6769f4-dc38-45ac-863b-c75fa672866a\") " pod="kube-system/global-pull-secret-syncer-bcmwc" Oct 30 20:37:17.766547 ip-10-0-138-149 kubenswrapper[2573]: I1030 20:37:17.766405 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/c2ff359c-cf92-4a5b-8610-f6eb41acbe73-env-overrides\") pod \"ovnkube-node-7x9jg\" (UID: \"c2ff359c-cf92-4a5b-8610-f6eb41acbe73\") " pod="openshift-ovn-kubernetes/ovnkube-node-7x9jg" Oct 30 20:37:17.766547 ip-10-0-138-149 kubenswrapper[2573]: I1030 20:37:17.766415 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/23527d33-4e73-4c36-ba42-7f0a5b7f62c6-etc-sysconfig\") pod \"tuned-w9npq\" (UID: \"23527d33-4e73-4c36-ba42-7f0a5b7f62c6\") " pod="openshift-cluster-node-tuning-operator/tuned-w9npq" Oct 30 20:37:17.766547 ip-10-0-138-149 kubenswrapper[2573]: I1030 20:37:17.766427 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k82pb\" (UniqueName: \"kubernetes.io/projected/662e676d-c357-4ac8-a813-d34096051aff-kube-api-access-k82pb\") pod \"aws-ebs-csi-driver-node-svf4h\" (UID: \"662e676d-c357-4ac8-a813-d34096051aff\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-svf4h" Oct 30 20:37:17.766547 ip-10-0-138-149 kubenswrapper[2573]: I1030 20:37:17.766526 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t5psr\" (UniqueName: \"kubernetes.io/projected/affdfb88-e208-4ca1-843d-efc8384c27a1-kube-api-access-t5psr\") pod \"iptables-alerter-kvq2l\" (UID: \"affdfb88-e208-4ca1-843d-efc8384c27a1\") " pod="openshift-network-operator/iptables-alerter-kvq2l" Oct 30 20:37:17.766875 ip-10-0-138-149 kubenswrapper[2573]: I1030 20:37:17.766552 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qjn9k\" (UniqueName: \"kubernetes.io/projected/022c4cf0-9797-4428-8fa3-9f3112f2ccc5-kube-api-access-qjn9k\") pod \"multus-dzbhx\" (UID: \"022c4cf0-9797-4428-8fa3-9f3112f2ccc5\") " pod="openshift-multus/multus-dzbhx" Oct 30 20:37:17.766875 ip-10-0-138-149 kubenswrapper[2573]: I1030 20:37:17.766591 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/497c44c4-2ece-4009-b88c-6ecfa9920110-agent-certs\") pod \"konnectivity-agent-cjrkj\" (UID: \"497c44c4-2ece-4009-b88c-6ecfa9920110\") " pod="kube-system/konnectivity-agent-cjrkj" Oct 30 20:37:17.766875 ip-10-0-138-149 kubenswrapper[2573]: I1030 20:37:17.766641 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/0d6769f4-dc38-45ac-863b-c75fa672866a-kubelet-config\") pod \"global-pull-secret-syncer-bcmwc\" (UID: \"0d6769f4-dc38-45ac-863b-c75fa672866a\") " pod="kube-system/global-pull-secret-syncer-bcmwc" Oct 30 20:37:17.766875 ip-10-0-138-149 kubenswrapper[2573]: I1030 20:37:17.766739 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e886897a-c6a3-414f-875f-aa112541b608-metrics-certs\") pod \"network-metrics-daemon-nhs9k\" (UID: \"e886897a-c6a3-414f-875f-aa112541b608\") " pod="openshift-multus/network-metrics-daemon-nhs9k" Oct 30 20:37:17.766875 ip-10-0-138-149 kubenswrapper[2573]: I1030 20:37:17.766762 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/b7fa45b4-283a-4439-82b7-778abf2e9deb-host\") pod \"node-ca-tlch5\" (UID: \"b7fa45b4-283a-4439-82b7-778abf2e9deb\") " pod="openshift-image-registry/node-ca-tlch5" Oct 30 20:37:17.766875 ip-10-0-138-149 kubenswrapper[2573]: I1030 20:37:17.766795 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/022c4cf0-9797-4428-8fa3-9f3112f2ccc5-cni-binary-copy\") pod \"multus-dzbhx\" (UID: \"022c4cf0-9797-4428-8fa3-9f3112f2ccc5\") " pod="openshift-multus/multus-dzbhx" Oct 30 20:37:17.766875 ip-10-0-138-149 kubenswrapper[2573]: E1030 20:37:17.766814 2573 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 20:37:17.766875 ip-10-0-138-149 kubenswrapper[2573]: I1030 20:37:17.766848 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/022c4cf0-9797-4428-8fa3-9f3112f2ccc5-host-run-k8s-cni-cncf-io\") pod \"multus-dzbhx\" (UID: \"022c4cf0-9797-4428-8fa3-9f3112f2ccc5\") " pod="openshift-multus/multus-dzbhx" Oct 30 20:37:17.767110 ip-10-0-138-149 kubenswrapper[2573]: I1030 20:37:17.766880 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/022c4cf0-9797-4428-8fa3-9f3112f2ccc5-etc-kubernetes\") pod \"multus-dzbhx\" (UID: \"022c4cf0-9797-4428-8fa3-9f3112f2ccc5\") " pod="openshift-multus/multus-dzbhx" Oct 30 20:37:17.767110 ip-10-0-138-149 kubenswrapper[2573]: I1030 20:37:17.766889 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/b7fa45b4-283a-4439-82b7-778abf2e9deb-host\") pod \"node-ca-tlch5\" (UID: \"b7fa45b4-283a-4439-82b7-778abf2e9deb\") " pod="openshift-image-registry/node-ca-tlch5" Oct 30 20:37:17.767110 ip-10-0-138-149 kubenswrapper[2573]: I1030 20:37:17.766899 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/022c4cf0-9797-4428-8fa3-9f3112f2ccc5-host-run-k8s-cni-cncf-io\") pod \"multus-dzbhx\" (UID: \"022c4cf0-9797-4428-8fa3-9f3112f2ccc5\") " pod="openshift-multus/multus-dzbhx" Oct 30 20:37:17.767110 ip-10-0-138-149 kubenswrapper[2573]: I1030 20:37:17.766940 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/022c4cf0-9797-4428-8fa3-9f3112f2ccc5-etc-kubernetes\") pod \"multus-dzbhx\" (UID: \"022c4cf0-9797-4428-8fa3-9f3112f2ccc5\") " pod="openshift-multus/multus-dzbhx" Oct 30 20:37:17.767110 ip-10-0-138-149 kubenswrapper[2573]: E1030 20:37:17.766943 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e886897a-c6a3-414f-875f-aa112541b608-metrics-certs podName:e886897a-c6a3-414f-875f-aa112541b608 nodeName:}" failed. No retries permitted until 2026-10-30 20:37:18.266912705 +0000 UTC m=+3.079364027 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/e886897a-c6a3-414f-875f-aa112541b608-metrics-certs") pod "network-metrics-daemon-nhs9k" (UID: "e886897a-c6a3-414f-875f-aa112541b608") : object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 20:37:17.767312 ip-10-0-138-149 kubenswrapper[2573]: I1030 20:37:17.766971 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/910c0d9f-c79b-4839-9a1b-f2d278d11fdb-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-blccx\" (UID: \"910c0d9f-c79b-4839-9a1b-f2d278d11fdb\") " pod="openshift-multus/multus-additional-cni-plugins-blccx" Oct 30 20:37:17.769752 ip-10-0-138-149 kubenswrapper[2573]: I1030 20:37:17.767353 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/666b6e91-4a60-45d9-9e3e-877867ded7fc-hosts-file\") pod \"node-resolver-nv2rk\" (UID: \"666b6e91-4a60-45d9-9e3e-877867ded7fc\") " pod="openshift-dns/node-resolver-nv2rk" Oct 30 20:37:17.769752 ip-10-0-138-149 kubenswrapper[2573]: I1030 20:37:17.767420 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/c2ff359c-cf92-4a5b-8610-f6eb41acbe73-host-slash\") pod \"ovnkube-node-7x9jg\" (UID: \"c2ff359c-cf92-4a5b-8610-f6eb41acbe73\") " pod="openshift-ovn-kubernetes/ovnkube-node-7x9jg" Oct 30 20:37:17.769752 ip-10-0-138-149 kubenswrapper[2573]: I1030 20:37:17.767467 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/23527d33-4e73-4c36-ba42-7f0a5b7f62c6-etc-kubernetes\") pod \"tuned-w9npq\" (UID: \"23527d33-4e73-4c36-ba42-7f0a5b7f62c6\") " pod="openshift-cluster-node-tuning-operator/tuned-w9npq" Oct 30 20:37:17.769752 ip-10-0-138-149 kubenswrapper[2573]: I1030 20:37:17.767519 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/c2ff359c-cf92-4a5b-8610-f6eb41acbe73-host-slash\") pod \"ovnkube-node-7x9jg\" (UID: \"c2ff359c-cf92-4a5b-8610-f6eb41acbe73\") " pod="openshift-ovn-kubernetes/ovnkube-node-7x9jg" Oct 30 20:37:17.769752 ip-10-0-138-149 kubenswrapper[2573]: I1030 20:37:17.767540 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/662e676d-c357-4ac8-a813-d34096051aff-socket-dir\") pod \"aws-ebs-csi-driver-node-svf4h\" (UID: \"662e676d-c357-4ac8-a813-d34096051aff\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-svf4h" Oct 30 20:37:17.769752 ip-10-0-138-149 kubenswrapper[2573]: I1030 20:37:17.767580 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/23527d33-4e73-4c36-ba42-7f0a5b7f62c6-etc-kubernetes\") pod \"tuned-w9npq\" (UID: \"23527d33-4e73-4c36-ba42-7f0a5b7f62c6\") " pod="openshift-cluster-node-tuning-operator/tuned-w9npq" Oct 30 20:37:17.769752 ip-10-0-138-149 kubenswrapper[2573]: I1030 20:37:17.767609 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/022c4cf0-9797-4428-8fa3-9f3112f2ccc5-multus-conf-dir\") pod \"multus-dzbhx\" (UID: \"022c4cf0-9797-4428-8fa3-9f3112f2ccc5\") " pod="openshift-multus/multus-dzbhx" Oct 30 20:37:17.769752 ip-10-0-138-149 kubenswrapper[2573]: I1030 20:37:17.767656 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/c2ff359c-cf92-4a5b-8610-f6eb41acbe73-etc-openvswitch\") pod \"ovnkube-node-7x9jg\" (UID: \"c2ff359c-cf92-4a5b-8610-f6eb41acbe73\") " pod="openshift-ovn-kubernetes/ovnkube-node-7x9jg" Oct 30 20:37:17.769752 ip-10-0-138-149 kubenswrapper[2573]: I1030 20:37:17.767683 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/23527d33-4e73-4c36-ba42-7f0a5b7f62c6-etc-sysctl-d\") pod \"tuned-w9npq\" (UID: \"23527d33-4e73-4c36-ba42-7f0a5b7f62c6\") " pod="openshift-cluster-node-tuning-operator/tuned-w9npq" Oct 30 20:37:17.769752 ip-10-0-138-149 kubenswrapper[2573]: I1030 20:37:17.767716 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/23527d33-4e73-4c36-ba42-7f0a5b7f62c6-sys\") pod \"tuned-w9npq\" (UID: \"23527d33-4e73-4c36-ba42-7f0a5b7f62c6\") " pod="openshift-cluster-node-tuning-operator/tuned-w9npq" Oct 30 20:37:17.769752 ip-10-0-138-149 kubenswrapper[2573]: I1030 20:37:17.767749 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/022c4cf0-9797-4428-8fa3-9f3112f2ccc5-host-var-lib-cni-bin\") pod \"multus-dzbhx\" (UID: \"022c4cf0-9797-4428-8fa3-9f3112f2ccc5\") " pod="openshift-multus/multus-dzbhx" Oct 30 20:37:17.769752 ip-10-0-138-149 kubenswrapper[2573]: I1030 20:37:17.767784 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/666b6e91-4a60-45d9-9e3e-877867ded7fc-tmp-dir\") pod \"node-resolver-nv2rk\" (UID: \"666b6e91-4a60-45d9-9e3e-877867ded7fc\") " pod="openshift-dns/node-resolver-nv2rk" Oct 30 20:37:17.769752 ip-10-0-138-149 kubenswrapper[2573]: I1030 20:37:17.767822 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2w5c8\" (UniqueName: \"kubernetes.io/projected/666b6e91-4a60-45d9-9e3e-877867ded7fc-kube-api-access-2w5c8\") pod \"node-resolver-nv2rk\" (UID: \"666b6e91-4a60-45d9-9e3e-877867ded7fc\") " pod="openshift-dns/node-resolver-nv2rk" Oct 30 20:37:17.769752 ip-10-0-138-149 kubenswrapper[2573]: I1030 20:37:17.767849 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/23527d33-4e73-4c36-ba42-7f0a5b7f62c6-etc-sysctl-d\") pod \"tuned-w9npq\" (UID: \"23527d33-4e73-4c36-ba42-7f0a5b7f62c6\") " pod="openshift-cluster-node-tuning-operator/tuned-w9npq" Oct 30 20:37:17.769752 ip-10-0-138-149 kubenswrapper[2573]: I1030 20:37:17.767852 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/c2ff359c-cf92-4a5b-8610-f6eb41acbe73-run-openvswitch\") pod \"ovnkube-node-7x9jg\" (UID: \"c2ff359c-cf92-4a5b-8610-f6eb41acbe73\") " pod="openshift-ovn-kubernetes/ovnkube-node-7x9jg" Oct 30 20:37:17.769752 ip-10-0-138-149 kubenswrapper[2573]: I1030 20:37:17.767885 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/c2ff359c-cf92-4a5b-8610-f6eb41acbe73-run-openvswitch\") pod \"ovnkube-node-7x9jg\" (UID: \"c2ff359c-cf92-4a5b-8610-f6eb41acbe73\") " pod="openshift-ovn-kubernetes/ovnkube-node-7x9jg" Oct 30 20:37:17.769752 ip-10-0-138-149 kubenswrapper[2573]: I1030 20:37:17.767894 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c2ff359c-cf92-4a5b-8610-f6eb41acbe73-host-run-ovn-kubernetes\") pod \"ovnkube-node-7x9jg\" (UID: \"c2ff359c-cf92-4a5b-8610-f6eb41acbe73\") " pod="openshift-ovn-kubernetes/ovnkube-node-7x9jg" Oct 30 20:37:17.770604 ip-10-0-138-149 kubenswrapper[2573]: I1030 20:37:17.767943 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/23527d33-4e73-4c36-ba42-7f0a5b7f62c6-etc-sysctl-conf\") pod \"tuned-w9npq\" (UID: \"23527d33-4e73-4c36-ba42-7f0a5b7f62c6\") " pod="openshift-cluster-node-tuning-operator/tuned-w9npq" Oct 30 20:37:17.770604 ip-10-0-138-149 kubenswrapper[2573]: I1030 20:37:17.767959 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/23527d33-4e73-4c36-ba42-7f0a5b7f62c6-sys\") pod \"tuned-w9npq\" (UID: \"23527d33-4e73-4c36-ba42-7f0a5b7f62c6\") " pod="openshift-cluster-node-tuning-operator/tuned-w9npq" Oct 30 20:37:17.770604 ip-10-0-138-149 kubenswrapper[2573]: I1030 20:37:17.767715 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/c2ff359c-cf92-4a5b-8610-f6eb41acbe73-etc-openvswitch\") pod \"ovnkube-node-7x9jg\" (UID: \"c2ff359c-cf92-4a5b-8610-f6eb41acbe73\") " pod="openshift-ovn-kubernetes/ovnkube-node-7x9jg" Oct 30 20:37:17.770604 ip-10-0-138-149 kubenswrapper[2573]: I1030 20:37:17.767976 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/23527d33-4e73-4c36-ba42-7f0a5b7f62c6-lib-modules\") pod \"tuned-w9npq\" (UID: \"23527d33-4e73-4c36-ba42-7f0a5b7f62c6\") " pod="openshift-cluster-node-tuning-operator/tuned-w9npq" Oct 30 20:37:17.770604 ip-10-0-138-149 kubenswrapper[2573]: I1030 20:37:17.768009 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/022c4cf0-9797-4428-8fa3-9f3112f2ccc5-host-var-lib-cni-bin\") pod \"multus-dzbhx\" (UID: \"022c4cf0-9797-4428-8fa3-9f3112f2ccc5\") " pod="openshift-multus/multus-dzbhx" Oct 30 20:37:17.770604 ip-10-0-138-149 kubenswrapper[2573]: I1030 20:37:17.768023 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/022c4cf0-9797-4428-8fa3-9f3112f2ccc5-host-run-netns\") pod \"multus-dzbhx\" (UID: \"022c4cf0-9797-4428-8fa3-9f3112f2ccc5\") " pod="openshift-multus/multus-dzbhx" Oct 30 20:37:17.770604 ip-10-0-138-149 kubenswrapper[2573]: I1030 20:37:17.767683 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/022c4cf0-9797-4428-8fa3-9f3112f2ccc5-multus-conf-dir\") pod \"multus-dzbhx\" (UID: \"022c4cf0-9797-4428-8fa3-9f3112f2ccc5\") " pod="openshift-multus/multus-dzbhx" Oct 30 20:37:17.770604 ip-10-0-138-149 kubenswrapper[2573]: I1030 20:37:17.768094 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/662e676d-c357-4ac8-a813-d34096051aff-registration-dir\") pod \"aws-ebs-csi-driver-node-svf4h\" (UID: \"662e676d-c357-4ac8-a813-d34096051aff\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-svf4h" Oct 30 20:37:17.770604 ip-10-0-138-149 kubenswrapper[2573]: I1030 20:37:17.768100 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/022c4cf0-9797-4428-8fa3-9f3112f2ccc5-host-run-netns\") pod \"multus-dzbhx\" (UID: \"022c4cf0-9797-4428-8fa3-9f3112f2ccc5\") " pod="openshift-multus/multus-dzbhx" Oct 30 20:37:17.770604 ip-10-0-138-149 kubenswrapper[2573]: I1030 20:37:17.768130 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/910c0d9f-c79b-4839-9a1b-f2d278d11fdb-cnibin\") pod \"multus-additional-cni-plugins-blccx\" (UID: \"910c0d9f-c79b-4839-9a1b-f2d278d11fdb\") " pod="openshift-multus/multus-additional-cni-plugins-blccx" Oct 30 20:37:17.770604 ip-10-0-138-149 kubenswrapper[2573]: I1030 20:37:17.768145 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c2ff359c-cf92-4a5b-8610-f6eb41acbe73-host-run-ovn-kubernetes\") pod \"ovnkube-node-7x9jg\" (UID: \"c2ff359c-cf92-4a5b-8610-f6eb41acbe73\") " pod="openshift-ovn-kubernetes/ovnkube-node-7x9jg" Oct 30 20:37:17.770604 ip-10-0-138-149 kubenswrapper[2573]: I1030 20:37:17.768170 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/affdfb88-e208-4ca1-843d-efc8384c27a1-iptables-alerter-script\") pod \"iptables-alerter-kvq2l\" (UID: \"affdfb88-e208-4ca1-843d-efc8384c27a1\") " pod="openshift-network-operator/iptables-alerter-kvq2l" Oct 30 20:37:17.770604 ip-10-0-138-149 kubenswrapper[2573]: I1030 20:37:17.768215 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/c2ff359c-cf92-4a5b-8610-f6eb41acbe73-host-kubelet\") pod \"ovnkube-node-7x9jg\" (UID: \"c2ff359c-cf92-4a5b-8610-f6eb41acbe73\") " pod="openshift-ovn-kubernetes/ovnkube-node-7x9jg" Oct 30 20:37:17.770604 ip-10-0-138-149 kubenswrapper[2573]: I1030 20:37:17.768268 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/c2ff359c-cf92-4a5b-8610-f6eb41acbe73-systemd-units\") pod \"ovnkube-node-7x9jg\" (UID: \"c2ff359c-cf92-4a5b-8610-f6eb41acbe73\") " pod="openshift-ovn-kubernetes/ovnkube-node-7x9jg" Oct 30 20:37:17.770604 ip-10-0-138-149 kubenswrapper[2573]: I1030 20:37:17.768306 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/c2ff359c-cf92-4a5b-8610-f6eb41acbe73-host-kubelet\") pod \"ovnkube-node-7x9jg\" (UID: \"c2ff359c-cf92-4a5b-8610-f6eb41acbe73\") " pod="openshift-ovn-kubernetes/ovnkube-node-7x9jg" Oct 30 20:37:17.770604 ip-10-0-138-149 kubenswrapper[2573]: I1030 20:37:17.768457 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/23527d33-4e73-4c36-ba42-7f0a5b7f62c6-etc-sysctl-conf\") pod \"tuned-w9npq\" (UID: \"23527d33-4e73-4c36-ba42-7f0a5b7f62c6\") " pod="openshift-cluster-node-tuning-operator/tuned-w9npq" Oct 30 20:37:17.770604 ip-10-0-138-149 kubenswrapper[2573]: I1030 20:37:17.768634 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/23527d33-4e73-4c36-ba42-7f0a5b7f62c6-lib-modules\") pod \"tuned-w9npq\" (UID: \"23527d33-4e73-4c36-ba42-7f0a5b7f62c6\") " pod="openshift-cluster-node-tuning-operator/tuned-w9npq" Oct 30 20:37:17.771383 ip-10-0-138-149 kubenswrapper[2573]: I1030 20:37:17.768692 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-28zhr\" (UniqueName: \"kubernetes.io/projected/e886897a-c6a3-414f-875f-aa112541b608-kube-api-access-28zhr\") pod \"network-metrics-daemon-nhs9k\" (UID: \"e886897a-c6a3-414f-875f-aa112541b608\") " pod="openshift-multus/network-metrics-daemon-nhs9k" Oct 30 20:37:17.771383 ip-10-0-138-149 kubenswrapper[2573]: I1030 20:37:17.768727 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/662e676d-c357-4ac8-a813-d34096051aff-device-dir\") pod \"aws-ebs-csi-driver-node-svf4h\" (UID: \"662e676d-c357-4ac8-a813-d34096051aff\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-svf4h" Oct 30 20:37:17.771383 ip-10-0-138-149 kubenswrapper[2573]: I1030 20:37:17.768801 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/affdfb88-e208-4ca1-843d-efc8384c27a1-host-slash\") pod \"iptables-alerter-kvq2l\" (UID: \"affdfb88-e208-4ca1-843d-efc8384c27a1\") " pod="openshift-network-operator/iptables-alerter-kvq2l" Oct 30 20:37:17.771383 ip-10-0-138-149 kubenswrapper[2573]: I1030 20:37:17.768834 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/23527d33-4e73-4c36-ba42-7f0a5b7f62c6-var-lib-kubelet\") pod \"tuned-w9npq\" (UID: \"23527d33-4e73-4c36-ba42-7f0a5b7f62c6\") " pod="openshift-cluster-node-tuning-operator/tuned-w9npq" Oct 30 20:37:17.771383 ip-10-0-138-149 kubenswrapper[2573]: I1030 20:37:17.768883 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xcs6c\" (UniqueName: \"kubernetes.io/projected/23527d33-4e73-4c36-ba42-7f0a5b7f62c6-kube-api-access-xcs6c\") pod \"tuned-w9npq\" (UID: \"23527d33-4e73-4c36-ba42-7f0a5b7f62c6\") " pod="openshift-cluster-node-tuning-operator/tuned-w9npq" Oct 30 20:37:17.771383 ip-10-0-138-149 kubenswrapper[2573]: I1030 20:37:17.768908 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-crg8q\" (UniqueName: \"kubernetes.io/projected/b7fa45b4-283a-4439-82b7-778abf2e9deb-kube-api-access-crg8q\") pod \"node-ca-tlch5\" (UID: \"b7fa45b4-283a-4439-82b7-778abf2e9deb\") " pod="openshift-image-registry/node-ca-tlch5" Oct 30 20:37:17.771383 ip-10-0-138-149 kubenswrapper[2573]: I1030 20:37:17.769021 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/c2ff359c-cf92-4a5b-8610-f6eb41acbe73-host-cni-netd\") pod \"ovnkube-node-7x9jg\" (UID: \"c2ff359c-cf92-4a5b-8610-f6eb41acbe73\") " pod="openshift-ovn-kubernetes/ovnkube-node-7x9jg" Oct 30 20:37:17.771383 ip-10-0-138-149 kubenswrapper[2573]: I1030 20:37:17.769061 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/c2ff359c-cf92-4a5b-8610-f6eb41acbe73-ovnkube-config\") pod \"ovnkube-node-7x9jg\" (UID: \"c2ff359c-cf92-4a5b-8610-f6eb41acbe73\") " pod="openshift-ovn-kubernetes/ovnkube-node-7x9jg" Oct 30 20:37:17.771383 ip-10-0-138-149 kubenswrapper[2573]: I1030 20:37:17.769109 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/23527d33-4e73-4c36-ba42-7f0a5b7f62c6-etc-modprobe-d\") pod \"tuned-w9npq\" (UID: \"23527d33-4e73-4c36-ba42-7f0a5b7f62c6\") " pod="openshift-cluster-node-tuning-operator/tuned-w9npq" Oct 30 20:37:17.771383 ip-10-0-138-149 kubenswrapper[2573]: I1030 20:37:17.769141 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/23527d33-4e73-4c36-ba42-7f0a5b7f62c6-run\") pod \"tuned-w9npq\" (UID: \"23527d33-4e73-4c36-ba42-7f0a5b7f62c6\") " pod="openshift-cluster-node-tuning-operator/tuned-w9npq" Oct 30 20:37:17.771383 ip-10-0-138-149 kubenswrapper[2573]: I1030 20:37:17.769216 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/23527d33-4e73-4c36-ba42-7f0a5b7f62c6-etc-tuned\") pod \"tuned-w9npq\" (UID: \"23527d33-4e73-4c36-ba42-7f0a5b7f62c6\") " pod="openshift-cluster-node-tuning-operator/tuned-w9npq" Oct 30 20:37:17.771383 ip-10-0-138-149 kubenswrapper[2573]: I1030 20:37:17.769250 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/c2ff359c-cf92-4a5b-8610-f6eb41acbe73-node-log\") pod \"ovnkube-node-7x9jg\" (UID: \"c2ff359c-cf92-4a5b-8610-f6eb41acbe73\") " pod="openshift-ovn-kubernetes/ovnkube-node-7x9jg" Oct 30 20:37:17.771383 ip-10-0-138-149 kubenswrapper[2573]: I1030 20:37:17.769295 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/c2ff359c-cf92-4a5b-8610-f6eb41acbe73-log-socket\") pod \"ovnkube-node-7x9jg\" (UID: \"c2ff359c-cf92-4a5b-8610-f6eb41acbe73\") " pod="openshift-ovn-kubernetes/ovnkube-node-7x9jg" Oct 30 20:37:17.771383 ip-10-0-138-149 kubenswrapper[2573]: I1030 20:37:17.769325 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/b7fa45b4-283a-4439-82b7-778abf2e9deb-serviceca\") pod \"node-ca-tlch5\" (UID: \"b7fa45b4-283a-4439-82b7-778abf2e9deb\") " pod="openshift-image-registry/node-ca-tlch5" Oct 30 20:37:17.771383 ip-10-0-138-149 kubenswrapper[2573]: I1030 20:37:17.769531 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/c2ff359c-cf92-4a5b-8610-f6eb41acbe73-log-socket\") pod \"ovnkube-node-7x9jg\" (UID: \"c2ff359c-cf92-4a5b-8610-f6eb41acbe73\") " pod="openshift-ovn-kubernetes/ovnkube-node-7x9jg" Oct 30 20:37:17.771383 ip-10-0-138-149 kubenswrapper[2573]: I1030 20:37:17.769781 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/c2ff359c-cf92-4a5b-8610-f6eb41acbe73-node-log\") pod \"ovnkube-node-7x9jg\" (UID: \"c2ff359c-cf92-4a5b-8610-f6eb41acbe73\") " pod="openshift-ovn-kubernetes/ovnkube-node-7x9jg" Oct 30 20:37:17.771383 ip-10-0-138-149 kubenswrapper[2573]: I1030 20:37:17.769798 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/23527d33-4e73-4c36-ba42-7f0a5b7f62c6-var-lib-kubelet\") pod \"tuned-w9npq\" (UID: \"23527d33-4e73-4c36-ba42-7f0a5b7f62c6\") " pod="openshift-cluster-node-tuning-operator/tuned-w9npq" Oct 30 20:37:17.772208 ip-10-0-138-149 kubenswrapper[2573]: I1030 20:37:17.769804 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/c2ff359c-cf92-4a5b-8610-f6eb41acbe73-host-cni-netd\") pod \"ovnkube-node-7x9jg\" (UID: \"c2ff359c-cf92-4a5b-8610-f6eb41acbe73\") " pod="openshift-ovn-kubernetes/ovnkube-node-7x9jg" Oct 30 20:37:17.772208 ip-10-0-138-149 kubenswrapper[2573]: I1030 20:37:17.769838 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/23527d33-4e73-4c36-ba42-7f0a5b7f62c6-run\") pod \"tuned-w9npq\" (UID: \"23527d33-4e73-4c36-ba42-7f0a5b7f62c6\") " pod="openshift-cluster-node-tuning-operator/tuned-w9npq" Oct 30 20:37:17.772208 ip-10-0-138-149 kubenswrapper[2573]: I1030 20:37:17.769864 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/662e676d-c357-4ac8-a813-d34096051aff-kubelet-dir\") pod \"aws-ebs-csi-driver-node-svf4h\" (UID: \"662e676d-c357-4ac8-a813-d34096051aff\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-svf4h" Oct 30 20:37:17.772208 ip-10-0-138-149 kubenswrapper[2573]: I1030 20:37:17.769898 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/c2ff359c-cf92-4a5b-8610-f6eb41acbe73-ovn-node-metrics-cert\") pod \"ovnkube-node-7x9jg\" (UID: \"c2ff359c-cf92-4a5b-8610-f6eb41acbe73\") " pod="openshift-ovn-kubernetes/ovnkube-node-7x9jg" Oct 30 20:37:17.772208 ip-10-0-138-149 kubenswrapper[2573]: I1030 20:37:17.769957 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/c2ff359c-cf92-4a5b-8610-f6eb41acbe73-run-systemd\") pod \"ovnkube-node-7x9jg\" (UID: \"c2ff359c-cf92-4a5b-8610-f6eb41acbe73\") " pod="openshift-ovn-kubernetes/ovnkube-node-7x9jg" Oct 30 20:37:17.772208 ip-10-0-138-149 kubenswrapper[2573]: I1030 20:37:17.769961 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/23527d33-4e73-4c36-ba42-7f0a5b7f62c6-etc-modprobe-d\") pod \"tuned-w9npq\" (UID: \"23527d33-4e73-4c36-ba42-7f0a5b7f62c6\") " pod="openshift-cluster-node-tuning-operator/tuned-w9npq" Oct 30 20:37:17.772208 ip-10-0-138-149 kubenswrapper[2573]: I1030 20:37:17.769991 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/c2ff359c-cf92-4a5b-8610-f6eb41acbe73-host-cni-bin\") pod \"ovnkube-node-7x9jg\" (UID: \"c2ff359c-cf92-4a5b-8610-f6eb41acbe73\") " pod="openshift-ovn-kubernetes/ovnkube-node-7x9jg" Oct 30 20:37:17.772208 ip-10-0-138-149 kubenswrapper[2573]: I1030 20:37:17.770006 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/c2ff359c-cf92-4a5b-8610-f6eb41acbe73-run-systemd\") pod \"ovnkube-node-7x9jg\" (UID: \"c2ff359c-cf92-4a5b-8610-f6eb41acbe73\") " pod="openshift-ovn-kubernetes/ovnkube-node-7x9jg" Oct 30 20:37:17.772208 ip-10-0-138-149 kubenswrapper[2573]: I1030 20:37:17.770024 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/022c4cf0-9797-4428-8fa3-9f3112f2ccc5-cnibin\") pod \"multus-dzbhx\" (UID: \"022c4cf0-9797-4428-8fa3-9f3112f2ccc5\") " pod="openshift-multus/multus-dzbhx" Oct 30 20:37:17.772208 ip-10-0-138-149 kubenswrapper[2573]: I1030 20:37:17.770040 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/c2ff359c-cf92-4a5b-8610-f6eb41acbe73-host-cni-bin\") pod \"ovnkube-node-7x9jg\" (UID: \"c2ff359c-cf92-4a5b-8610-f6eb41acbe73\") " pod="openshift-ovn-kubernetes/ovnkube-node-7x9jg" Oct 30 20:37:17.772208 ip-10-0-138-149 kubenswrapper[2573]: I1030 20:37:17.770045 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/c2ff359c-cf92-4a5b-8610-f6eb41acbe73-ovnkube-config\") pod \"ovnkube-node-7x9jg\" (UID: \"c2ff359c-cf92-4a5b-8610-f6eb41acbe73\") " pod="openshift-ovn-kubernetes/ovnkube-node-7x9jg" Oct 30 20:37:17.772208 ip-10-0-138-149 kubenswrapper[2573]: I1030 20:37:17.770059 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/497c44c4-2ece-4009-b88c-6ecfa9920110-konnectivity-ca\") pod \"konnectivity-agent-cjrkj\" (UID: \"497c44c4-2ece-4009-b88c-6ecfa9920110\") " pod="kube-system/konnectivity-agent-cjrkj" Oct 30 20:37:17.772208 ip-10-0-138-149 kubenswrapper[2573]: I1030 20:37:17.770082 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/022c4cf0-9797-4428-8fa3-9f3112f2ccc5-cnibin\") pod \"multus-dzbhx\" (UID: \"022c4cf0-9797-4428-8fa3-9f3112f2ccc5\") " pod="openshift-multus/multus-dzbhx" Oct 30 20:37:17.772208 ip-10-0-138-149 kubenswrapper[2573]: I1030 20:37:17.770132 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/23527d33-4e73-4c36-ba42-7f0a5b7f62c6-host\") pod \"tuned-w9npq\" (UID: \"23527d33-4e73-4c36-ba42-7f0a5b7f62c6\") " pod="openshift-cluster-node-tuning-operator/tuned-w9npq" Oct 30 20:37:17.772208 ip-10-0-138-149 kubenswrapper[2573]: I1030 20:37:17.770211 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/23527d33-4e73-4c36-ba42-7f0a5b7f62c6-tmp\") pod \"tuned-w9npq\" (UID: \"23527d33-4e73-4c36-ba42-7f0a5b7f62c6\") " pod="openshift-cluster-node-tuning-operator/tuned-w9npq" Oct 30 20:37:17.772208 ip-10-0-138-149 kubenswrapper[2573]: I1030 20:37:17.770231 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/022c4cf0-9797-4428-8fa3-9f3112f2ccc5-hostroot\") pod \"multus-dzbhx\" (UID: \"022c4cf0-9797-4428-8fa3-9f3112f2ccc5\") " pod="openshift-multus/multus-dzbhx" Oct 30 20:37:17.772208 ip-10-0-138-149 kubenswrapper[2573]: I1030 20:37:17.770295 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/022c4cf0-9797-4428-8fa3-9f3112f2ccc5-host-run-multus-certs\") pod \"multus-dzbhx\" (UID: \"022c4cf0-9797-4428-8fa3-9f3112f2ccc5\") " pod="openshift-multus/multus-dzbhx" Oct 30 20:37:17.772208 ip-10-0-138-149 kubenswrapper[2573]: I1030 20:37:17.770317 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/b7fa45b4-283a-4439-82b7-778abf2e9deb-serviceca\") pod \"node-ca-tlch5\" (UID: \"b7fa45b4-283a-4439-82b7-778abf2e9deb\") " pod="openshift-image-registry/node-ca-tlch5" Oct 30 20:37:17.773161 ip-10-0-138-149 kubenswrapper[2573]: I1030 20:37:17.770332 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/0d6769f4-dc38-45ac-863b-c75fa672866a-original-pull-secret\") pod \"global-pull-secret-syncer-bcmwc\" (UID: \"0d6769f4-dc38-45ac-863b-c75fa672866a\") " pod="kube-system/global-pull-secret-syncer-bcmwc" Oct 30 20:37:17.773161 ip-10-0-138-149 kubenswrapper[2573]: I1030 20:37:17.770370 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/662e676d-c357-4ac8-a813-d34096051aff-sys-fs\") pod \"aws-ebs-csi-driver-node-svf4h\" (UID: \"662e676d-c357-4ac8-a813-d34096051aff\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-svf4h" Oct 30 20:37:17.773161 ip-10-0-138-149 kubenswrapper[2573]: I1030 20:37:17.770425 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/022c4cf0-9797-4428-8fa3-9f3112f2ccc5-host-run-multus-certs\") pod \"multus-dzbhx\" (UID: \"022c4cf0-9797-4428-8fa3-9f3112f2ccc5\") " pod="openshift-multus/multus-dzbhx" Oct 30 20:37:17.773161 ip-10-0-138-149 kubenswrapper[2573]: I1030 20:37:17.770492 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/23527d33-4e73-4c36-ba42-7f0a5b7f62c6-host\") pod \"tuned-w9npq\" (UID: \"23527d33-4e73-4c36-ba42-7f0a5b7f62c6\") " pod="openshift-cluster-node-tuning-operator/tuned-w9npq" Oct 30 20:37:17.773161 ip-10-0-138-149 kubenswrapper[2573]: I1030 20:37:17.770493 2573 swap_util.go:78] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Oct 30 20:37:17.773161 ip-10-0-138-149 kubenswrapper[2573]: I1030 20:37:17.770529 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/022c4cf0-9797-4428-8fa3-9f3112f2ccc5-hostroot\") pod \"multus-dzbhx\" (UID: \"022c4cf0-9797-4428-8fa3-9f3112f2ccc5\") " pod="openshift-multus/multus-dzbhx" Oct 30 20:37:17.773161 ip-10-0-138-149 kubenswrapper[2573]: I1030 20:37:17.770557 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/022c4cf0-9797-4428-8fa3-9f3112f2ccc5-multus-daemon-config\") pod \"multus-dzbhx\" (UID: \"022c4cf0-9797-4428-8fa3-9f3112f2ccc5\") " pod="openshift-multus/multus-dzbhx" Oct 30 20:37:17.773161 ip-10-0-138-149 kubenswrapper[2573]: I1030 20:37:17.770588 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/c2ff359c-cf92-4a5b-8610-f6eb41acbe73-ovnkube-script-lib\") pod \"ovnkube-node-7x9jg\" (UID: \"c2ff359c-cf92-4a5b-8610-f6eb41acbe73\") " pod="openshift-ovn-kubernetes/ovnkube-node-7x9jg" Oct 30 20:37:17.773161 ip-10-0-138-149 kubenswrapper[2573]: I1030 20:37:17.770691 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/23527d33-4e73-4c36-ba42-7f0a5b7f62c6-etc-systemd\") pod \"tuned-w9npq\" (UID: \"23527d33-4e73-4c36-ba42-7f0a5b7f62c6\") " pod="openshift-cluster-node-tuning-operator/tuned-w9npq" Oct 30 20:37:17.773161 ip-10-0-138-149 kubenswrapper[2573]: I1030 20:37:17.770729 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/022c4cf0-9797-4428-8fa3-9f3112f2ccc5-system-cni-dir\") pod \"multus-dzbhx\" (UID: \"022c4cf0-9797-4428-8fa3-9f3112f2ccc5\") " pod="openshift-multus/multus-dzbhx" Oct 30 20:37:17.773161 ip-10-0-138-149 kubenswrapper[2573]: I1030 20:37:17.770763 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/022c4cf0-9797-4428-8fa3-9f3112f2ccc5-multus-socket-dir-parent\") pod \"multus-dzbhx\" (UID: \"022c4cf0-9797-4428-8fa3-9f3112f2ccc5\") " pod="openshift-multus/multus-dzbhx" Oct 30 20:37:17.773161 ip-10-0-138-149 kubenswrapper[2573]: I1030 20:37:17.770828 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/022c4cf0-9797-4428-8fa3-9f3112f2ccc5-host-var-lib-kubelet\") pod \"multus-dzbhx\" (UID: \"022c4cf0-9797-4428-8fa3-9f3112f2ccc5\") " pod="openshift-multus/multus-dzbhx" Oct 30 20:37:17.773161 ip-10-0-138-149 kubenswrapper[2573]: I1030 20:37:17.770850 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/022c4cf0-9797-4428-8fa3-9f3112f2ccc5-multus-socket-dir-parent\") pod \"multus-dzbhx\" (UID: \"022c4cf0-9797-4428-8fa3-9f3112f2ccc5\") " pod="openshift-multus/multus-dzbhx" Oct 30 20:37:17.773161 ip-10-0-138-149 kubenswrapper[2573]: I1030 20:37:17.770866 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/910c0d9f-c79b-4839-9a1b-f2d278d11fdb-system-cni-dir\") pod \"multus-additional-cni-plugins-blccx\" (UID: \"910c0d9f-c79b-4839-9a1b-f2d278d11fdb\") " pod="openshift-multus/multus-additional-cni-plugins-blccx" Oct 30 20:37:17.773161 ip-10-0-138-149 kubenswrapper[2573]: I1030 20:37:17.770897 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-24qwx\" (UniqueName: \"kubernetes.io/projected/c2ff359c-cf92-4a5b-8610-f6eb41acbe73-kube-api-access-24qwx\") pod \"ovnkube-node-7x9jg\" (UID: \"c2ff359c-cf92-4a5b-8610-f6eb41acbe73\") " pod="openshift-ovn-kubernetes/ovnkube-node-7x9jg" Oct 30 20:37:17.773161 ip-10-0-138-149 kubenswrapper[2573]: I1030 20:37:17.770919 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/910c0d9f-c79b-4839-9a1b-f2d278d11fdb-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-blccx\" (UID: \"910c0d9f-c79b-4839-9a1b-f2d278d11fdb\") " pod="openshift-multus/multus-additional-cni-plugins-blccx" Oct 30 20:37:17.773161 ip-10-0-138-149 kubenswrapper[2573]: I1030 20:37:17.770927 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/23527d33-4e73-4c36-ba42-7f0a5b7f62c6-etc-systemd\") pod \"tuned-w9npq\" (UID: \"23527d33-4e73-4c36-ba42-7f0a5b7f62c6\") " pod="openshift-cluster-node-tuning-operator/tuned-w9npq" Oct 30 20:37:17.773161 ip-10-0-138-149 kubenswrapper[2573]: I1030 20:37:17.770940 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/c2ff359c-cf92-4a5b-8610-f6eb41acbe73-host-run-netns\") pod \"ovnkube-node-7x9jg\" (UID: \"c2ff359c-cf92-4a5b-8610-f6eb41acbe73\") " pod="openshift-ovn-kubernetes/ovnkube-node-7x9jg" Oct 30 20:37:17.773974 ip-10-0-138-149 kubenswrapper[2573]: I1030 20:37:17.771009 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/022c4cf0-9797-4428-8fa3-9f3112f2ccc5-host-var-lib-kubelet\") pod \"multus-dzbhx\" (UID: \"022c4cf0-9797-4428-8fa3-9f3112f2ccc5\") " pod="openshift-multus/multus-dzbhx" Oct 30 20:37:17.773974 ip-10-0-138-149 kubenswrapper[2573]: I1030 20:37:17.771106 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/c2ff359c-cf92-4a5b-8610-f6eb41acbe73-ovnkube-script-lib\") pod \"ovnkube-node-7x9jg\" (UID: \"c2ff359c-cf92-4a5b-8610-f6eb41acbe73\") " pod="openshift-ovn-kubernetes/ovnkube-node-7x9jg" Oct 30 20:37:17.773974 ip-10-0-138-149 kubenswrapper[2573]: I1030 20:37:17.771149 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/022c4cf0-9797-4428-8fa3-9f3112f2ccc5-system-cni-dir\") pod \"multus-dzbhx\" (UID: \"022c4cf0-9797-4428-8fa3-9f3112f2ccc5\") " pod="openshift-multus/multus-dzbhx" Oct 30 20:37:17.773974 ip-10-0-138-149 kubenswrapper[2573]: I1030 20:37:17.771187 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/c2ff359c-cf92-4a5b-8610-f6eb41acbe73-host-run-netns\") pod \"ovnkube-node-7x9jg\" (UID: \"c2ff359c-cf92-4a5b-8610-f6eb41acbe73\") " pod="openshift-ovn-kubernetes/ovnkube-node-7x9jg" Oct 30 20:37:17.773974 ip-10-0-138-149 kubenswrapper[2573]: I1030 20:37:17.771228 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/022c4cf0-9797-4428-8fa3-9f3112f2ccc5-multus-daemon-config\") pod \"multus-dzbhx\" (UID: \"022c4cf0-9797-4428-8fa3-9f3112f2ccc5\") " pod="openshift-multus/multus-dzbhx" Oct 30 20:37:17.773974 ip-10-0-138-149 kubenswrapper[2573]: I1030 20:37:17.771255 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/c2ff359c-cf92-4a5b-8610-f6eb41acbe73-systemd-units\") pod \"ovnkube-node-7x9jg\" (UID: \"c2ff359c-cf92-4a5b-8610-f6eb41acbe73\") " pod="openshift-ovn-kubernetes/ovnkube-node-7x9jg" Oct 30 20:37:17.773974 ip-10-0-138-149 kubenswrapper[2573]: I1030 20:37:17.773858 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/23527d33-4e73-4c36-ba42-7f0a5b7f62c6-etc-tuned\") pod \"tuned-w9npq\" (UID: \"23527d33-4e73-4c36-ba42-7f0a5b7f62c6\") " pod="openshift-cluster-node-tuning-operator/tuned-w9npq" Oct 30 20:37:17.773974 ip-10-0-138-149 kubenswrapper[2573]: I1030 20:37:17.773894 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/23527d33-4e73-4c36-ba42-7f0a5b7f62c6-tmp\") pod \"tuned-w9npq\" (UID: \"23527d33-4e73-4c36-ba42-7f0a5b7f62c6\") " pod="openshift-cluster-node-tuning-operator/tuned-w9npq" Oct 30 20:37:17.774220 ip-10-0-138-149 kubenswrapper[2573]: I1030 20:37:17.774071 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/c2ff359c-cf92-4a5b-8610-f6eb41acbe73-ovn-node-metrics-cert\") pod \"ovnkube-node-7x9jg\" (UID: \"c2ff359c-cf92-4a5b-8610-f6eb41acbe73\") " pod="openshift-ovn-kubernetes/ovnkube-node-7x9jg" Oct 30 20:37:17.828517 ip-10-0-138-149 kubenswrapper[2573]: E1030 20:37:17.828495 2573 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Oct 30 20:37:17.828517 ip-10-0-138-149 kubenswrapper[2573]: E1030 20:37:17.828514 2573 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Oct 30 20:37:17.828692 ip-10-0-138-149 kubenswrapper[2573]: E1030 20:37:17.828525 2573 projected.go:196] Error preparing data for projected volume kube-api-access-bc9kz for pod openshift-network-diagnostics/network-check-target-ws655: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 20:37:17.828692 ip-10-0-138-149 kubenswrapper[2573]: E1030 20:37:17.828595 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/567a72f6-035d-4c61-acaf-f46008029168-kube-api-access-bc9kz podName:567a72f6-035d-4c61-acaf-f46008029168 nodeName:}" failed. No retries permitted until 2026-10-30 20:37:18.328566024 +0000 UTC m=+3.141017341 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-bc9kz" (UniqueName: "kubernetes.io/projected/567a72f6-035d-4c61-acaf-f46008029168-kube-api-access-bc9kz") pod "network-check-target-ws655" (UID: "567a72f6-035d-4c61-acaf-f46008029168") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 20:37:17.830397 ip-10-0-138-149 kubenswrapper[2573]: I1030 20:37:17.830373 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-crg8q\" (UniqueName: \"kubernetes.io/projected/b7fa45b4-283a-4439-82b7-778abf2e9deb-kube-api-access-crg8q\") pod \"node-ca-tlch5\" (UID: \"b7fa45b4-283a-4439-82b7-778abf2e9deb\") " pod="openshift-image-registry/node-ca-tlch5" Oct 30 20:37:17.831133 ip-10-0-138-149 kubenswrapper[2573]: I1030 20:37:17.831113 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qjn9k\" (UniqueName: \"kubernetes.io/projected/022c4cf0-9797-4428-8fa3-9f3112f2ccc5-kube-api-access-qjn9k\") pod \"multus-dzbhx\" (UID: \"022c4cf0-9797-4428-8fa3-9f3112f2ccc5\") " pod="openshift-multus/multus-dzbhx" Oct 30 20:37:17.831221 ip-10-0-138-149 kubenswrapper[2573]: I1030 20:37:17.831150 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xcs6c\" (UniqueName: \"kubernetes.io/projected/23527d33-4e73-4c36-ba42-7f0a5b7f62c6-kube-api-access-xcs6c\") pod \"tuned-w9npq\" (UID: \"23527d33-4e73-4c36-ba42-7f0a5b7f62c6\") " pod="openshift-cluster-node-tuning-operator/tuned-w9npq" Oct 30 20:37:17.831221 ip-10-0-138-149 kubenswrapper[2573]: I1030 20:37:17.831211 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-28zhr\" (UniqueName: \"kubernetes.io/projected/e886897a-c6a3-414f-875f-aa112541b608-kube-api-access-28zhr\") pod \"network-metrics-daemon-nhs9k\" (UID: \"e886897a-c6a3-414f-875f-aa112541b608\") " pod="openshift-multus/network-metrics-daemon-nhs9k" Oct 30 20:37:17.832716 ip-10-0-138-149 kubenswrapper[2573]: I1030 20:37:17.832697 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-24qwx\" (UniqueName: \"kubernetes.io/projected/c2ff359c-cf92-4a5b-8610-f6eb41acbe73-kube-api-access-24qwx\") pod \"ovnkube-node-7x9jg\" (UID: \"c2ff359c-cf92-4a5b-8610-f6eb41acbe73\") " pod="openshift-ovn-kubernetes/ovnkube-node-7x9jg" Oct 30 20:37:17.871548 ip-10-0-138-149 kubenswrapper[2573]: I1030 20:37:17.871528 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/666b6e91-4a60-45d9-9e3e-877867ded7fc-tmp-dir\") pod \"node-resolver-nv2rk\" (UID: \"666b6e91-4a60-45d9-9e3e-877867ded7fc\") " pod="openshift-dns/node-resolver-nv2rk" Oct 30 20:37:17.871636 ip-10-0-138-149 kubenswrapper[2573]: I1030 20:37:17.871554 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2w5c8\" (UniqueName: \"kubernetes.io/projected/666b6e91-4a60-45d9-9e3e-877867ded7fc-kube-api-access-2w5c8\") pod \"node-resolver-nv2rk\" (UID: \"666b6e91-4a60-45d9-9e3e-877867ded7fc\") " pod="openshift-dns/node-resolver-nv2rk" Oct 30 20:37:17.871636 ip-10-0-138-149 kubenswrapper[2573]: I1030 20:37:17.871573 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/662e676d-c357-4ac8-a813-d34096051aff-registration-dir\") pod \"aws-ebs-csi-driver-node-svf4h\" (UID: \"662e676d-c357-4ac8-a813-d34096051aff\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-svf4h" Oct 30 20:37:17.871636 ip-10-0-138-149 kubenswrapper[2573]: I1030 20:37:17.871599 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/910c0d9f-c79b-4839-9a1b-f2d278d11fdb-cnibin\") pod \"multus-additional-cni-plugins-blccx\" (UID: \"910c0d9f-c79b-4839-9a1b-f2d278d11fdb\") " pod="openshift-multus/multus-additional-cni-plugins-blccx" Oct 30 20:37:17.871636 ip-10-0-138-149 kubenswrapper[2573]: I1030 20:37:17.871614 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/affdfb88-e208-4ca1-843d-efc8384c27a1-iptables-alerter-script\") pod \"iptables-alerter-kvq2l\" (UID: \"affdfb88-e208-4ca1-843d-efc8384c27a1\") " pod="openshift-network-operator/iptables-alerter-kvq2l" Oct 30 20:37:17.871636 ip-10-0-138-149 kubenswrapper[2573]: I1030 20:37:17.871630 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/662e676d-c357-4ac8-a813-d34096051aff-device-dir\") pod \"aws-ebs-csi-driver-node-svf4h\" (UID: \"662e676d-c357-4ac8-a813-d34096051aff\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-svf4h" Oct 30 20:37:17.871846 ip-10-0-138-149 kubenswrapper[2573]: I1030 20:37:17.871645 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/affdfb88-e208-4ca1-843d-efc8384c27a1-host-slash\") pod \"iptables-alerter-kvq2l\" (UID: \"affdfb88-e208-4ca1-843d-efc8384c27a1\") " pod="openshift-network-operator/iptables-alerter-kvq2l" Oct 30 20:37:17.871846 ip-10-0-138-149 kubenswrapper[2573]: I1030 20:37:17.871665 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/662e676d-c357-4ac8-a813-d34096051aff-kubelet-dir\") pod \"aws-ebs-csi-driver-node-svf4h\" (UID: \"662e676d-c357-4ac8-a813-d34096051aff\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-svf4h" Oct 30 20:37:17.871846 ip-10-0-138-149 kubenswrapper[2573]: I1030 20:37:17.871722 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/662e676d-c357-4ac8-a813-d34096051aff-registration-dir\") pod \"aws-ebs-csi-driver-node-svf4h\" (UID: \"662e676d-c357-4ac8-a813-d34096051aff\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-svf4h" Oct 30 20:37:17.871846 ip-10-0-138-149 kubenswrapper[2573]: I1030 20:37:17.871729 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/662e676d-c357-4ac8-a813-d34096051aff-kubelet-dir\") pod \"aws-ebs-csi-driver-node-svf4h\" (UID: \"662e676d-c357-4ac8-a813-d34096051aff\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-svf4h" Oct 30 20:37:17.871846 ip-10-0-138-149 kubenswrapper[2573]: I1030 20:37:17.871766 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/662e676d-c357-4ac8-a813-d34096051aff-device-dir\") pod \"aws-ebs-csi-driver-node-svf4h\" (UID: \"662e676d-c357-4ac8-a813-d34096051aff\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-svf4h" Oct 30 20:37:17.871846 ip-10-0-138-149 kubenswrapper[2573]: I1030 20:37:17.871778 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/affdfb88-e208-4ca1-843d-efc8384c27a1-host-slash\") pod \"iptables-alerter-kvq2l\" (UID: \"affdfb88-e208-4ca1-843d-efc8384c27a1\") " pod="openshift-network-operator/iptables-alerter-kvq2l" Oct 30 20:37:17.872101 ip-10-0-138-149 kubenswrapper[2573]: I1030 20:37:17.871921 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/910c0d9f-c79b-4839-9a1b-f2d278d11fdb-cnibin\") pod \"multus-additional-cni-plugins-blccx\" (UID: \"910c0d9f-c79b-4839-9a1b-f2d278d11fdb\") " pod="openshift-multus/multus-additional-cni-plugins-blccx" Oct 30 20:37:17.872101 ip-10-0-138-149 kubenswrapper[2573]: I1030 20:37:17.871943 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/497c44c4-2ece-4009-b88c-6ecfa9920110-konnectivity-ca\") pod \"konnectivity-agent-cjrkj\" (UID: \"497c44c4-2ece-4009-b88c-6ecfa9920110\") " pod="kube-system/konnectivity-agent-cjrkj" Oct 30 20:37:17.872101 ip-10-0-138-149 kubenswrapper[2573]: I1030 20:37:17.871959 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/666b6e91-4a60-45d9-9e3e-877867ded7fc-tmp-dir\") pod \"node-resolver-nv2rk\" (UID: \"666b6e91-4a60-45d9-9e3e-877867ded7fc\") " pod="openshift-dns/node-resolver-nv2rk" Oct 30 20:37:17.872101 ip-10-0-138-149 kubenswrapper[2573]: I1030 20:37:17.871964 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/0d6769f4-dc38-45ac-863b-c75fa672866a-original-pull-secret\") pod \"global-pull-secret-syncer-bcmwc\" (UID: \"0d6769f4-dc38-45ac-863b-c75fa672866a\") " pod="kube-system/global-pull-secret-syncer-bcmwc" Oct 30 20:37:17.872101 ip-10-0-138-149 kubenswrapper[2573]: E1030 20:37:17.872020 2573 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Oct 30 20:37:17.872101 ip-10-0-138-149 kubenswrapper[2573]: I1030 20:37:17.872053 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/662e676d-c357-4ac8-a813-d34096051aff-sys-fs\") pod \"aws-ebs-csi-driver-node-svf4h\" (UID: \"662e676d-c357-4ac8-a813-d34096051aff\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-svf4h" Oct 30 20:37:17.872101 ip-10-0-138-149 kubenswrapper[2573]: E1030 20:37:17.872074 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0d6769f4-dc38-45ac-863b-c75fa672866a-original-pull-secret podName:0d6769f4-dc38-45ac-863b-c75fa672866a nodeName:}" failed. No retries permitted until 2026-10-30 20:37:18.372061462 +0000 UTC m=+3.184512750 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/0d6769f4-dc38-45ac-863b-c75fa672866a-original-pull-secret") pod "global-pull-secret-syncer-bcmwc" (UID: "0d6769f4-dc38-45ac-863b-c75fa672866a") : object "kube-system"/"original-pull-secret" not registered Oct 30 20:37:17.872101 ip-10-0-138-149 kubenswrapper[2573]: I1030 20:37:17.872095 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/910c0d9f-c79b-4839-9a1b-f2d278d11fdb-system-cni-dir\") pod \"multus-additional-cni-plugins-blccx\" (UID: \"910c0d9f-c79b-4839-9a1b-f2d278d11fdb\") " pod="openshift-multus/multus-additional-cni-plugins-blccx" Oct 30 20:37:17.872101 ip-10-0-138-149 kubenswrapper[2573]: I1030 20:37:17.872096 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/662e676d-c357-4ac8-a813-d34096051aff-sys-fs\") pod \"aws-ebs-csi-driver-node-svf4h\" (UID: \"662e676d-c357-4ac8-a813-d34096051aff\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-svf4h" Oct 30 20:37:17.872553 ip-10-0-138-149 kubenswrapper[2573]: I1030 20:37:17.872114 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/910c0d9f-c79b-4839-9a1b-f2d278d11fdb-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-blccx\" (UID: \"910c0d9f-c79b-4839-9a1b-f2d278d11fdb\") " pod="openshift-multus/multus-additional-cni-plugins-blccx" Oct 30 20:37:17.872553 ip-10-0-138-149 kubenswrapper[2573]: I1030 20:37:17.872134 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/662e676d-c357-4ac8-a813-d34096051aff-etc-selinux\") pod \"aws-ebs-csi-driver-node-svf4h\" (UID: \"662e676d-c357-4ac8-a813-d34096051aff\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-svf4h" Oct 30 20:37:17.872553 ip-10-0-138-149 kubenswrapper[2573]: I1030 20:37:17.872136 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/910c0d9f-c79b-4839-9a1b-f2d278d11fdb-system-cni-dir\") pod \"multus-additional-cni-plugins-blccx\" (UID: \"910c0d9f-c79b-4839-9a1b-f2d278d11fdb\") " pod="openshift-multus/multus-additional-cni-plugins-blccx" Oct 30 20:37:17.872553 ip-10-0-138-149 kubenswrapper[2573]: I1030 20:37:17.872157 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/910c0d9f-c79b-4839-9a1b-f2d278d11fdb-tuning-conf-dir\") pod \"multus-additional-cni-plugins-blccx\" (UID: \"910c0d9f-c79b-4839-9a1b-f2d278d11fdb\") " pod="openshift-multus/multus-additional-cni-plugins-blccx" Oct 30 20:37:17.872553 ip-10-0-138-149 kubenswrapper[2573]: I1030 20:37:17.872175 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/affdfb88-e208-4ca1-843d-efc8384c27a1-iptables-alerter-script\") pod \"iptables-alerter-kvq2l\" (UID: \"affdfb88-e208-4ca1-843d-efc8384c27a1\") " pod="openshift-network-operator/iptables-alerter-kvq2l" Oct 30 20:37:17.872553 ip-10-0-138-149 kubenswrapper[2573]: I1030 20:37:17.872178 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/910c0d9f-c79b-4839-9a1b-f2d278d11fdb-cni-binary-copy\") pod \"multus-additional-cni-plugins-blccx\" (UID: \"910c0d9f-c79b-4839-9a1b-f2d278d11fdb\") " pod="openshift-multus/multus-additional-cni-plugins-blccx" Oct 30 20:37:17.872553 ip-10-0-138-149 kubenswrapper[2573]: I1030 20:37:17.872223 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qs9cm\" (UniqueName: \"kubernetes.io/projected/910c0d9f-c79b-4839-9a1b-f2d278d11fdb-kube-api-access-qs9cm\") pod \"multus-additional-cni-plugins-blccx\" (UID: \"910c0d9f-c79b-4839-9a1b-f2d278d11fdb\") " pod="openshift-multus/multus-additional-cni-plugins-blccx" Oct 30 20:37:17.872553 ip-10-0-138-149 kubenswrapper[2573]: I1030 20:37:17.872255 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/0d6769f4-dc38-45ac-863b-c75fa672866a-dbus\") pod \"global-pull-secret-syncer-bcmwc\" (UID: \"0d6769f4-dc38-45ac-863b-c75fa672866a\") " pod="kube-system/global-pull-secret-syncer-bcmwc" Oct 30 20:37:17.872553 ip-10-0-138-149 kubenswrapper[2573]: I1030 20:37:17.872281 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k82pb\" (UniqueName: \"kubernetes.io/projected/662e676d-c357-4ac8-a813-d34096051aff-kube-api-access-k82pb\") pod \"aws-ebs-csi-driver-node-svf4h\" (UID: \"662e676d-c357-4ac8-a813-d34096051aff\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-svf4h" Oct 30 20:37:17.872553 ip-10-0-138-149 kubenswrapper[2573]: I1030 20:37:17.872308 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t5psr\" (UniqueName: \"kubernetes.io/projected/affdfb88-e208-4ca1-843d-efc8384c27a1-kube-api-access-t5psr\") pod \"iptables-alerter-kvq2l\" (UID: \"affdfb88-e208-4ca1-843d-efc8384c27a1\") " pod="openshift-network-operator/iptables-alerter-kvq2l" Oct 30 20:37:17.872553 ip-10-0-138-149 kubenswrapper[2573]: I1030 20:37:17.872338 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/497c44c4-2ece-4009-b88c-6ecfa9920110-agent-certs\") pod \"konnectivity-agent-cjrkj\" (UID: \"497c44c4-2ece-4009-b88c-6ecfa9920110\") " pod="kube-system/konnectivity-agent-cjrkj" Oct 30 20:37:17.872553 ip-10-0-138-149 kubenswrapper[2573]: I1030 20:37:17.872349 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/0d6769f4-dc38-45ac-863b-c75fa672866a-dbus\") pod \"global-pull-secret-syncer-bcmwc\" (UID: \"0d6769f4-dc38-45ac-863b-c75fa672866a\") " pod="kube-system/global-pull-secret-syncer-bcmwc" Oct 30 20:37:17.872553 ip-10-0-138-149 kubenswrapper[2573]: I1030 20:37:17.872361 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/0d6769f4-dc38-45ac-863b-c75fa672866a-kubelet-config\") pod \"global-pull-secret-syncer-bcmwc\" (UID: \"0d6769f4-dc38-45ac-863b-c75fa672866a\") " pod="kube-system/global-pull-secret-syncer-bcmwc" Oct 30 20:37:17.872553 ip-10-0-138-149 kubenswrapper[2573]: I1030 20:37:17.872404 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/910c0d9f-c79b-4839-9a1b-f2d278d11fdb-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-blccx\" (UID: \"910c0d9f-c79b-4839-9a1b-f2d278d11fdb\") " pod="openshift-multus/multus-additional-cni-plugins-blccx" Oct 30 20:37:17.872553 ip-10-0-138-149 kubenswrapper[2573]: I1030 20:37:17.872430 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/666b6e91-4a60-45d9-9e3e-877867ded7fc-hosts-file\") pod \"node-resolver-nv2rk\" (UID: \"666b6e91-4a60-45d9-9e3e-877867ded7fc\") " pod="openshift-dns/node-resolver-nv2rk" Oct 30 20:37:17.872553 ip-10-0-138-149 kubenswrapper[2573]: I1030 20:37:17.872455 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/662e676d-c357-4ac8-a813-d34096051aff-socket-dir\") pod \"aws-ebs-csi-driver-node-svf4h\" (UID: \"662e676d-c357-4ac8-a813-d34096051aff\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-svf4h" Oct 30 20:37:17.873106 ip-10-0-138-149 kubenswrapper[2573]: I1030 20:37:17.872360 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/662e676d-c357-4ac8-a813-d34096051aff-etc-selinux\") pod \"aws-ebs-csi-driver-node-svf4h\" (UID: \"662e676d-c357-4ac8-a813-d34096051aff\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-svf4h" Oct 30 20:37:17.873106 ip-10-0-138-149 kubenswrapper[2573]: I1030 20:37:17.872513 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/910c0d9f-c79b-4839-9a1b-f2d278d11fdb-tuning-conf-dir\") pod \"multus-additional-cni-plugins-blccx\" (UID: \"910c0d9f-c79b-4839-9a1b-f2d278d11fdb\") " pod="openshift-multus/multus-additional-cni-plugins-blccx" Oct 30 20:37:17.873106 ip-10-0-138-149 kubenswrapper[2573]: I1030 20:37:17.872539 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/497c44c4-2ece-4009-b88c-6ecfa9920110-konnectivity-ca\") pod \"konnectivity-agent-cjrkj\" (UID: \"497c44c4-2ece-4009-b88c-6ecfa9920110\") " pod="kube-system/konnectivity-agent-cjrkj" Oct 30 20:37:17.873106 ip-10-0-138-149 kubenswrapper[2573]: I1030 20:37:17.872542 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/0d6769f4-dc38-45ac-863b-c75fa672866a-kubelet-config\") pod \"global-pull-secret-syncer-bcmwc\" (UID: \"0d6769f4-dc38-45ac-863b-c75fa672866a\") " pod="kube-system/global-pull-secret-syncer-bcmwc" Oct 30 20:37:17.873106 ip-10-0-138-149 kubenswrapper[2573]: I1030 20:37:17.872584 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/666b6e91-4a60-45d9-9e3e-877867ded7fc-hosts-file\") pod \"node-resolver-nv2rk\" (UID: \"666b6e91-4a60-45d9-9e3e-877867ded7fc\") " pod="openshift-dns/node-resolver-nv2rk" Oct 30 20:37:17.873106 ip-10-0-138-149 kubenswrapper[2573]: I1030 20:37:17.872726 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/910c0d9f-c79b-4839-9a1b-f2d278d11fdb-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-blccx\" (UID: \"910c0d9f-c79b-4839-9a1b-f2d278d11fdb\") " pod="openshift-multus/multus-additional-cni-plugins-blccx" Oct 30 20:37:17.873523 ip-10-0-138-149 kubenswrapper[2573]: I1030 20:37:17.873504 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/910c0d9f-c79b-4839-9a1b-f2d278d11fdb-cni-binary-copy\") pod \"multus-additional-cni-plugins-blccx\" (UID: \"910c0d9f-c79b-4839-9a1b-f2d278d11fdb\") " pod="openshift-multus/multus-additional-cni-plugins-blccx" Oct 30 20:37:17.873562 ip-10-0-138-149 kubenswrapper[2573]: I1030 20:37:17.873514 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/662e676d-c357-4ac8-a813-d34096051aff-socket-dir\") pod \"aws-ebs-csi-driver-node-svf4h\" (UID: \"662e676d-c357-4ac8-a813-d34096051aff\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-svf4h" Oct 30 20:37:17.873781 ip-10-0-138-149 kubenswrapper[2573]: I1030 20:37:17.873764 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/910c0d9f-c79b-4839-9a1b-f2d278d11fdb-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-blccx\" (UID: \"910c0d9f-c79b-4839-9a1b-f2d278d11fdb\") " pod="openshift-multus/multus-additional-cni-plugins-blccx" Oct 30 20:37:17.875285 ip-10-0-138-149 kubenswrapper[2573]: I1030 20:37:17.875263 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/497c44c4-2ece-4009-b88c-6ecfa9920110-agent-certs\") pod \"konnectivity-agent-cjrkj\" (UID: \"497c44c4-2ece-4009-b88c-6ecfa9920110\") " pod="kube-system/konnectivity-agent-cjrkj" Oct 30 20:37:17.933069 ip-10-0-138-149 kubenswrapper[2573]: I1030 20:37:17.933041 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qs9cm\" (UniqueName: \"kubernetes.io/projected/910c0d9f-c79b-4839-9a1b-f2d278d11fdb-kube-api-access-qs9cm\") pod \"multus-additional-cni-plugins-blccx\" (UID: \"910c0d9f-c79b-4839-9a1b-f2d278d11fdb\") " pod="openshift-multus/multus-additional-cni-plugins-blccx" Oct 30 20:37:17.934928 ip-10-0-138-149 kubenswrapper[2573]: I1030 20:37:17.934901 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t5psr\" (UniqueName: \"kubernetes.io/projected/affdfb88-e208-4ca1-843d-efc8384c27a1-kube-api-access-t5psr\") pod \"iptables-alerter-kvq2l\" (UID: \"affdfb88-e208-4ca1-843d-efc8384c27a1\") " pod="openshift-network-operator/iptables-alerter-kvq2l" Oct 30 20:37:17.935266 ip-10-0-138-149 kubenswrapper[2573]: I1030 20:37:17.935243 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k82pb\" (UniqueName: \"kubernetes.io/projected/662e676d-c357-4ac8-a813-d34096051aff-kube-api-access-k82pb\") pod \"aws-ebs-csi-driver-node-svf4h\" (UID: \"662e676d-c357-4ac8-a813-d34096051aff\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-svf4h" Oct 30 20:37:17.935369 ip-10-0-138-149 kubenswrapper[2573]: I1030 20:37:17.935355 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2w5c8\" (UniqueName: \"kubernetes.io/projected/666b6e91-4a60-45d9-9e3e-877867ded7fc-kube-api-access-2w5c8\") pod \"node-resolver-nv2rk\" (UID: \"666b6e91-4a60-45d9-9e3e-877867ded7fc\") " pod="openshift-dns/node-resolver-nv2rk" Oct 30 20:37:17.953360 ip-10-0-138-149 kubenswrapper[2573]: I1030 20:37:17.953342 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-tlch5" Oct 30 20:37:17.959789 ip-10-0-138-149 kubenswrapper[2573]: I1030 20:37:17.959768 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-dzbhx" Oct 30 20:37:17.963438 ip-10-0-138-149 kubenswrapper[2573]: W1030 20:37:17.963416 2573 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb7fa45b4_283a_4439_82b7_778abf2e9deb.slice/crio-afaaaa72e718e44097a463278115e2c807138c3ff8eaddbe3af48fc8e2c92322 WatchSource:0}: Error finding container afaaaa72e718e44097a463278115e2c807138c3ff8eaddbe3af48fc8e2c92322: Status 404 returned error can't find the container with id afaaaa72e718e44097a463278115e2c807138c3ff8eaddbe3af48fc8e2c92322 Oct 30 20:37:17.968855 ip-10-0-138-149 kubenswrapper[2573]: W1030 20:37:17.968837 2573 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod022c4cf0_9797_4428_8fa3_9f3112f2ccc5.slice/crio-cc2ac291099ece3de0526ea3ae8a263d9310f204ebe3a96aa41bdfb6b10df725 WatchSource:0}: Error finding container cc2ac291099ece3de0526ea3ae8a263d9310f204ebe3a96aa41bdfb6b10df725: Status 404 returned error can't find the container with id cc2ac291099ece3de0526ea3ae8a263d9310f204ebe3a96aa41bdfb6b10df725 Oct 30 20:37:17.981318 ip-10-0-138-149 kubenswrapper[2573]: I1030 20:37:17.981303 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-w9npq" Oct 30 20:37:17.989840 ip-10-0-138-149 kubenswrapper[2573]: I1030 20:37:17.989822 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-nv2rk" Oct 30 20:37:17.990098 ip-10-0-138-149 kubenswrapper[2573]: W1030 20:37:17.990073 2573 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod23527d33_4e73_4c36_ba42_7f0a5b7f62c6.slice/crio-e71424147ffba20e356af8b7abf5247f2e2aa0582df7a40cb6cc6b3e01ebab9f WatchSource:0}: Error finding container e71424147ffba20e356af8b7abf5247f2e2aa0582df7a40cb6cc6b3e01ebab9f: Status 404 returned error can't find the container with id e71424147ffba20e356af8b7abf5247f2e2aa0582df7a40cb6cc6b3e01ebab9f Oct 30 20:37:17.994024 ip-10-0-138-149 kubenswrapper[2573]: I1030 20:37:17.994005 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-7x9jg" Oct 30 20:37:17.996140 ip-10-0-138-149 kubenswrapper[2573]: I1030 20:37:17.996125 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-kvq2l" Oct 30 20:37:17.998410 ip-10-0-138-149 kubenswrapper[2573]: W1030 20:37:17.998390 2573 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod666b6e91_4a60_45d9_9e3e_877867ded7fc.slice/crio-88e0061ecb608b11dff851012f3eed16592fbdd5de53a886497f7b93664d46f8 WatchSource:0}: Error finding container 88e0061ecb608b11dff851012f3eed16592fbdd5de53a886497f7b93664d46f8: Status 404 returned error can't find the container with id 88e0061ecb608b11dff851012f3eed16592fbdd5de53a886497f7b93664d46f8 Oct 30 20:37:18.003228 ip-10-0-138-149 kubenswrapper[2573]: I1030 20:37:18.003208 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-cjrkj" Oct 30 20:37:18.003463 ip-10-0-138-149 kubenswrapper[2573]: W1030 20:37:18.003447 2573 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc2ff359c_cf92_4a5b_8610_f6eb41acbe73.slice/crio-19be35dbc9b22f4f2936dcf2085288bf8165912b6264f685ef62823add265a29 WatchSource:0}: Error finding container 19be35dbc9b22f4f2936dcf2085288bf8165912b6264f685ef62823add265a29: Status 404 returned error can't find the container with id 19be35dbc9b22f4f2936dcf2085288bf8165912b6264f685ef62823add265a29 Oct 30 20:37:18.006869 ip-10-0-138-149 kubenswrapper[2573]: W1030 20:37:18.006848 2573 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaffdfb88_e208_4ca1_843d_efc8384c27a1.slice/crio-51320e7f03a5a3ba862c18c42484ea5b0fbe760a99611cf3cc11c780cd27bd38 WatchSource:0}: Error finding container 51320e7f03a5a3ba862c18c42484ea5b0fbe760a99611cf3cc11c780cd27bd38: Status 404 returned error can't find the container with id 51320e7f03a5a3ba862c18c42484ea5b0fbe760a99611cf3cc11c780cd27bd38 Oct 30 20:37:18.010043 ip-10-0-138-149 kubenswrapper[2573]: I1030 20:37:18.010025 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-blccx" Oct 30 20:37:18.012436 ip-10-0-138-149 kubenswrapper[2573]: W1030 20:37:18.012418 2573 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod497c44c4_2ece_4009_b88c_6ecfa9920110.slice/crio-10168eee9d7308abc6d1b01ecafd86974dd32d77377945c05b967dedad930adc WatchSource:0}: Error finding container 10168eee9d7308abc6d1b01ecafd86974dd32d77377945c05b967dedad930adc: Status 404 returned error can't find the container with id 10168eee9d7308abc6d1b01ecafd86974dd32d77377945c05b967dedad930adc Oct 30 20:37:18.014526 ip-10-0-138-149 kubenswrapper[2573]: I1030 20:37:18.014511 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-svf4h" Oct 30 20:37:18.019610 ip-10-0-138-149 kubenswrapper[2573]: W1030 20:37:18.019591 2573 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod910c0d9f_c79b_4839_9a1b_f2d278d11fdb.slice/crio-1afe02640a96809248834d4e5ef0671c215576bc3e9d994f39e65cbf46e658e8 WatchSource:0}: Error finding container 1afe02640a96809248834d4e5ef0671c215576bc3e9d994f39e65cbf46e658e8: Status 404 returned error can't find the container with id 1afe02640a96809248834d4e5ef0671c215576bc3e9d994f39e65cbf46e658e8 Oct 30 20:37:18.023628 ip-10-0-138-149 kubenswrapper[2573]: W1030 20:37:18.023610 2573 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod662e676d_c357_4ac8_a813_d34096051aff.slice/crio-3e4aceef3fbbadd826485bb72e82b4999deefbb20d450e2ed6e48e9007593077 WatchSource:0}: Error finding container 3e4aceef3fbbadd826485bb72e82b4999deefbb20d450e2ed6e48e9007593077: Status 404 returned error can't find the container with id 3e4aceef3fbbadd826485bb72e82b4999deefbb20d450e2ed6e48e9007593077 Oct 30 20:37:18.073451 ip-10-0-138-149 kubenswrapper[2573]: I1030 20:37:18.073431 2573 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 20:37:18.275214 ip-10-0-138-149 kubenswrapper[2573]: I1030 20:37:18.275132 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e886897a-c6a3-414f-875f-aa112541b608-metrics-certs\") pod \"network-metrics-daemon-nhs9k\" (UID: \"e886897a-c6a3-414f-875f-aa112541b608\") " pod="openshift-multus/network-metrics-daemon-nhs9k" Oct 30 20:37:18.275379 ip-10-0-138-149 kubenswrapper[2573]: E1030 20:37:18.275270 2573 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 20:37:18.275379 ip-10-0-138-149 kubenswrapper[2573]: E1030 20:37:18.275344 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e886897a-c6a3-414f-875f-aa112541b608-metrics-certs podName:e886897a-c6a3-414f-875f-aa112541b608 nodeName:}" failed. No retries permitted until 2026-10-30 20:37:19.275323662 +0000 UTC m=+4.087774948 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/e886897a-c6a3-414f-875f-aa112541b608-metrics-certs") pod "network-metrics-daemon-nhs9k" (UID: "e886897a-c6a3-414f-875f-aa112541b608") : object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 20:37:18.375647 ip-10-0-138-149 kubenswrapper[2573]: I1030 20:37:18.375608 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/0d6769f4-dc38-45ac-863b-c75fa672866a-original-pull-secret\") pod \"global-pull-secret-syncer-bcmwc\" (UID: \"0d6769f4-dc38-45ac-863b-c75fa672866a\") " pod="kube-system/global-pull-secret-syncer-bcmwc" Oct 30 20:37:18.375840 ip-10-0-138-149 kubenswrapper[2573]: I1030 20:37:18.375660 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bc9kz\" (UniqueName: \"kubernetes.io/projected/567a72f6-035d-4c61-acaf-f46008029168-kube-api-access-bc9kz\") pod \"network-check-target-ws655\" (UID: \"567a72f6-035d-4c61-acaf-f46008029168\") " pod="openshift-network-diagnostics/network-check-target-ws655" Oct 30 20:37:18.375840 ip-10-0-138-149 kubenswrapper[2573]: E1030 20:37:18.375760 2573 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Oct 30 20:37:18.375840 ip-10-0-138-149 kubenswrapper[2573]: E1030 20:37:18.375781 2573 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Oct 30 20:37:18.375840 ip-10-0-138-149 kubenswrapper[2573]: E1030 20:37:18.375795 2573 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Oct 30 20:37:18.375840 ip-10-0-138-149 kubenswrapper[2573]: E1030 20:37:18.375808 2573 projected.go:196] Error preparing data for projected volume kube-api-access-bc9kz for pod openshift-network-diagnostics/network-check-target-ws655: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 20:37:18.375840 ip-10-0-138-149 kubenswrapper[2573]: E1030 20:37:18.375838 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0d6769f4-dc38-45ac-863b-c75fa672866a-original-pull-secret podName:0d6769f4-dc38-45ac-863b-c75fa672866a nodeName:}" failed. No retries permitted until 2026-10-30 20:37:19.375817107 +0000 UTC m=+4.188268401 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/0d6769f4-dc38-45ac-863b-c75fa672866a-original-pull-secret") pod "global-pull-secret-syncer-bcmwc" (UID: "0d6769f4-dc38-45ac-863b-c75fa672866a") : object "kube-system"/"original-pull-secret" not registered Oct 30 20:37:18.376122 ip-10-0-138-149 kubenswrapper[2573]: E1030 20:37:18.375859 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/567a72f6-035d-4c61-acaf-f46008029168-kube-api-access-bc9kz podName:567a72f6-035d-4c61-acaf-f46008029168 nodeName:}" failed. No retries permitted until 2026-10-30 20:37:19.375849804 +0000 UTC m=+4.188301107 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-bc9kz" (UniqueName: "kubernetes.io/projected/567a72f6-035d-4c61-acaf-f46008029168-kube-api-access-bc9kz") pod "network-check-target-ws655" (UID: "567a72f6-035d-4c61-acaf-f46008029168") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 20:37:18.739525 ip-10-0-138-149 kubenswrapper[2573]: I1030 20:37:18.739438 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-svf4h" event={"ID":"662e676d-c357-4ac8-a813-d34096051aff","Type":"ContainerStarted","Data":"3e4aceef3fbbadd826485bb72e82b4999deefbb20d450e2ed6e48e9007593077"} Oct 30 20:37:18.741181 ip-10-0-138-149 kubenswrapper[2573]: I1030 20:37:18.741142 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-blccx" event={"ID":"910c0d9f-c79b-4839-9a1b-f2d278d11fdb","Type":"ContainerStarted","Data":"1afe02640a96809248834d4e5ef0671c215576bc3e9d994f39e65cbf46e658e8"} Oct 30 20:37:18.742636 ip-10-0-138-149 kubenswrapper[2573]: I1030 20:37:18.742609 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-kvq2l" event={"ID":"affdfb88-e208-4ca1-843d-efc8384c27a1","Type":"ContainerStarted","Data":"51320e7f03a5a3ba862c18c42484ea5b0fbe760a99611cf3cc11c780cd27bd38"} Oct 30 20:37:18.744264 ip-10-0-138-149 kubenswrapper[2573]: I1030 20:37:18.744240 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-7x9jg" event={"ID":"c2ff359c-cf92-4a5b-8610-f6eb41acbe73","Type":"ContainerStarted","Data":"19be35dbc9b22f4f2936dcf2085288bf8165912b6264f685ef62823add265a29"} Oct 30 20:37:18.746219 ip-10-0-138-149 kubenswrapper[2573]: I1030 20:37:18.746192 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-cjrkj" event={"ID":"497c44c4-2ece-4009-b88c-6ecfa9920110","Type":"ContainerStarted","Data":"10168eee9d7308abc6d1b01ecafd86974dd32d77377945c05b967dedad930adc"} Oct 30 20:37:18.747497 ip-10-0-138-149 kubenswrapper[2573]: I1030 20:37:18.747471 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-nv2rk" event={"ID":"666b6e91-4a60-45d9-9e3e-877867ded7fc","Type":"ContainerStarted","Data":"88e0061ecb608b11dff851012f3eed16592fbdd5de53a886497f7b93664d46f8"} Oct 30 20:37:18.748819 ip-10-0-138-149 kubenswrapper[2573]: I1030 20:37:18.748775 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-w9npq" event={"ID":"23527d33-4e73-4c36-ba42-7f0a5b7f62c6","Type":"ContainerStarted","Data":"e71424147ffba20e356af8b7abf5247f2e2aa0582df7a40cb6cc6b3e01ebab9f"} Oct 30 20:37:18.749374 ip-10-0-138-149 kubenswrapper[2573]: I1030 20:37:18.749343 2573 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-10-29 20:32:18 +0000 UTC" deadline="2028-04-19 09:45:39.279705466 +0000 UTC" Oct 30 20:37:18.749374 ip-10-0-138-149 kubenswrapper[2573]: I1030 20:37:18.749375 2573 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="12877h8m20.530336726s" Oct 30 20:37:18.749940 ip-10-0-138-149 kubenswrapper[2573]: I1030 20:37:18.749916 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-dzbhx" event={"ID":"022c4cf0-9797-4428-8fa3-9f3112f2ccc5","Type":"ContainerStarted","Data":"cc2ac291099ece3de0526ea3ae8a263d9310f204ebe3a96aa41bdfb6b10df725"} Oct 30 20:37:18.751149 ip-10-0-138-149 kubenswrapper[2573]: I1030 20:37:18.751126 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-tlch5" event={"ID":"b7fa45b4-283a-4439-82b7-778abf2e9deb","Type":"ContainerStarted","Data":"afaaaa72e718e44097a463278115e2c807138c3ff8eaddbe3af48fc8e2c92322"} Oct 30 20:37:19.283265 ip-10-0-138-149 kubenswrapper[2573]: I1030 20:37:19.282999 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e886897a-c6a3-414f-875f-aa112541b608-metrics-certs\") pod \"network-metrics-daemon-nhs9k\" (UID: \"e886897a-c6a3-414f-875f-aa112541b608\") " pod="openshift-multus/network-metrics-daemon-nhs9k" Oct 30 20:37:19.283265 ip-10-0-138-149 kubenswrapper[2573]: E1030 20:37:19.283162 2573 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 20:37:19.283265 ip-10-0-138-149 kubenswrapper[2573]: E1030 20:37:19.283229 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e886897a-c6a3-414f-875f-aa112541b608-metrics-certs podName:e886897a-c6a3-414f-875f-aa112541b608 nodeName:}" failed. No retries permitted until 2026-10-30 20:37:21.283208872 +0000 UTC m=+6.095660164 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/e886897a-c6a3-414f-875f-aa112541b608-metrics-certs") pod "network-metrics-daemon-nhs9k" (UID: "e886897a-c6a3-414f-875f-aa112541b608") : object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 20:37:19.383690 ip-10-0-138-149 kubenswrapper[2573]: I1030 20:37:19.383651 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/0d6769f4-dc38-45ac-863b-c75fa672866a-original-pull-secret\") pod \"global-pull-secret-syncer-bcmwc\" (UID: \"0d6769f4-dc38-45ac-863b-c75fa672866a\") " pod="kube-system/global-pull-secret-syncer-bcmwc" Oct 30 20:37:19.383893 ip-10-0-138-149 kubenswrapper[2573]: I1030 20:37:19.383723 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bc9kz\" (UniqueName: \"kubernetes.io/projected/567a72f6-035d-4c61-acaf-f46008029168-kube-api-access-bc9kz\") pod \"network-check-target-ws655\" (UID: \"567a72f6-035d-4c61-acaf-f46008029168\") " pod="openshift-network-diagnostics/network-check-target-ws655" Oct 30 20:37:19.383965 ip-10-0-138-149 kubenswrapper[2573]: E1030 20:37:19.383903 2573 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Oct 30 20:37:19.383965 ip-10-0-138-149 kubenswrapper[2573]: E1030 20:37:19.383920 2573 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Oct 30 20:37:19.383965 ip-10-0-138-149 kubenswrapper[2573]: E1030 20:37:19.383934 2573 projected.go:196] Error preparing data for projected volume kube-api-access-bc9kz for pod openshift-network-diagnostics/network-check-target-ws655: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 20:37:19.384125 ip-10-0-138-149 kubenswrapper[2573]: E1030 20:37:19.383990 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/567a72f6-035d-4c61-acaf-f46008029168-kube-api-access-bc9kz podName:567a72f6-035d-4c61-acaf-f46008029168 nodeName:}" failed. No retries permitted until 2026-10-30 20:37:21.383970924 +0000 UTC m=+6.196422237 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-bc9kz" (UniqueName: "kubernetes.io/projected/567a72f6-035d-4c61-acaf-f46008029168-kube-api-access-bc9kz") pod "network-check-target-ws655" (UID: "567a72f6-035d-4c61-acaf-f46008029168") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 20:37:19.384125 ip-10-0-138-149 kubenswrapper[2573]: E1030 20:37:19.384061 2573 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Oct 30 20:37:19.384125 ip-10-0-138-149 kubenswrapper[2573]: E1030 20:37:19.384096 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0d6769f4-dc38-45ac-863b-c75fa672866a-original-pull-secret podName:0d6769f4-dc38-45ac-863b-c75fa672866a nodeName:}" failed. No retries permitted until 2026-10-30 20:37:21.384084711 +0000 UTC m=+6.196536002 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/0d6769f4-dc38-45ac-863b-c75fa672866a-original-pull-secret") pod "global-pull-secret-syncer-bcmwc" (UID: "0d6769f4-dc38-45ac-863b-c75fa672866a") : object "kube-system"/"original-pull-secret" not registered Oct 30 20:37:19.733311 ip-10-0-138-149 kubenswrapper[2573]: I1030 20:37:19.733237 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-bcmwc" Oct 30 20:37:19.733462 ip-10-0-138-149 kubenswrapper[2573]: E1030 20:37:19.733379 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-bcmwc" podUID="0d6769f4-dc38-45ac-863b-c75fa672866a" Oct 30 20:37:19.733538 ip-10-0-138-149 kubenswrapper[2573]: I1030 20:37:19.733472 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-nhs9k" Oct 30 20:37:19.733596 ip-10-0-138-149 kubenswrapper[2573]: E1030 20:37:19.733557 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-nhs9k" podUID="e886897a-c6a3-414f-875f-aa112541b608" Oct 30 20:37:19.733666 ip-10-0-138-149 kubenswrapper[2573]: I1030 20:37:19.733614 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ws655" Oct 30 20:37:19.733727 ip-10-0-138-149 kubenswrapper[2573]: E1030 20:37:19.733701 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-ws655" podUID="567a72f6-035d-4c61-acaf-f46008029168" Oct 30 20:37:19.750421 ip-10-0-138-149 kubenswrapper[2573]: I1030 20:37:19.750335 2573 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-10-29 20:32:18 +0000 UTC" deadline="2028-05-07 14:45:33.610802048 +0000 UTC" Oct 30 20:37:19.750421 ip-10-0-138-149 kubenswrapper[2573]: I1030 20:37:19.750390 2573 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13314h8m13.860416418s" Oct 30 20:37:21.302798 ip-10-0-138-149 kubenswrapper[2573]: I1030 20:37:21.302765 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e886897a-c6a3-414f-875f-aa112541b608-metrics-certs\") pod \"network-metrics-daemon-nhs9k\" (UID: \"e886897a-c6a3-414f-875f-aa112541b608\") " pod="openshift-multus/network-metrics-daemon-nhs9k" Oct 30 20:37:21.303167 ip-10-0-138-149 kubenswrapper[2573]: E1030 20:37:21.302910 2573 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 20:37:21.303167 ip-10-0-138-149 kubenswrapper[2573]: E1030 20:37:21.302979 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e886897a-c6a3-414f-875f-aa112541b608-metrics-certs podName:e886897a-c6a3-414f-875f-aa112541b608 nodeName:}" failed. No retries permitted until 2026-10-30 20:37:25.302959541 +0000 UTC m=+10.115410834 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/e886897a-c6a3-414f-875f-aa112541b608-metrics-certs") pod "network-metrics-daemon-nhs9k" (UID: "e886897a-c6a3-414f-875f-aa112541b608") : object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 20:37:21.403318 ip-10-0-138-149 kubenswrapper[2573]: I1030 20:37:21.403084 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/0d6769f4-dc38-45ac-863b-c75fa672866a-original-pull-secret\") pod \"global-pull-secret-syncer-bcmwc\" (UID: \"0d6769f4-dc38-45ac-863b-c75fa672866a\") " pod="kube-system/global-pull-secret-syncer-bcmwc" Oct 30 20:37:21.403455 ip-10-0-138-149 kubenswrapper[2573]: I1030 20:37:21.403385 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bc9kz\" (UniqueName: \"kubernetes.io/projected/567a72f6-035d-4c61-acaf-f46008029168-kube-api-access-bc9kz\") pod \"network-check-target-ws655\" (UID: \"567a72f6-035d-4c61-acaf-f46008029168\") " pod="openshift-network-diagnostics/network-check-target-ws655" Oct 30 20:37:21.403598 ip-10-0-138-149 kubenswrapper[2573]: E1030 20:37:21.403581 2573 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Oct 30 20:37:21.403681 ip-10-0-138-149 kubenswrapper[2573]: E1030 20:37:21.403610 2573 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Oct 30 20:37:21.403681 ip-10-0-138-149 kubenswrapper[2573]: E1030 20:37:21.403638 2573 projected.go:196] Error preparing data for projected volume kube-api-access-bc9kz for pod openshift-network-diagnostics/network-check-target-ws655: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 20:37:21.403786 ip-10-0-138-149 kubenswrapper[2573]: E1030 20:37:21.403692 2573 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Oct 30 20:37:21.403786 ip-10-0-138-149 kubenswrapper[2573]: E1030 20:37:21.403705 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/567a72f6-035d-4c61-acaf-f46008029168-kube-api-access-bc9kz podName:567a72f6-035d-4c61-acaf-f46008029168 nodeName:}" failed. No retries permitted until 2026-10-30 20:37:25.40368438 +0000 UTC m=+10.216135683 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-bc9kz" (UniqueName: "kubernetes.io/projected/567a72f6-035d-4c61-acaf-f46008029168-kube-api-access-bc9kz") pod "network-check-target-ws655" (UID: "567a72f6-035d-4c61-acaf-f46008029168") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 20:37:21.403786 ip-10-0-138-149 kubenswrapper[2573]: E1030 20:37:21.403733 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0d6769f4-dc38-45ac-863b-c75fa672866a-original-pull-secret podName:0d6769f4-dc38-45ac-863b-c75fa672866a nodeName:}" failed. No retries permitted until 2026-10-30 20:37:25.403720705 +0000 UTC m=+10.216172008 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/0d6769f4-dc38-45ac-863b-c75fa672866a-original-pull-secret") pod "global-pull-secret-syncer-bcmwc" (UID: "0d6769f4-dc38-45ac-863b-c75fa672866a") : object "kube-system"/"original-pull-secret" not registered Oct 30 20:37:21.731984 ip-10-0-138-149 kubenswrapper[2573]: I1030 20:37:21.731957 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-bcmwc" Oct 30 20:37:21.732249 ip-10-0-138-149 kubenswrapper[2573]: E1030 20:37:21.732090 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-bcmwc" podUID="0d6769f4-dc38-45ac-863b-c75fa672866a" Oct 30 20:37:21.732249 ip-10-0-138-149 kubenswrapper[2573]: I1030 20:37:21.732155 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-nhs9k" Oct 30 20:37:21.732464 ip-10-0-138-149 kubenswrapper[2573]: E1030 20:37:21.732249 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-nhs9k" podUID="e886897a-c6a3-414f-875f-aa112541b608" Oct 30 20:37:21.732464 ip-10-0-138-149 kubenswrapper[2573]: I1030 20:37:21.732291 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ws655" Oct 30 20:37:21.732464 ip-10-0-138-149 kubenswrapper[2573]: E1030 20:37:21.732359 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-ws655" podUID="567a72f6-035d-4c61-acaf-f46008029168" Oct 30 20:37:21.795520 ip-10-0-138-149 kubenswrapper[2573]: I1030 20:37:21.794527 2573 generic.go:356] "Generic (PLEG): container finished" podID="60d05e7d5fab914587e076383d73f1df" containerID="f87aaaf5fc68a52fc39a2ffb73ec2f494c38bd0f2efbd90e2a2ba5a311044017" exitCode=0 Oct 30 20:37:21.795520 ip-10-0-138-149 kubenswrapper[2573]: I1030 20:37:21.794623 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-149.us-west-1.compute.internal" event={"ID":"60d05e7d5fab914587e076383d73f1df","Type":"ContainerDied","Data":"f87aaaf5fc68a52fc39a2ffb73ec2f494c38bd0f2efbd90e2a2ba5a311044017"} Oct 30 20:37:21.816829 ip-10-0-138-149 kubenswrapper[2573]: I1030 20:37:21.816770 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-138-149.us-west-1.compute.internal" event={"ID":"0d2e0d42ed810f19933d4cd655966ed4","Type":"ContainerStarted","Data":"189448b3eda32f8fbaa0c723099f6d88ab5b649ccb1077284ac1384b1e6b6fcc"} Oct 30 20:37:22.039378 ip-10-0-138-149 kubenswrapper[2573]: I1030 20:37:22.039317 2573 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-138-149.us-west-1.compute.internal" podStartSLOduration=5.039299032 podStartE2EDuration="5.039299032s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 20:37:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:37:22.038106398 +0000 UTC m=+6.850557709" watchObservedRunningTime="2026-10-30 20:37:22.039299032 +0000 UTC m=+6.851750344" Oct 30 20:37:22.822402 ip-10-0-138-149 kubenswrapper[2573]: I1030 20:37:22.821736 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-149.us-west-1.compute.internal" event={"ID":"60d05e7d5fab914587e076383d73f1df","Type":"ContainerStarted","Data":"56e0009de6d65db637c2c16b3018480161a58ad176e5f6395d81ece7ba6b2d3d"} Oct 30 20:37:22.830908 ip-10-0-138-149 kubenswrapper[2573]: I1030 20:37:22.830793 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-cjrkj" event={"ID":"497c44c4-2ece-4009-b88c-6ecfa9920110","Type":"ContainerStarted","Data":"d6f9056d04277ad54b8484471c56cfa3aeda7d59dfa6fb438e16a3d080ab2525"} Oct 30 20:37:22.838379 ip-10-0-138-149 kubenswrapper[2573]: I1030 20:37:22.837469 2573 generic.go:356] "Generic (PLEG): container finished" podID="910c0d9f-c79b-4839-9a1b-f2d278d11fdb" containerID="51ae0082eac1fe9c20081136f6a5f46288ab3cb06de3e424497f41191fc2b3af" exitCode=0 Oct 30 20:37:22.838379 ip-10-0-138-149 kubenswrapper[2573]: I1030 20:37:22.838315 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-blccx" event={"ID":"910c0d9f-c79b-4839-9a1b-f2d278d11fdb","Type":"ContainerDied","Data":"51ae0082eac1fe9c20081136f6a5f46288ab3cb06de3e424497f41191fc2b3af"} Oct 30 20:37:23.069319 ip-10-0-138-149 kubenswrapper[2573]: I1030 20:37:23.069063 2573 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-149.us-west-1.compute.internal" podStartSLOduration=6.069042316 podStartE2EDuration="6.069042316s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 20:37:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:37:22.943636853 +0000 UTC m=+7.756088169" watchObservedRunningTime="2026-10-30 20:37:23.069042316 +0000 UTC m=+7.881493631" Oct 30 20:37:23.190348 ip-10-0-138-149 kubenswrapper[2573]: I1030 20:37:23.190239 2573 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/konnectivity-agent-cjrkj" podStartSLOduration=1.6235379220000001 podStartE2EDuration="6.190219081s" totalImagesPullingTime="4.566681159s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:37:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:37:23.190020429 +0000 UTC m=+8.002471740" watchObservedRunningTime="2026-10-30 20:37:23.190219081 +0000 UTC m=+8.002670395" Oct 30 20:37:23.732461 ip-10-0-138-149 kubenswrapper[2573]: I1030 20:37:23.732398 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-bcmwc" Oct 30 20:37:23.732702 ip-10-0-138-149 kubenswrapper[2573]: E1030 20:37:23.732537 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-bcmwc" podUID="0d6769f4-dc38-45ac-863b-c75fa672866a" Oct 30 20:37:23.732702 ip-10-0-138-149 kubenswrapper[2573]: I1030 20:37:23.732617 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-nhs9k" Oct 30 20:37:23.732829 ip-10-0-138-149 kubenswrapper[2573]: E1030 20:37:23.732704 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-nhs9k" podUID="e886897a-c6a3-414f-875f-aa112541b608" Oct 30 20:37:23.732829 ip-10-0-138-149 kubenswrapper[2573]: I1030 20:37:23.732750 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ws655" Oct 30 20:37:23.732829 ip-10-0-138-149 kubenswrapper[2573]: E1030 20:37:23.732795 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-ws655" podUID="567a72f6-035d-4c61-acaf-f46008029168" Oct 30 20:37:24.846061 ip-10-0-138-149 kubenswrapper[2573]: I1030 20:37:24.846016 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-tlch5" event={"ID":"b7fa45b4-283a-4439-82b7-778abf2e9deb","Type":"ContainerStarted","Data":"07834be95c8eb955004a4f42dacbaaec0d71c984cb2f01b1b90ca78f2ef560e6"} Oct 30 20:37:25.339373 ip-10-0-138-149 kubenswrapper[2573]: I1030 20:37:25.339311 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e886897a-c6a3-414f-875f-aa112541b608-metrics-certs\") pod \"network-metrics-daemon-nhs9k\" (UID: \"e886897a-c6a3-414f-875f-aa112541b608\") " pod="openshift-multus/network-metrics-daemon-nhs9k" Oct 30 20:37:25.339543 ip-10-0-138-149 kubenswrapper[2573]: E1030 20:37:25.339480 2573 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 20:37:25.339818 ip-10-0-138-149 kubenswrapper[2573]: E1030 20:37:25.339544 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e886897a-c6a3-414f-875f-aa112541b608-metrics-certs podName:e886897a-c6a3-414f-875f-aa112541b608 nodeName:}" failed. No retries permitted until 2026-10-30 20:37:33.339525704 +0000 UTC m=+18.151977008 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/e886897a-c6a3-414f-875f-aa112541b608-metrics-certs") pod "network-metrics-daemon-nhs9k" (UID: "e886897a-c6a3-414f-875f-aa112541b608") : object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 20:37:25.440769 ip-10-0-138-149 kubenswrapper[2573]: I1030 20:37:25.440668 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/0d6769f4-dc38-45ac-863b-c75fa672866a-original-pull-secret\") pod \"global-pull-secret-syncer-bcmwc\" (UID: \"0d6769f4-dc38-45ac-863b-c75fa672866a\") " pod="kube-system/global-pull-secret-syncer-bcmwc" Oct 30 20:37:25.440769 ip-10-0-138-149 kubenswrapper[2573]: I1030 20:37:25.440726 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bc9kz\" (UniqueName: \"kubernetes.io/projected/567a72f6-035d-4c61-acaf-f46008029168-kube-api-access-bc9kz\") pod \"network-check-target-ws655\" (UID: \"567a72f6-035d-4c61-acaf-f46008029168\") " pod="openshift-network-diagnostics/network-check-target-ws655" Oct 30 20:37:25.440998 ip-10-0-138-149 kubenswrapper[2573]: E1030 20:37:25.440879 2573 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Oct 30 20:37:25.440998 ip-10-0-138-149 kubenswrapper[2573]: E1030 20:37:25.440898 2573 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Oct 30 20:37:25.440998 ip-10-0-138-149 kubenswrapper[2573]: E1030 20:37:25.440909 2573 projected.go:196] Error preparing data for projected volume kube-api-access-bc9kz for pod openshift-network-diagnostics/network-check-target-ws655: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 20:37:25.440998 ip-10-0-138-149 kubenswrapper[2573]: E1030 20:37:25.440962 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/567a72f6-035d-4c61-acaf-f46008029168-kube-api-access-bc9kz podName:567a72f6-035d-4c61-acaf-f46008029168 nodeName:}" failed. No retries permitted until 2026-10-30 20:37:33.44094375 +0000 UTC m=+18.253395059 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-bc9kz" (UniqueName: "kubernetes.io/projected/567a72f6-035d-4c61-acaf-f46008029168-kube-api-access-bc9kz") pod "network-check-target-ws655" (UID: "567a72f6-035d-4c61-acaf-f46008029168") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 20:37:25.441240 ip-10-0-138-149 kubenswrapper[2573]: E1030 20:37:25.441036 2573 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Oct 30 20:37:25.441240 ip-10-0-138-149 kubenswrapper[2573]: E1030 20:37:25.441073 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0d6769f4-dc38-45ac-863b-c75fa672866a-original-pull-secret podName:0d6769f4-dc38-45ac-863b-c75fa672866a nodeName:}" failed. No retries permitted until 2026-10-30 20:37:33.441061707 +0000 UTC m=+18.253512996 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/0d6769f4-dc38-45ac-863b-c75fa672866a-original-pull-secret") pod "global-pull-secret-syncer-bcmwc" (UID: "0d6769f4-dc38-45ac-863b-c75fa672866a") : object "kube-system"/"original-pull-secret" not registered Oct 30 20:37:25.730757 ip-10-0-138-149 kubenswrapper[2573]: I1030 20:37:25.730673 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ws655" Oct 30 20:37:25.730908 ip-10-0-138-149 kubenswrapper[2573]: E1030 20:37:25.730805 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-ws655" podUID="567a72f6-035d-4c61-acaf-f46008029168" Oct 30 20:37:25.730908 ip-10-0-138-149 kubenswrapper[2573]: I1030 20:37:25.730887 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-bcmwc" Oct 30 20:37:25.731030 ip-10-0-138-149 kubenswrapper[2573]: E1030 20:37:25.730962 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-bcmwc" podUID="0d6769f4-dc38-45ac-863b-c75fa672866a" Oct 30 20:37:25.731866 ip-10-0-138-149 kubenswrapper[2573]: I1030 20:37:25.731846 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-nhs9k" Oct 30 20:37:25.731995 ip-10-0-138-149 kubenswrapper[2573]: E1030 20:37:25.731929 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-nhs9k" podUID="e886897a-c6a3-414f-875f-aa112541b608" Oct 30 20:37:25.853933 ip-10-0-138-149 kubenswrapper[2573]: I1030 20:37:25.853843 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-svf4h" event={"ID":"662e676d-c357-4ac8-a813-d34096051aff","Type":"ContainerStarted","Data":"2a939d01c4a240a2eb5c83c07deb9849a1f9f4fea49609732f996bc7d48eb915"} Oct 30 20:37:26.212523 ip-10-0-138-149 kubenswrapper[2573]: I1030 20:37:26.211795 2573 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-cjrkj" Oct 30 20:37:26.212904 ip-10-0-138-149 kubenswrapper[2573]: I1030 20:37:26.212882 2573 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-cjrkj" Oct 30 20:37:26.329177 ip-10-0-138-149 kubenswrapper[2573]: I1030 20:37:26.328855 2573 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-image-registry/node-ca-tlch5" podStartSLOduration=4.374314845 podStartE2EDuration="10.328836037s" totalImagesPullingTime="5.954521192s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 20:37:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:37:24.967505732 +0000 UTC m=+9.779957045" watchObservedRunningTime="2026-10-30 20:37:26.328836037 +0000 UTC m=+11.141287347" Oct 30 20:37:26.863833 ip-10-0-138-149 kubenswrapper[2573]: I1030 20:37:26.863541 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-dzbhx" event={"ID":"022c4cf0-9797-4428-8fa3-9f3112f2ccc5","Type":"ContainerStarted","Data":"edbd0e93291211f22f989cd5c229adeefa4a57b7f87d9ff16114435a498f570c"} Oct 30 20:37:26.864227 ip-10-0-138-149 kubenswrapper[2573]: I1030 20:37:26.863998 2573 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-cjrkj" Oct 30 20:37:26.864312 ip-10-0-138-149 kubenswrapper[2573]: I1030 20:37:26.864289 2573 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-cjrkj" Oct 30 20:37:26.955043 ip-10-0-138-149 kubenswrapper[2573]: I1030 20:37:26.955016 2573 plugin_watcher.go:201] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Oct 30 20:37:27.095141 ip-10-0-138-149 kubenswrapper[2573]: I1030 20:37:27.095085 2573 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/multus-dzbhx" podStartSLOduration=2.6068716309999997 podStartE2EDuration="11.095072216s" totalImagesPullingTime="8.488200585s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 20:37:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:37:26.978899879 +0000 UTC m=+11.791351188" watchObservedRunningTime="2026-10-30 20:37:27.095072216 +0000 UTC m=+11.907523523" Oct 30 20:37:27.614885 ip-10-0-138-149 kubenswrapper[2573]: I1030 20:37:27.614772 2573 reconciler.go:167] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-10-30T20:37:26.955038946Z","UUID":"bad35c61-b01a-4eaa-8577-abb7783b7d70","Handler":null,"Name":"","Endpoint":""} Oct 30 20:37:27.616551 ip-10-0-138-149 kubenswrapper[2573]: I1030 20:37:27.616528 2573 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Oct 30 20:37:27.616677 ip-10-0-138-149 kubenswrapper[2573]: I1030 20:37:27.616559 2573 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Oct 30 20:37:27.729586 ip-10-0-138-149 kubenswrapper[2573]: I1030 20:37:27.729543 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-nhs9k" Oct 30 20:37:27.729728 ip-10-0-138-149 kubenswrapper[2573]: I1030 20:37:27.729543 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-bcmwc" Oct 30 20:37:27.729728 ip-10-0-138-149 kubenswrapper[2573]: I1030 20:37:27.729697 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ws655" Oct 30 20:37:27.729728 ip-10-0-138-149 kubenswrapper[2573]: E1030 20:37:27.729691 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-nhs9k" podUID="e886897a-c6a3-414f-875f-aa112541b608" Oct 30 20:37:27.729886 ip-10-0-138-149 kubenswrapper[2573]: E1030 20:37:27.729782 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-bcmwc" podUID="0d6769f4-dc38-45ac-863b-c75fa672866a" Oct 30 20:37:27.729886 ip-10-0-138-149 kubenswrapper[2573]: E1030 20:37:27.729831 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-ws655" podUID="567a72f6-035d-4c61-acaf-f46008029168" Oct 30 20:37:27.868179 ip-10-0-138-149 kubenswrapper[2573]: I1030 20:37:27.868140 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-nv2rk" event={"ID":"666b6e91-4a60-45d9-9e3e-877867ded7fc","Type":"ContainerStarted","Data":"4c267570f44bea9162dcd29b1bd727c8f4f41bdadf63c6e206d5bc78a4b8f373"} Oct 30 20:37:27.870695 ip-10-0-138-149 kubenswrapper[2573]: I1030 20:37:27.870615 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-svf4h" event={"ID":"662e676d-c357-4ac8-a813-d34096051aff","Type":"ContainerStarted","Data":"6f5349bb133ef169609a705a7b4ee3a2a5227fdd0bbbe4212cb4305ab6a5abf0"} Oct 30 20:37:27.872661 ip-10-0-138-149 kubenswrapper[2573]: I1030 20:37:27.872631 2573 generic.go:356] "Generic (PLEG): container finished" podID="910c0d9f-c79b-4839-9a1b-f2d278d11fdb" containerID="5814699cadaf62a2bf4a4282326b8dcba613ce6e0559c80f56610ca8fa8bf2b3" exitCode=0 Oct 30 20:37:27.872773 ip-10-0-138-149 kubenswrapper[2573]: I1030 20:37:27.872735 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-blccx" event={"ID":"910c0d9f-c79b-4839-9a1b-f2d278d11fdb","Type":"ContainerDied","Data":"5814699cadaf62a2bf4a4282326b8dcba613ce6e0559c80f56610ca8fa8bf2b3"} Oct 30 20:37:28.102731 ip-10-0-138-149 kubenswrapper[2573]: I1030 20:37:28.102638 2573 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-dns/node-resolver-nv2rk" podStartSLOduration=2.384065716 podStartE2EDuration="11.102622786s" totalImagesPullingTime="8.71855707s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 20:37:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:37:27.983425063 +0000 UTC m=+12.795876374" watchObservedRunningTime="2026-10-30 20:37:28.102622786 +0000 UTC m=+12.915074093" Oct 30 20:37:28.877018 ip-10-0-138-149 kubenswrapper[2573]: I1030 20:37:28.876918 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-svf4h" event={"ID":"662e676d-c357-4ac8-a813-d34096051aff","Type":"ContainerStarted","Data":"5db7e5d6108b3819b29247ec70fcfaaaf569471f5c05b71d798921c61e6efa4d"} Oct 30 20:37:28.878496 ip-10-0-138-149 kubenswrapper[2573]: I1030 20:37:28.878474 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-kvq2l" event={"ID":"affdfb88-e208-4ca1-843d-efc8384c27a1","Type":"ContainerStarted","Data":"153f9a3f67923a4848bb1b4dcf02792af5198ec7a8f8549180e124f3c595e052"} Oct 30 20:37:28.879838 ip-10-0-138-149 kubenswrapper[2573]: I1030 20:37:28.879810 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-w9npq" event={"ID":"23527d33-4e73-4c36-ba42-7f0a5b7f62c6","Type":"ContainerStarted","Data":"0696559ac41536365d09cb35d0106c56be4ce74d77ca63800fe57179a6846c12"} Oct 30 20:37:28.993885 ip-10-0-138-149 kubenswrapper[2573]: I1030 20:37:28.993836 2573 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-svf4h" podStartSLOduration=1.935806935 podStartE2EDuration="11.993822448s" totalImagesPullingTime="10.058015513s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 20:37:17 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:37:28.993210842 +0000 UTC m=+13.805662152" watchObservedRunningTime="2026-10-30 20:37:28.993822448 +0000 UTC m=+13.806273755" Oct 30 20:37:29.105615 ip-10-0-138-149 kubenswrapper[2573]: I1030 20:37:29.105543 2573 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-w9npq" podStartSLOduration=2.638778307 podStartE2EDuration="13.105527632s" totalImagesPullingTime="10.466749325s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 20:37:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:37:29.105497246 +0000 UTC m=+13.917948565" watchObservedRunningTime="2026-10-30 20:37:29.105527632 +0000 UTC m=+13.917978941" Oct 30 20:37:29.729361 ip-10-0-138-149 kubenswrapper[2573]: I1030 20:37:29.729329 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-nhs9k" Oct 30 20:37:29.729537 ip-10-0-138-149 kubenswrapper[2573]: I1030 20:37:29.729381 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ws655" Oct 30 20:37:29.729537 ip-10-0-138-149 kubenswrapper[2573]: I1030 20:37:29.729376 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-bcmwc" Oct 30 20:37:29.729537 ip-10-0-138-149 kubenswrapper[2573]: E1030 20:37:29.729470 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-ws655" podUID="567a72f6-035d-4c61-acaf-f46008029168" Oct 30 20:37:29.729725 ip-10-0-138-149 kubenswrapper[2573]: E1030 20:37:29.729531 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-bcmwc" podUID="0d6769f4-dc38-45ac-863b-c75fa672866a" Oct 30 20:37:29.729725 ip-10-0-138-149 kubenswrapper[2573]: E1030 20:37:29.729643 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-nhs9k" podUID="e886897a-c6a3-414f-875f-aa112541b608" Oct 30 20:37:29.883522 ip-10-0-138-149 kubenswrapper[2573]: I1030 20:37:29.883489 2573 generic.go:356] "Generic (PLEG): container finished" podID="910c0d9f-c79b-4839-9a1b-f2d278d11fdb" containerID="c27d3efddeec26f74a3705d8eb76ea39320cd98af64fabce37e8bae458c8c87a" exitCode=0 Oct 30 20:37:29.884033 ip-10-0-138-149 kubenswrapper[2573]: I1030 20:37:29.883600 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-blccx" event={"ID":"910c0d9f-c79b-4839-9a1b-f2d278d11fdb","Type":"ContainerDied","Data":"c27d3efddeec26f74a3705d8eb76ea39320cd98af64fabce37e8bae458c8c87a"} Oct 30 20:37:30.106752 ip-10-0-138-149 kubenswrapper[2573]: I1030 20:37:30.106668 2573 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-kvq2l" podStartSLOduration=4.153716229 podStartE2EDuration="13.106654425s" totalImagesPullingTime="8.952938196s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 20:37:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:37:29.21609511 +0000 UTC m=+14.028546419" watchObservedRunningTime="2026-10-30 20:37:30.106654425 +0000 UTC m=+14.919105730" Oct 30 20:37:31.732145 ip-10-0-138-149 kubenswrapper[2573]: I1030 20:37:31.731974 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ws655" Oct 30 20:37:31.732596 ip-10-0-138-149 kubenswrapper[2573]: E1030 20:37:31.732209 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-ws655" podUID="567a72f6-035d-4c61-acaf-f46008029168" Oct 30 20:37:31.732596 ip-10-0-138-149 kubenswrapper[2573]: I1030 20:37:31.732057 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-bcmwc" Oct 30 20:37:31.732596 ip-10-0-138-149 kubenswrapper[2573]: E1030 20:37:31.732268 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-bcmwc" podUID="0d6769f4-dc38-45ac-863b-c75fa672866a" Oct 30 20:37:31.732596 ip-10-0-138-149 kubenswrapper[2573]: I1030 20:37:31.732050 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-nhs9k" Oct 30 20:37:31.732596 ip-10-0-138-149 kubenswrapper[2573]: E1030 20:37:31.732327 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-nhs9k" podUID="e886897a-c6a3-414f-875f-aa112541b608" Oct 30 20:37:31.889365 ip-10-0-138-149 kubenswrapper[2573]: I1030 20:37:31.889330 2573 generic.go:356] "Generic (PLEG): container finished" podID="910c0d9f-c79b-4839-9a1b-f2d278d11fdb" containerID="49857697620af61532dbf1d52834ddac9bad36f61ec5936b3424359ce1c69970" exitCode=0 Oct 30 20:37:31.889529 ip-10-0-138-149 kubenswrapper[2573]: I1030 20:37:31.889374 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-blccx" event={"ID":"910c0d9f-c79b-4839-9a1b-f2d278d11fdb","Type":"ContainerDied","Data":"49857697620af61532dbf1d52834ddac9bad36f61ec5936b3424359ce1c69970"} Oct 30 20:37:33.405682 ip-10-0-138-149 kubenswrapper[2573]: I1030 20:37:33.405639 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e886897a-c6a3-414f-875f-aa112541b608-metrics-certs\") pod \"network-metrics-daemon-nhs9k\" (UID: \"e886897a-c6a3-414f-875f-aa112541b608\") " pod="openshift-multus/network-metrics-daemon-nhs9k" Oct 30 20:37:33.406132 ip-10-0-138-149 kubenswrapper[2573]: E1030 20:37:33.405793 2573 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 20:37:33.406132 ip-10-0-138-149 kubenswrapper[2573]: E1030 20:37:33.405877 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e886897a-c6a3-414f-875f-aa112541b608-metrics-certs podName:e886897a-c6a3-414f-875f-aa112541b608 nodeName:}" failed. No retries permitted until 2026-10-30 20:37:49.405855791 +0000 UTC m=+34.218307093 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/e886897a-c6a3-414f-875f-aa112541b608-metrics-certs") pod "network-metrics-daemon-nhs9k" (UID: "e886897a-c6a3-414f-875f-aa112541b608") : object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 20:37:33.506688 ip-10-0-138-149 kubenswrapper[2573]: I1030 20:37:33.506654 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bc9kz\" (UniqueName: \"kubernetes.io/projected/567a72f6-035d-4c61-acaf-f46008029168-kube-api-access-bc9kz\") pod \"network-check-target-ws655\" (UID: \"567a72f6-035d-4c61-acaf-f46008029168\") " pod="openshift-network-diagnostics/network-check-target-ws655" Oct 30 20:37:33.506837 ip-10-0-138-149 kubenswrapper[2573]: I1030 20:37:33.506746 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/0d6769f4-dc38-45ac-863b-c75fa672866a-original-pull-secret\") pod \"global-pull-secret-syncer-bcmwc\" (UID: \"0d6769f4-dc38-45ac-863b-c75fa672866a\") " pod="kube-system/global-pull-secret-syncer-bcmwc" Oct 30 20:37:33.506837 ip-10-0-138-149 kubenswrapper[2573]: E1030 20:37:33.506827 2573 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Oct 30 20:37:33.506937 ip-10-0-138-149 kubenswrapper[2573]: E1030 20:37:33.506851 2573 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Oct 30 20:37:33.506937 ip-10-0-138-149 kubenswrapper[2573]: E1030 20:37:33.506861 2573 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Oct 30 20:37:33.506937 ip-10-0-138-149 kubenswrapper[2573]: E1030 20:37:33.506917 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0d6769f4-dc38-45ac-863b-c75fa672866a-original-pull-secret podName:0d6769f4-dc38-45ac-863b-c75fa672866a nodeName:}" failed. No retries permitted until 2026-10-30 20:37:49.506899292 +0000 UTC m=+34.319350581 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/0d6769f4-dc38-45ac-863b-c75fa672866a-original-pull-secret") pod "global-pull-secret-syncer-bcmwc" (UID: "0d6769f4-dc38-45ac-863b-c75fa672866a") : object "kube-system"/"original-pull-secret" not registered Oct 30 20:37:33.507096 ip-10-0-138-149 kubenswrapper[2573]: E1030 20:37:33.506864 2573 projected.go:196] Error preparing data for projected volume kube-api-access-bc9kz for pod openshift-network-diagnostics/network-check-target-ws655: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 20:37:33.507096 ip-10-0-138-149 kubenswrapper[2573]: E1030 20:37:33.507041 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/567a72f6-035d-4c61-acaf-f46008029168-kube-api-access-bc9kz podName:567a72f6-035d-4c61-acaf-f46008029168 nodeName:}" failed. No retries permitted until 2026-10-30 20:37:49.506990099 +0000 UTC m=+34.319441398 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-bc9kz" (UniqueName: "kubernetes.io/projected/567a72f6-035d-4c61-acaf-f46008029168-kube-api-access-bc9kz") pod "network-check-target-ws655" (UID: "567a72f6-035d-4c61-acaf-f46008029168") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 20:37:33.733645 ip-10-0-138-149 kubenswrapper[2573]: I1030 20:37:33.733573 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-bcmwc" Oct 30 20:37:33.733788 ip-10-0-138-149 kubenswrapper[2573]: E1030 20:37:33.733691 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-bcmwc" podUID="0d6769f4-dc38-45ac-863b-c75fa672866a" Oct 30 20:37:33.734028 ip-10-0-138-149 kubenswrapper[2573]: I1030 20:37:33.734005 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-nhs9k" Oct 30 20:37:33.734127 ip-10-0-138-149 kubenswrapper[2573]: E1030 20:37:33.734107 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-nhs9k" podUID="e886897a-c6a3-414f-875f-aa112541b608" Oct 30 20:37:33.734188 ip-10-0-138-149 kubenswrapper[2573]: I1030 20:37:33.734157 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ws655" Oct 30 20:37:33.734237 ip-10-0-138-149 kubenswrapper[2573]: E1030 20:37:33.734222 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-ws655" podUID="567a72f6-035d-4c61-acaf-f46008029168" Oct 30 20:37:34.898131 ip-10-0-138-149 kubenswrapper[2573]: I1030 20:37:34.898088 2573 generic.go:356] "Generic (PLEG): container finished" podID="910c0d9f-c79b-4839-9a1b-f2d278d11fdb" containerID="3997d28afa3738cf1530b6d2b2ed3e034bc9873d5b8c50ea05c7c3570e2c1ea2" exitCode=0 Oct 30 20:37:34.898538 ip-10-0-138-149 kubenswrapper[2573]: I1030 20:37:34.898148 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-blccx" event={"ID":"910c0d9f-c79b-4839-9a1b-f2d278d11fdb","Type":"ContainerDied","Data":"3997d28afa3738cf1530b6d2b2ed3e034bc9873d5b8c50ea05c7c3570e2c1ea2"} Oct 30 20:37:35.731040 ip-10-0-138-149 kubenswrapper[2573]: I1030 20:37:35.731003 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-bcmwc" Oct 30 20:37:35.731210 ip-10-0-138-149 kubenswrapper[2573]: E1030 20:37:35.731118 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-bcmwc" podUID="0d6769f4-dc38-45ac-863b-c75fa672866a" Oct 30 20:37:35.731526 ip-10-0-138-149 kubenswrapper[2573]: I1030 20:37:35.731498 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-nhs9k" Oct 30 20:37:35.731647 ip-10-0-138-149 kubenswrapper[2573]: E1030 20:37:35.731626 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-nhs9k" podUID="e886897a-c6a3-414f-875f-aa112541b608" Oct 30 20:37:35.731725 ip-10-0-138-149 kubenswrapper[2573]: I1030 20:37:35.731682 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ws655" Oct 30 20:37:35.731764 ip-10-0-138-149 kubenswrapper[2573]: E1030 20:37:35.731750 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-ws655" podUID="567a72f6-035d-4c61-acaf-f46008029168" Oct 30 20:37:35.903193 ip-10-0-138-149 kubenswrapper[2573]: I1030 20:37:35.903152 2573 generic.go:356] "Generic (PLEG): container finished" podID="910c0d9f-c79b-4839-9a1b-f2d278d11fdb" containerID="518c10ef1a5ae0d2822952207f1f5bcff92efbef74e13186da746539ac48d3e3" exitCode=0 Oct 30 20:37:35.903193 ip-10-0-138-149 kubenswrapper[2573]: I1030 20:37:35.903196 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-blccx" event={"ID":"910c0d9f-c79b-4839-9a1b-f2d278d11fdb","Type":"ContainerDied","Data":"518c10ef1a5ae0d2822952207f1f5bcff92efbef74e13186da746539ac48d3e3"} Oct 30 20:37:36.797956 ip-10-0-138-149 kubenswrapper[2573]: I1030 20:37:36.797924 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 20:37:36.909298 ip-10-0-138-149 kubenswrapper[2573]: I1030 20:37:36.909259 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-blccx" event={"ID":"910c0d9f-c79b-4839-9a1b-f2d278d11fdb","Type":"ContainerStarted","Data":"077bcb8b31e150c006acdd60fffe7195057cc559c017cf336569032b903998fb"} Oct 30 20:37:36.957819 ip-10-0-138-149 kubenswrapper[2573]: I1030 20:37:36.957789 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 20:37:37.077410 ip-10-0-138-149 kubenswrapper[2573]: I1030 20:37:37.077306 2573 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-blccx" podStartSLOduration=8.65178662 podStartE2EDuration="21.077288175s" totalImagesPullingTime="12.425501555s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 20:37:16 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:37:37.07662168 +0000 UTC m=+21.889072989" watchObservedRunningTime="2026-10-30 20:37:37.077288175 +0000 UTC m=+21.889739485" Oct 30 20:37:37.198711 ip-10-0-138-149 kubenswrapper[2573]: I1030 20:37:37.198677 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 20:37:37.340277 ip-10-0-138-149 kubenswrapper[2573]: I1030 20:37:37.340209 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 20:37:37.732351 ip-10-0-138-149 kubenswrapper[2573]: I1030 20:37:37.732282 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ws655" Oct 30 20:37:37.732351 ip-10-0-138-149 kubenswrapper[2573]: I1030 20:37:37.732316 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-bcmwc" Oct 30 20:37:37.732507 ip-10-0-138-149 kubenswrapper[2573]: I1030 20:37:37.732407 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-nhs9k" Oct 30 20:37:37.732507 ip-10-0-138-149 kubenswrapper[2573]: E1030 20:37:37.732413 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-ws655" podUID="567a72f6-035d-4c61-acaf-f46008029168" Oct 30 20:37:37.732507 ip-10-0-138-149 kubenswrapper[2573]: E1030 20:37:37.732478 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-bcmwc" podUID="0d6769f4-dc38-45ac-863b-c75fa672866a" Oct 30 20:37:37.732633 ip-10-0-138-149 kubenswrapper[2573]: E1030 20:37:37.732533 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-nhs9k" podUID="e886897a-c6a3-414f-875f-aa112541b608" Oct 30 20:37:38.573884 ip-10-0-138-149 kubenswrapper[2573]: I1030 20:37:38.573681 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 20:37:38.771019 ip-10-0-138-149 kubenswrapper[2573]: I1030 20:37:38.770989 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 20:37:38.916168 ip-10-0-138-149 kubenswrapper[2573]: I1030 20:37:38.916147 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-7x9jg_c2ff359c-cf92-4a5b-8610-f6eb41acbe73/ovn-acl-logging/0.log" Oct 30 20:37:38.916500 ip-10-0-138-149 kubenswrapper[2573]: I1030 20:37:38.916481 2573 generic.go:356] "Generic (PLEG): container finished" podID="c2ff359c-cf92-4a5b-8610-f6eb41acbe73" containerID="29de0471a6b007a7c1375d5961078e11c1ca7d03092695918b4db8e245ea19b3" exitCode=1 Oct 30 20:37:38.916550 ip-10-0-138-149 kubenswrapper[2573]: I1030 20:37:38.916525 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-7x9jg" event={"ID":"c2ff359c-cf92-4a5b-8610-f6eb41acbe73","Type":"ContainerStarted","Data":"d1d33fdbf8a62ae9dfb3f51191cfacb701a9431ab16738d0cceafbe0b4b9182b"} Oct 30 20:37:38.916599 ip-10-0-138-149 kubenswrapper[2573]: I1030 20:37:38.916588 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-7x9jg" event={"ID":"c2ff359c-cf92-4a5b-8610-f6eb41acbe73","Type":"ContainerStarted","Data":"20652247aeeb886039be3f2915e43fa4e1930decdc411220e2d0f8fec6998499"} Oct 30 20:37:38.916631 ip-10-0-138-149 kubenswrapper[2573]: I1030 20:37:38.916620 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-7x9jg" event={"ID":"c2ff359c-cf92-4a5b-8610-f6eb41acbe73","Type":"ContainerStarted","Data":"43f218bb3ff373d2100d637a14db6ffe0d8acbdd880e07044e5d579e68679356"} Oct 30 20:37:38.916665 ip-10-0-138-149 kubenswrapper[2573]: I1030 20:37:38.916651 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-7x9jg" event={"ID":"c2ff359c-cf92-4a5b-8610-f6eb41acbe73","Type":"ContainerStarted","Data":"f60403159d0dac7e8189b81bc36efdc4e9546ac27f145cdc9a313791e556ff83"} Oct 30 20:37:38.916695 ip-10-0-138-149 kubenswrapper[2573]: I1030 20:37:38.916683 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-7x9jg" event={"ID":"c2ff359c-cf92-4a5b-8610-f6eb41acbe73","Type":"ContainerDied","Data":"29de0471a6b007a7c1375d5961078e11c1ca7d03092695918b4db8e245ea19b3"} Oct 30 20:37:38.916725 ip-10-0-138-149 kubenswrapper[2573]: I1030 20:37:38.916710 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-7x9jg" event={"ID":"c2ff359c-cf92-4a5b-8610-f6eb41acbe73","Type":"ContainerStarted","Data":"2c88c6fdcb91c105ac5217e9432b8423439b739c1e6c7e31e4ff5c4b95ae46c3"} Oct 30 20:37:39.729549 ip-10-0-138-149 kubenswrapper[2573]: I1030 20:37:39.729520 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ws655" Oct 30 20:37:39.729549 ip-10-0-138-149 kubenswrapper[2573]: I1030 20:37:39.729541 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-nhs9k" Oct 30 20:37:39.730226 ip-10-0-138-149 kubenswrapper[2573]: I1030 20:37:39.729606 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-bcmwc" Oct 30 20:37:39.730226 ip-10-0-138-149 kubenswrapper[2573]: E1030 20:37:39.729692 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-ws655" podUID="567a72f6-035d-4c61-acaf-f46008029168" Oct 30 20:37:39.730226 ip-10-0-138-149 kubenswrapper[2573]: E1030 20:37:39.729812 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-bcmwc" podUID="0d6769f4-dc38-45ac-863b-c75fa672866a" Oct 30 20:37:39.730226 ip-10-0-138-149 kubenswrapper[2573]: E1030 20:37:39.729918 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-nhs9k" podUID="e886897a-c6a3-414f-875f-aa112541b608" Oct 30 20:37:40.923708 ip-10-0-138-149 kubenswrapper[2573]: I1030 20:37:40.923638 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-7x9jg_c2ff359c-cf92-4a5b-8610-f6eb41acbe73/ovn-acl-logging/0.log" Oct 30 20:37:40.924092 ip-10-0-138-149 kubenswrapper[2573]: I1030 20:37:40.923992 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-7x9jg" event={"ID":"c2ff359c-cf92-4a5b-8610-f6eb41acbe73","Type":"ContainerStarted","Data":"bb42a4f0c69901199d4393b5003000dc8bbb5c1224e0f0dd458eaf4faab6ccd6"} Oct 30 20:37:41.728891 ip-10-0-138-149 kubenswrapper[2573]: I1030 20:37:41.728860 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-nhs9k" Oct 30 20:37:41.729049 ip-10-0-138-149 kubenswrapper[2573]: E1030 20:37:41.728977 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-nhs9k" podUID="e886897a-c6a3-414f-875f-aa112541b608" Oct 30 20:37:41.729049 ip-10-0-138-149 kubenswrapper[2573]: I1030 20:37:41.729035 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ws655" Oct 30 20:37:41.729125 ip-10-0-138-149 kubenswrapper[2573]: I1030 20:37:41.729072 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-bcmwc" Oct 30 20:37:41.729165 ip-10-0-138-149 kubenswrapper[2573]: E1030 20:37:41.729149 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-bcmwc" podUID="0d6769f4-dc38-45ac-863b-c75fa672866a" Oct 30 20:37:41.729234 ip-10-0-138-149 kubenswrapper[2573]: E1030 20:37:41.729217 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-ws655" podUID="567a72f6-035d-4c61-acaf-f46008029168" Oct 30 20:37:41.902632 ip-10-0-138-149 kubenswrapper[2573]: I1030 20:37:41.902603 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 20:37:42.035487 ip-10-0-138-149 kubenswrapper[2573]: I1030 20:37:42.035421 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 20:37:42.188899 ip-10-0-138-149 kubenswrapper[2573]: I1030 20:37:42.188868 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 20:37:42.330783 ip-10-0-138-149 kubenswrapper[2573]: I1030 20:37:42.330711 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 20:37:42.459752 ip-10-0-138-149 kubenswrapper[2573]: I1030 20:37:42.459720 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 20:37:42.587718 ip-10-0-138-149 kubenswrapper[2573]: I1030 20:37:42.587650 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 20:37:42.910459 ip-10-0-138-149 kubenswrapper[2573]: I1030 20:37:42.910429 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 20:37:42.931948 ip-10-0-138-149 kubenswrapper[2573]: I1030 20:37:42.931822 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-7x9jg_c2ff359c-cf92-4a5b-8610-f6eb41acbe73/ovn-acl-logging/0.log" Oct 30 20:37:42.932373 ip-10-0-138-149 kubenswrapper[2573]: I1030 20:37:42.932354 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-7x9jg" event={"ID":"c2ff359c-cf92-4a5b-8610-f6eb41acbe73","Type":"ContainerStarted","Data":"f9365e017228d8753d99d0892ae7a686023cd86a91b1221006038ccb2591c837"} Oct 30 20:37:42.932769 ip-10-0-138-149 kubenswrapper[2573]: I1030 20:37:42.932748 2573 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-7x9jg" Oct 30 20:37:42.932861 ip-10-0-138-149 kubenswrapper[2573]: I1030 20:37:42.932835 2573 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-7x9jg" Oct 30 20:37:42.932914 ip-10-0-138-149 kubenswrapper[2573]: I1030 20:37:42.932869 2573 scope.go:122] "RemoveContainer" containerID="29de0471a6b007a7c1375d5961078e11c1ca7d03092695918b4db8e245ea19b3" Oct 30 20:37:42.932914 ip-10-0-138-149 kubenswrapper[2573]: I1030 20:37:42.932886 2573 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-7x9jg" Oct 30 20:37:42.948224 ip-10-0-138-149 kubenswrapper[2573]: I1030 20:37:42.948205 2573 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-7x9jg" Oct 30 20:37:42.948406 ip-10-0-138-149 kubenswrapper[2573]: I1030 20:37:42.948391 2573 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-7x9jg" Oct 30 20:37:43.033261 ip-10-0-138-149 kubenswrapper[2573]: I1030 20:37:43.033232 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 20:37:43.158133 ip-10-0-138-149 kubenswrapper[2573]: I1030 20:37:43.158105 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 20:37:43.729604 ip-10-0-138-149 kubenswrapper[2573]: I1030 20:37:43.729561 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-nhs9k" Oct 30 20:37:43.729604 ip-10-0-138-149 kubenswrapper[2573]: I1030 20:37:43.729595 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ws655" Oct 30 20:37:43.729795 ip-10-0-138-149 kubenswrapper[2573]: E1030 20:37:43.729671 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-nhs9k" podUID="e886897a-c6a3-414f-875f-aa112541b608" Oct 30 20:37:43.729795 ip-10-0-138-149 kubenswrapper[2573]: I1030 20:37:43.729710 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-bcmwc" Oct 30 20:37:43.729795 ip-10-0-138-149 kubenswrapper[2573]: E1030 20:37:43.729779 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-bcmwc" podUID="0d6769f4-dc38-45ac-863b-c75fa672866a" Oct 30 20:37:43.729913 ip-10-0-138-149 kubenswrapper[2573]: E1030 20:37:43.729875 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-ws655" podUID="567a72f6-035d-4c61-acaf-f46008029168" Oct 30 20:37:43.937630 ip-10-0-138-149 kubenswrapper[2573]: I1030 20:37:43.937604 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-7x9jg_c2ff359c-cf92-4a5b-8610-f6eb41acbe73/ovn-acl-logging/0.log" Oct 30 20:37:43.937955 ip-10-0-138-149 kubenswrapper[2573]: I1030 20:37:43.937937 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-7x9jg" event={"ID":"c2ff359c-cf92-4a5b-8610-f6eb41acbe73","Type":"ContainerStarted","Data":"8c668b8614a591f20d84f8ae41f3d0d4b68db9c691656f47c0ef25e854373b45"} Oct 30 20:37:43.982254 ip-10-0-138-149 kubenswrapper[2573]: I1030 20:37:43.982190 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 20:37:44.140239 ip-10-0-138-149 kubenswrapper[2573]: I1030 20:37:44.140212 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 20:37:44.314552 ip-10-0-138-149 kubenswrapper[2573]: I1030 20:37:44.314474 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 20:37:45.050892 ip-10-0-138-149 kubenswrapper[2573]: I1030 20:37:45.050849 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 20:37:45.658961 ip-10-0-138-149 kubenswrapper[2573]: I1030 20:37:45.658909 2573 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-7x9jg" podStartSLOduration=9.59489697 podStartE2EDuration="29.65889395s" totalImagesPullingTime="20.06399698s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 20:37:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:37:44.117888292 +0000 UTC m=+28.930339602" watchObservedRunningTime="2026-10-30 20:37:45.65889395 +0000 UTC m=+30.471345257" Oct 30 20:37:45.659471 ip-10-0-138-149 kubenswrapper[2573]: I1030 20:37:45.659452 2573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-bcmwc"] Oct 30 20:37:45.659631 ip-10-0-138-149 kubenswrapper[2573]: I1030 20:37:45.659611 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-bcmwc" Oct 30 20:37:45.659770 ip-10-0-138-149 kubenswrapper[2573]: E1030 20:37:45.659712 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-bcmwc" podUID="0d6769f4-dc38-45ac-863b-c75fa672866a" Oct 30 20:37:45.661317 ip-10-0-138-149 kubenswrapper[2573]: I1030 20:37:45.661293 2573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-nhs9k"] Oct 30 20:37:45.661403 ip-10-0-138-149 kubenswrapper[2573]: I1030 20:37:45.661392 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-nhs9k" Oct 30 20:37:45.661516 ip-10-0-138-149 kubenswrapper[2573]: E1030 20:37:45.661491 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-nhs9k" podUID="e886897a-c6a3-414f-875f-aa112541b608" Oct 30 20:37:45.708216 ip-10-0-138-149 kubenswrapper[2573]: I1030 20:37:45.708194 2573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-ws655"] Oct 30 20:37:45.708312 ip-10-0-138-149 kubenswrapper[2573]: I1030 20:37:45.708271 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ws655" Oct 30 20:37:45.708351 ip-10-0-138-149 kubenswrapper[2573]: E1030 20:37:45.708332 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-ws655" podUID="567a72f6-035d-4c61-acaf-f46008029168" Oct 30 20:37:46.194028 ip-10-0-138-149 kubenswrapper[2573]: I1030 20:37:46.193993 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 20:37:46.392690 ip-10-0-138-149 kubenswrapper[2573]: I1030 20:37:46.392658 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 20:37:46.440880 ip-10-0-138-149 kubenswrapper[2573]: I1030 20:37:46.440858 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-nv2rk_666b6e91-4a60-45d9-9e3e-877867ded7fc/dns-node-resolver/0.log" Oct 30 20:37:46.689555 ip-10-0-138-149 kubenswrapper[2573]: I1030 20:37:46.689526 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 20:37:46.855220 ip-10-0-138-149 kubenswrapper[2573]: I1030 20:37:46.855186 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 20:37:47.396181 ip-10-0-138-149 kubenswrapper[2573]: I1030 20:37:47.396154 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 20:37:47.565657 ip-10-0-138-149 kubenswrapper[2573]: I1030 20:37:47.565625 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-tlch5_b7fa45b4-283a-4439-82b7-778abf2e9deb/node-ca/0.log" Oct 30 20:37:47.729270 ip-10-0-138-149 kubenswrapper[2573]: I1030 20:37:47.729200 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-nhs9k" Oct 30 20:37:47.729270 ip-10-0-138-149 kubenswrapper[2573]: I1030 20:37:47.729228 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-bcmwc" Oct 30 20:37:47.729270 ip-10-0-138-149 kubenswrapper[2573]: I1030 20:37:47.729259 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ws655" Oct 30 20:37:47.729698 ip-10-0-138-149 kubenswrapper[2573]: E1030 20:37:47.729344 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-ws655" podUID="567a72f6-035d-4c61-acaf-f46008029168" Oct 30 20:37:47.729698 ip-10-0-138-149 kubenswrapper[2573]: E1030 20:37:47.729405 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-bcmwc" podUID="0d6769f4-dc38-45ac-863b-c75fa672866a" Oct 30 20:37:47.729698 ip-10-0-138-149 kubenswrapper[2573]: E1030 20:37:47.729468 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-nhs9k" podUID="e886897a-c6a3-414f-875f-aa112541b608" Oct 30 20:37:47.820426 ip-10-0-138-149 kubenswrapper[2573]: I1030 20:37:47.820398 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 20:37:48.000892 ip-10-0-138-149 kubenswrapper[2573]: I1030 20:37:48.000800 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 20:37:48.199237 ip-10-0-138-149 kubenswrapper[2573]: I1030 20:37:48.199197 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 20:37:48.342726 ip-10-0-138-149 kubenswrapper[2573]: I1030 20:37:48.342649 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 20:37:48.596059 ip-10-0-138-149 kubenswrapper[2573]: I1030 20:37:48.596028 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 20:37:48.801850 ip-10-0-138-149 kubenswrapper[2573]: I1030 20:37:48.801816 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 20:37:49.388726 ip-10-0-138-149 kubenswrapper[2573]: I1030 20:37:49.388695 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 20:37:49.434555 ip-10-0-138-149 kubenswrapper[2573]: I1030 20:37:49.434527 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e886897a-c6a3-414f-875f-aa112541b608-metrics-certs\") pod \"network-metrics-daemon-nhs9k\" (UID: \"e886897a-c6a3-414f-875f-aa112541b608\") " pod="openshift-multus/network-metrics-daemon-nhs9k" Oct 30 20:37:49.434730 ip-10-0-138-149 kubenswrapper[2573]: E1030 20:37:49.434651 2573 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 20:37:49.434730 ip-10-0-138-149 kubenswrapper[2573]: E1030 20:37:49.434713 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e886897a-c6a3-414f-875f-aa112541b608-metrics-certs podName:e886897a-c6a3-414f-875f-aa112541b608 nodeName:}" failed. No retries permitted until 2026-10-30 20:38:21.434694535 +0000 UTC m=+66.247145836 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/e886897a-c6a3-414f-875f-aa112541b608-metrics-certs") pod "network-metrics-daemon-nhs9k" (UID: "e886897a-c6a3-414f-875f-aa112541b608") : object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 20:37:49.535383 ip-10-0-138-149 kubenswrapper[2573]: I1030 20:37:49.535356 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/0d6769f4-dc38-45ac-863b-c75fa672866a-original-pull-secret\") pod \"global-pull-secret-syncer-bcmwc\" (UID: \"0d6769f4-dc38-45ac-863b-c75fa672866a\") " pod="kube-system/global-pull-secret-syncer-bcmwc" Oct 30 20:37:49.535511 ip-10-0-138-149 kubenswrapper[2573]: I1030 20:37:49.535391 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bc9kz\" (UniqueName: \"kubernetes.io/projected/567a72f6-035d-4c61-acaf-f46008029168-kube-api-access-bc9kz\") pod \"network-check-target-ws655\" (UID: \"567a72f6-035d-4c61-acaf-f46008029168\") " pod="openshift-network-diagnostics/network-check-target-ws655" Oct 30 20:37:49.535511 ip-10-0-138-149 kubenswrapper[2573]: E1030 20:37:49.535492 2573 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Oct 30 20:37:49.535511 ip-10-0-138-149 kubenswrapper[2573]: E1030 20:37:49.535502 2573 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Oct 30 20:37:49.535621 ip-10-0-138-149 kubenswrapper[2573]: E1030 20:37:49.535514 2573 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Oct 30 20:37:49.535621 ip-10-0-138-149 kubenswrapper[2573]: E1030 20:37:49.535523 2573 projected.go:196] Error preparing data for projected volume kube-api-access-bc9kz for pod openshift-network-diagnostics/network-check-target-ws655: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 20:37:49.535621 ip-10-0-138-149 kubenswrapper[2573]: E1030 20:37:49.535548 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0d6769f4-dc38-45ac-863b-c75fa672866a-original-pull-secret podName:0d6769f4-dc38-45ac-863b-c75fa672866a nodeName:}" failed. No retries permitted until 2026-10-30 20:38:21.535531005 +0000 UTC m=+66.347982292 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/0d6769f4-dc38-45ac-863b-c75fa672866a-original-pull-secret") pod "global-pull-secret-syncer-bcmwc" (UID: "0d6769f4-dc38-45ac-863b-c75fa672866a") : object "kube-system"/"original-pull-secret" not registered Oct 30 20:37:49.535621 ip-10-0-138-149 kubenswrapper[2573]: E1030 20:37:49.535560 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/567a72f6-035d-4c61-acaf-f46008029168-kube-api-access-bc9kz podName:567a72f6-035d-4c61-acaf-f46008029168 nodeName:}" failed. No retries permitted until 2026-10-30 20:38:21.535554761 +0000 UTC m=+66.348006051 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-bc9kz" (UniqueName: "kubernetes.io/projected/567a72f6-035d-4c61-acaf-f46008029168-kube-api-access-bc9kz") pod "network-check-target-ws655" (UID: "567a72f6-035d-4c61-acaf-f46008029168") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 20:37:49.729448 ip-10-0-138-149 kubenswrapper[2573]: I1030 20:37:49.729367 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ws655" Oct 30 20:37:49.729448 ip-10-0-138-149 kubenswrapper[2573]: I1030 20:37:49.729389 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-nhs9k" Oct 30 20:37:49.729681 ip-10-0-138-149 kubenswrapper[2573]: I1030 20:37:49.729462 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-bcmwc" Oct 30 20:37:49.729681 ip-10-0-138-149 kubenswrapper[2573]: E1030 20:37:49.729478 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-ws655" podUID="567a72f6-035d-4c61-acaf-f46008029168" Oct 30 20:37:49.729681 ip-10-0-138-149 kubenswrapper[2573]: E1030 20:37:49.729550 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-bcmwc" podUID="0d6769f4-dc38-45ac-863b-c75fa672866a" Oct 30 20:37:49.729681 ip-10-0-138-149 kubenswrapper[2573]: E1030 20:37:49.729638 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-nhs9k" podUID="e886897a-c6a3-414f-875f-aa112541b608" Oct 30 20:37:49.987499 ip-10-0-138-149 kubenswrapper[2573]: I1030 20:37:49.987420 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 20:37:50.191514 ip-10-0-138-149 kubenswrapper[2573]: I1030 20:37:50.191471 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 20:37:50.593943 ip-10-0-138-149 kubenswrapper[2573]: I1030 20:37:50.593911 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 20:37:50.788310 ip-10-0-138-149 kubenswrapper[2573]: I1030 20:37:50.788277 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 20:37:51.386817 ip-10-0-138-149 kubenswrapper[2573]: I1030 20:37:51.386757 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 20:37:51.505677 ip-10-0-138-149 kubenswrapper[2573]: I1030 20:37:51.505644 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 20:37:51.729249 ip-10-0-138-149 kubenswrapper[2573]: I1030 20:37:51.729164 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-bcmwc" Oct 30 20:37:51.729401 ip-10-0-138-149 kubenswrapper[2573]: I1030 20:37:51.729167 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ws655" Oct 30 20:37:51.729401 ip-10-0-138-149 kubenswrapper[2573]: E1030 20:37:51.729287 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-bcmwc" podUID="0d6769f4-dc38-45ac-863b-c75fa672866a" Oct 30 20:37:51.729401 ip-10-0-138-149 kubenswrapper[2573]: I1030 20:37:51.729169 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-nhs9k" Oct 30 20:37:51.729548 ip-10-0-138-149 kubenswrapper[2573]: E1030 20:37:51.729328 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-ws655" podUID="567a72f6-035d-4c61-acaf-f46008029168" Oct 30 20:37:51.729548 ip-10-0-138-149 kubenswrapper[2573]: E1030 20:37:51.729434 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-nhs9k" podUID="e886897a-c6a3-414f-875f-aa112541b608" Oct 30 20:37:52.010818 ip-10-0-138-149 kubenswrapper[2573]: I1030 20:37:52.010740 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 20:37:52.174089 ip-10-0-138-149 kubenswrapper[2573]: I1030 20:37:52.174059 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 20:37:52.291398 ip-10-0-138-149 kubenswrapper[2573]: I1030 20:37:52.291324 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 20:37:52.587047 ip-10-0-138-149 kubenswrapper[2573]: I1030 20:37:52.586802 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 20:37:52.788740 ip-10-0-138-149 kubenswrapper[2573]: I1030 20:37:52.788708 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 20:37:53.384744 ip-10-0-138-149 kubenswrapper[2573]: I1030 20:37:53.384712 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 20:37:53.502776 ip-10-0-138-149 kubenswrapper[2573]: I1030 20:37:53.502742 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 20:37:53.729554 ip-10-0-138-149 kubenswrapper[2573]: I1030 20:37:53.729482 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-nhs9k" Oct 30 20:37:53.729907 ip-10-0-138-149 kubenswrapper[2573]: I1030 20:37:53.729486 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-bcmwc" Oct 30 20:37:53.729907 ip-10-0-138-149 kubenswrapper[2573]: E1030 20:37:53.729629 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-nhs9k" podUID="e886897a-c6a3-414f-875f-aa112541b608" Oct 30 20:37:53.729907 ip-10-0-138-149 kubenswrapper[2573]: I1030 20:37:53.729677 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ws655" Oct 30 20:37:53.729907 ip-10-0-138-149 kubenswrapper[2573]: E1030 20:37:53.729770 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-bcmwc" podUID="0d6769f4-dc38-45ac-863b-c75fa672866a" Oct 30 20:37:53.729907 ip-10-0-138-149 kubenswrapper[2573]: E1030 20:37:53.729836 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-ws655" podUID="567a72f6-035d-4c61-acaf-f46008029168" Oct 30 20:37:53.989151 ip-10-0-138-149 kubenswrapper[2573]: I1030 20:37:53.989080 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 20:37:54.189460 ip-10-0-138-149 kubenswrapper[2573]: I1030 20:37:54.189427 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 20:37:54.616662 ip-10-0-138-149 kubenswrapper[2573]: I1030 20:37:54.616631 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 20:37:54.783113 ip-10-0-138-149 kubenswrapper[2573]: I1030 20:37:54.783082 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 20:37:55.275459 ip-10-0-138-149 kubenswrapper[2573]: I1030 20:37:55.275430 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 20:37:55.398325 ip-10-0-138-149 kubenswrapper[2573]: I1030 20:37:55.398289 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 20:37:55.544723 ip-10-0-138-149 kubenswrapper[2573]: I1030 20:37:55.544648 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 20:37:55.662851 ip-10-0-138-149 kubenswrapper[2573]: I1030 20:37:55.662817 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 20:37:55.730403 ip-10-0-138-149 kubenswrapper[2573]: I1030 20:37:55.730376 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-nhs9k" Oct 30 20:37:55.730595 ip-10-0-138-149 kubenswrapper[2573]: E1030 20:37:55.730461 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-nhs9k" podUID="e886897a-c6a3-414f-875f-aa112541b608" Oct 30 20:37:55.730595 ip-10-0-138-149 kubenswrapper[2573]: I1030 20:37:55.730530 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ws655" Oct 30 20:37:55.730731 ip-10-0-138-149 kubenswrapper[2573]: I1030 20:37:55.730633 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-bcmwc" Oct 30 20:37:55.730731 ip-10-0-138-149 kubenswrapper[2573]: E1030 20:37:55.730637 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-ws655" podUID="567a72f6-035d-4c61-acaf-f46008029168" Oct 30 20:37:55.730731 ip-10-0-138-149 kubenswrapper[2573]: E1030 20:37:55.730716 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-bcmwc" podUID="0d6769f4-dc38-45ac-863b-c75fa672866a" Oct 30 20:37:55.989767 ip-10-0-138-149 kubenswrapper[2573]: I1030 20:37:55.989737 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 20:37:56.185667 ip-10-0-138-149 kubenswrapper[2573]: I1030 20:37:56.185643 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 20:37:56.317942 ip-10-0-138-149 kubenswrapper[2573]: I1030 20:37:56.317876 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 20:37:56.445861 ip-10-0-138-149 kubenswrapper[2573]: I1030 20:37:56.445827 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 20:37:56.780402 ip-10-0-138-149 kubenswrapper[2573]: I1030 20:37:56.780372 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 20:37:57.193342 ip-10-0-138-149 kubenswrapper[2573]: I1030 20:37:57.193313 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 20:37:57.390734 ip-10-0-138-149 kubenswrapper[2573]: I1030 20:37:57.390704 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 20:37:57.729860 ip-10-0-138-149 kubenswrapper[2573]: I1030 20:37:57.729827 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-bcmwc" Oct 30 20:37:57.729860 ip-10-0-138-149 kubenswrapper[2573]: I1030 20:37:57.729845 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ws655" Oct 30 20:37:57.729860 ip-10-0-138-149 kubenswrapper[2573]: I1030 20:37:57.729831 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-nhs9k" Oct 30 20:37:57.730163 ip-10-0-138-149 kubenswrapper[2573]: E1030 20:37:57.729948 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-ws655" podUID="567a72f6-035d-4c61-acaf-f46008029168" Oct 30 20:37:57.730163 ip-10-0-138-149 kubenswrapper[2573]: E1030 20:37:57.729976 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-nhs9k" podUID="e886897a-c6a3-414f-875f-aa112541b608" Oct 30 20:37:57.730163 ip-10-0-138-149 kubenswrapper[2573]: E1030 20:37:57.730056 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-bcmwc" podUID="0d6769f4-dc38-45ac-863b-c75fa672866a" Oct 30 20:37:59.152933 ip-10-0-138-149 kubenswrapper[2573]: I1030 20:37:59.152900 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 20:37:59.290234 ip-10-0-138-149 kubenswrapper[2573]: I1030 20:37:59.290204 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 20:37:59.729805 ip-10-0-138-149 kubenswrapper[2573]: I1030 20:37:59.729772 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ws655" Oct 30 20:37:59.729987 ip-10-0-138-149 kubenswrapper[2573]: I1030 20:37:59.729776 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-bcmwc" Oct 30 20:37:59.729987 ip-10-0-138-149 kubenswrapper[2573]: E1030 20:37:59.729893 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-ws655" podUID="567a72f6-035d-4c61-acaf-f46008029168" Oct 30 20:37:59.729987 ip-10-0-138-149 kubenswrapper[2573]: I1030 20:37:59.729776 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-nhs9k" Oct 30 20:37:59.729987 ip-10-0-138-149 kubenswrapper[2573]: E1030 20:37:59.729957 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-bcmwc" podUID="0d6769f4-dc38-45ac-863b-c75fa672866a" Oct 30 20:37:59.730196 ip-10-0-138-149 kubenswrapper[2573]: E1030 20:37:59.730023 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-nhs9k" podUID="e886897a-c6a3-414f-875f-aa112541b608" Oct 30 20:38:01.665219 ip-10-0-138-149 kubenswrapper[2573]: I1030 20:38:01.665182 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 20:38:01.729463 ip-10-0-138-149 kubenswrapper[2573]: I1030 20:38:01.729427 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-bcmwc" Oct 30 20:38:01.729463 ip-10-0-138-149 kubenswrapper[2573]: I1030 20:38:01.729454 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ws655" Oct 30 20:38:01.729739 ip-10-0-138-149 kubenswrapper[2573]: E1030 20:38:01.729549 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-bcmwc" podUID="0d6769f4-dc38-45ac-863b-c75fa672866a" Oct 30 20:38:01.729739 ip-10-0-138-149 kubenswrapper[2573]: I1030 20:38:01.729608 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-nhs9k" Oct 30 20:38:01.729739 ip-10-0-138-149 kubenswrapper[2573]: E1030 20:38:01.729683 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-nhs9k" podUID="e886897a-c6a3-414f-875f-aa112541b608" Oct 30 20:38:01.729901 ip-10-0-138-149 kubenswrapper[2573]: E1030 20:38:01.729785 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-ws655" podUID="567a72f6-035d-4c61-acaf-f46008029168" Oct 30 20:38:01.811337 ip-10-0-138-149 kubenswrapper[2573]: I1030 20:38:01.811303 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 20:38:03.115736 ip-10-0-138-149 kubenswrapper[2573]: I1030 20:38:03.115693 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 20:38:03.420409 ip-10-0-138-149 kubenswrapper[2573]: I1030 20:38:03.420332 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 20:38:03.640542 ip-10-0-138-149 kubenswrapper[2573]: I1030 20:38:03.640514 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 20:38:03.729532 ip-10-0-138-149 kubenswrapper[2573]: I1030 20:38:03.729444 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-bcmwc" Oct 30 20:38:03.729532 ip-10-0-138-149 kubenswrapper[2573]: I1030 20:38:03.729473 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ws655" Oct 30 20:38:03.729709 ip-10-0-138-149 kubenswrapper[2573]: I1030 20:38:03.729452 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-nhs9k" Oct 30 20:38:03.729709 ip-10-0-138-149 kubenswrapper[2573]: E1030 20:38:03.729560 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-bcmwc" podUID="0d6769f4-dc38-45ac-863b-c75fa672866a" Oct 30 20:38:03.729709 ip-10-0-138-149 kubenswrapper[2573]: E1030 20:38:03.729630 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-ws655" podUID="567a72f6-035d-4c61-acaf-f46008029168" Oct 30 20:38:03.729709 ip-10-0-138-149 kubenswrapper[2573]: E1030 20:38:03.729692 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-nhs9k" podUID="e886897a-c6a3-414f-875f-aa112541b608" Oct 30 20:38:03.877427 ip-10-0-138-149 kubenswrapper[2573]: I1030 20:38:03.877394 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 20:38:04.508018 ip-10-0-138-149 kubenswrapper[2573]: I1030 20:38:04.507943 2573 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-138-149.us-west-1.compute.internal" event="NodeReady" Oct 30 20:38:04.508395 ip-10-0-138-149 kubenswrapper[2573]: I1030 20:38:04.508060 2573 kubelet_node_status.go:477] "Fast updating node status as it just became ready" Oct 30 20:38:04.623816 ip-10-0-138-149 kubenswrapper[2573]: I1030 20:38:04.623788 2573 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-brbhg"] Oct 30 20:38:04.626746 ip-10-0-138-149 kubenswrapper[2573]: I1030 20:38:04.626731 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-brbhg" Oct 30 20:38:04.645425 ip-10-0-138-149 kubenswrapper[2573]: I1030 20:38:04.645403 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sg8pj\" (UniqueName: \"kubernetes.io/projected/6a710ade-5a54-4595-89a8-636653c02fc5-kube-api-access-sg8pj\") pod \"ingress-canary-brbhg\" (UID: \"6a710ade-5a54-4595-89a8-636653c02fc5\") " pod="openshift-ingress-canary/ingress-canary-brbhg" Oct 30 20:38:04.645551 ip-10-0-138-149 kubenswrapper[2573]: I1030 20:38:04.645445 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/6a710ade-5a54-4595-89a8-636653c02fc5-cert\") pod \"ingress-canary-brbhg\" (UID: \"6a710ade-5a54-4595-89a8-636653c02fc5\") " pod="openshift-ingress-canary/ingress-canary-brbhg" Oct 30 20:38:04.672397 ip-10-0-138-149 kubenswrapper[2573]: I1030 20:38:04.672372 2573 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-q6mx7"] Oct 30 20:38:04.675256 ip-10-0-138-149 kubenswrapper[2573]: I1030 20:38:04.675242 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-q6mx7" Oct 30 20:38:04.680526 ip-10-0-138-149 kubenswrapper[2573]: I1030 20:38:04.680489 2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Oct 30 20:38:04.680648 ip-10-0-138-149 kubenswrapper[2573]: I1030 20:38:04.680635 2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Oct 30 20:38:04.680715 ip-10-0-138-149 kubenswrapper[2573]: I1030 20:38:04.680702 2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Oct 30 20:38:04.680905 ip-10-0-138-149 kubenswrapper[2573]: I1030 20:38:04.680800 2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"ingress-canary-dockercfg-h8n42\"" Oct 30 20:38:04.689321 ip-10-0-138-149 kubenswrapper[2573]: I1030 20:38:04.689302 2573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-brbhg"] Oct 30 20:38:04.708418 ip-10-0-138-149 kubenswrapper[2573]: I1030 20:38:04.708392 2573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-q6mx7"] Oct 30 20:38:04.720317 ip-10-0-138-149 kubenswrapper[2573]: I1030 20:38:04.720298 2573 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-8wwgv"] Oct 30 20:38:04.723054 ip-10-0-138-149 kubenswrapper[2573]: I1030 20:38:04.723041 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-8wwgv" Oct 30 20:38:04.746226 ip-10-0-138-149 kubenswrapper[2573]: I1030 20:38:04.746204 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/743b90bd-3330-4751-a853-80b97fee0890-config-volume\") pod \"dns-default-8wwgv\" (UID: \"743b90bd-3330-4751-a853-80b97fee0890\") " pod="openshift-dns/dns-default-8wwgv" Oct 30 20:38:04.746320 ip-10-0-138-149 kubenswrapper[2573]: I1030 20:38:04.746236 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/743b90bd-3330-4751-a853-80b97fee0890-tmp-dir\") pod \"dns-default-8wwgv\" (UID: \"743b90bd-3330-4751-a853-80b97fee0890\") " pod="openshift-dns/dns-default-8wwgv" Oct 30 20:38:04.746320 ip-10-0-138-149 kubenswrapper[2573]: I1030 20:38:04.746261 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sg8pj\" (UniqueName: \"kubernetes.io/projected/6a710ade-5a54-4595-89a8-636653c02fc5-kube-api-access-sg8pj\") pod \"ingress-canary-brbhg\" (UID: \"6a710ade-5a54-4595-89a8-636653c02fc5\") " pod="openshift-ingress-canary/ingress-canary-brbhg" Oct 30 20:38:04.746320 ip-10-0-138-149 kubenswrapper[2573]: I1030 20:38:04.746277 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/743b90bd-3330-4751-a853-80b97fee0890-metrics-tls\") pod \"dns-default-8wwgv\" (UID: \"743b90bd-3330-4751-a853-80b97fee0890\") " pod="openshift-dns/dns-default-8wwgv" Oct 30 20:38:04.746431 ip-10-0-138-149 kubenswrapper[2573]: I1030 20:38:04.746326 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5x2gc\" (UniqueName: \"kubernetes.io/projected/743b90bd-3330-4751-a853-80b97fee0890-kube-api-access-5x2gc\") pod \"dns-default-8wwgv\" (UID: \"743b90bd-3330-4751-a853-80b97fee0890\") " pod="openshift-dns/dns-default-8wwgv" Oct 30 20:38:04.746499 ip-10-0-138-149 kubenswrapper[2573]: I1030 20:38:04.746481 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/6a710ade-5a54-4595-89a8-636653c02fc5-cert\") pod \"ingress-canary-brbhg\" (UID: \"6a710ade-5a54-4595-89a8-636653c02fc5\") " pod="openshift-ingress-canary/ingress-canary-brbhg" Oct 30 20:38:04.746546 ip-10-0-138-149 kubenswrapper[2573]: I1030 20:38:04.746517 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fwkdk\" (UniqueName: \"kubernetes.io/projected/59406b2a-a784-4d6e-aedf-96e11ff99f68-kube-api-access-fwkdk\") pod \"insights-runtime-extractor-q6mx7\" (UID: \"59406b2a-a784-4d6e-aedf-96e11ff99f68\") " pod="openshift-insights/insights-runtime-extractor-q6mx7" Oct 30 20:38:04.746619 ip-10-0-138-149 kubenswrapper[2573]: I1030 20:38:04.746559 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/59406b2a-a784-4d6e-aedf-96e11ff99f68-crio-socket\") pod \"insights-runtime-extractor-q6mx7\" (UID: \"59406b2a-a784-4d6e-aedf-96e11ff99f68\") " pod="openshift-insights/insights-runtime-extractor-q6mx7" Oct 30 20:38:04.746619 ip-10-0-138-149 kubenswrapper[2573]: I1030 20:38:04.746605 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/59406b2a-a784-4d6e-aedf-96e11ff99f68-data-volume\") pod \"insights-runtime-extractor-q6mx7\" (UID: \"59406b2a-a784-4d6e-aedf-96e11ff99f68\") " pod="openshift-insights/insights-runtime-extractor-q6mx7" Oct 30 20:38:04.746722 ip-10-0-138-149 kubenswrapper[2573]: I1030 20:38:04.746666 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/59406b2a-a784-4d6e-aedf-96e11ff99f68-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-q6mx7\" (UID: \"59406b2a-a784-4d6e-aedf-96e11ff99f68\") " pod="openshift-insights/insights-runtime-extractor-q6mx7" Oct 30 20:38:04.746722 ip-10-0-138-149 kubenswrapper[2573]: I1030 20:38:04.746693 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/59406b2a-a784-4d6e-aedf-96e11ff99f68-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-q6mx7\" (UID: \"59406b2a-a784-4d6e-aedf-96e11ff99f68\") " pod="openshift-insights/insights-runtime-extractor-q6mx7" Oct 30 20:38:04.750476 ip-10-0-138-149 kubenswrapper[2573]: I1030 20:38:04.750453 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/6a710ade-5a54-4595-89a8-636653c02fc5-cert\") pod \"ingress-canary-brbhg\" (UID: \"6a710ade-5a54-4595-89a8-636653c02fc5\") " pod="openshift-ingress-canary/ingress-canary-brbhg" Oct 30 20:38:04.782325 ip-10-0-138-149 kubenswrapper[2573]: I1030 20:38:04.782270 2573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-8wwgv"] Oct 30 20:38:04.783154 ip-10-0-138-149 kubenswrapper[2573]: I1030 20:38:04.783129 2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Oct 30 20:38:04.783269 ip-10-0-138-149 kubenswrapper[2573]: I1030 20:38:04.783138 2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Oct 30 20:38:04.783269 ip-10-0-138-149 kubenswrapper[2573]: I1030 20:38:04.783165 2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Oct 30 20:38:04.783269 ip-10-0-138-149 kubenswrapper[2573]: I1030 20:38:04.783237 2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Oct 30 20:38:04.783269 ip-10-0-138-149 kubenswrapper[2573]: I1030 20:38:04.783254 2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-2srwc\"" Oct 30 20:38:04.783438 ip-10-0-138-149 kubenswrapper[2573]: I1030 20:38:04.783280 2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Oct 30 20:38:04.783438 ip-10-0-138-149 kubenswrapper[2573]: I1030 20:38:04.783371 2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-wsjv8\"" Oct 30 20:38:04.783438 ip-10-0-138-149 kubenswrapper[2573]: I1030 20:38:04.783405 2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Oct 30 20:38:04.809441 ip-10-0-138-149 kubenswrapper[2573]: I1030 20:38:04.809422 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sg8pj\" (UniqueName: \"kubernetes.io/projected/6a710ade-5a54-4595-89a8-636653c02fc5-kube-api-access-sg8pj\") pod \"ingress-canary-brbhg\" (UID: \"6a710ade-5a54-4595-89a8-636653c02fc5\") " pod="openshift-ingress-canary/ingress-canary-brbhg" Oct 30 20:38:04.846975 ip-10-0-138-149 kubenswrapper[2573]: I1030 20:38:04.846950 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/59406b2a-a784-4d6e-aedf-96e11ff99f68-crio-socket\") pod \"insights-runtime-extractor-q6mx7\" (UID: \"59406b2a-a784-4d6e-aedf-96e11ff99f68\") " pod="openshift-insights/insights-runtime-extractor-q6mx7" Oct 30 20:38:04.847092 ip-10-0-138-149 kubenswrapper[2573]: I1030 20:38:04.846987 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/59406b2a-a784-4d6e-aedf-96e11ff99f68-data-volume\") pod \"insights-runtime-extractor-q6mx7\" (UID: \"59406b2a-a784-4d6e-aedf-96e11ff99f68\") " pod="openshift-insights/insights-runtime-extractor-q6mx7" Oct 30 20:38:04.847092 ip-10-0-138-149 kubenswrapper[2573]: I1030 20:38:04.847010 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/59406b2a-a784-4d6e-aedf-96e11ff99f68-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-q6mx7\" (UID: \"59406b2a-a784-4d6e-aedf-96e11ff99f68\") " pod="openshift-insights/insights-runtime-extractor-q6mx7" Oct 30 20:38:04.847092 ip-10-0-138-149 kubenswrapper[2573]: I1030 20:38:04.847072 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/59406b2a-a784-4d6e-aedf-96e11ff99f68-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-q6mx7\" (UID: \"59406b2a-a784-4d6e-aedf-96e11ff99f68\") " pod="openshift-insights/insights-runtime-extractor-q6mx7" Oct 30 20:38:04.847250 ip-10-0-138-149 kubenswrapper[2573]: I1030 20:38:04.847131 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/743b90bd-3330-4751-a853-80b97fee0890-config-volume\") pod \"dns-default-8wwgv\" (UID: \"743b90bd-3330-4751-a853-80b97fee0890\") " pod="openshift-dns/dns-default-8wwgv" Oct 30 20:38:04.847250 ip-10-0-138-149 kubenswrapper[2573]: I1030 20:38:04.847162 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/59406b2a-a784-4d6e-aedf-96e11ff99f68-crio-socket\") pod \"insights-runtime-extractor-q6mx7\" (UID: \"59406b2a-a784-4d6e-aedf-96e11ff99f68\") " pod="openshift-insights/insights-runtime-extractor-q6mx7" Oct 30 20:38:04.847250 ip-10-0-138-149 kubenswrapper[2573]: I1030 20:38:04.847174 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/743b90bd-3330-4751-a853-80b97fee0890-tmp-dir\") pod \"dns-default-8wwgv\" (UID: \"743b90bd-3330-4751-a853-80b97fee0890\") " pod="openshift-dns/dns-default-8wwgv" Oct 30 20:38:04.847250 ip-10-0-138-149 kubenswrapper[2573]: I1030 20:38:04.847221 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/743b90bd-3330-4751-a853-80b97fee0890-metrics-tls\") pod \"dns-default-8wwgv\" (UID: \"743b90bd-3330-4751-a853-80b97fee0890\") " pod="openshift-dns/dns-default-8wwgv" Oct 30 20:38:04.847436 ip-10-0-138-149 kubenswrapper[2573]: I1030 20:38:04.847311 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/59406b2a-a784-4d6e-aedf-96e11ff99f68-data-volume\") pod \"insights-runtime-extractor-q6mx7\" (UID: \"59406b2a-a784-4d6e-aedf-96e11ff99f68\") " pod="openshift-insights/insights-runtime-extractor-q6mx7" Oct 30 20:38:04.847436 ip-10-0-138-149 kubenswrapper[2573]: I1030 20:38:04.847314 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5x2gc\" (UniqueName: \"kubernetes.io/projected/743b90bd-3330-4751-a853-80b97fee0890-kube-api-access-5x2gc\") pod \"dns-default-8wwgv\" (UID: \"743b90bd-3330-4751-a853-80b97fee0890\") " pod="openshift-dns/dns-default-8wwgv" Oct 30 20:38:04.847436 ip-10-0-138-149 kubenswrapper[2573]: I1030 20:38:04.847396 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fwkdk\" (UniqueName: \"kubernetes.io/projected/59406b2a-a784-4d6e-aedf-96e11ff99f68-kube-api-access-fwkdk\") pod \"insights-runtime-extractor-q6mx7\" (UID: \"59406b2a-a784-4d6e-aedf-96e11ff99f68\") " pod="openshift-insights/insights-runtime-extractor-q6mx7" Oct 30 20:38:04.847598 ip-10-0-138-149 kubenswrapper[2573]: I1030 20:38:04.847582 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/743b90bd-3330-4751-a853-80b97fee0890-tmp-dir\") pod \"dns-default-8wwgv\" (UID: \"743b90bd-3330-4751-a853-80b97fee0890\") " pod="openshift-dns/dns-default-8wwgv" Oct 30 20:38:04.847710 ip-10-0-138-149 kubenswrapper[2573]: I1030 20:38:04.847694 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/59406b2a-a784-4d6e-aedf-96e11ff99f68-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-q6mx7\" (UID: \"59406b2a-a784-4d6e-aedf-96e11ff99f68\") " pod="openshift-insights/insights-runtime-extractor-q6mx7" Oct 30 20:38:04.847884 ip-10-0-138-149 kubenswrapper[2573]: I1030 20:38:04.847866 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/743b90bd-3330-4751-a853-80b97fee0890-config-volume\") pod \"dns-default-8wwgv\" (UID: \"743b90bd-3330-4751-a853-80b97fee0890\") " pod="openshift-dns/dns-default-8wwgv" Oct 30 20:38:04.849425 ip-10-0-138-149 kubenswrapper[2573]: I1030 20:38:04.849399 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/59406b2a-a784-4d6e-aedf-96e11ff99f68-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-q6mx7\" (UID: \"59406b2a-a784-4d6e-aedf-96e11ff99f68\") " pod="openshift-insights/insights-runtime-extractor-q6mx7" Oct 30 20:38:04.849522 ip-10-0-138-149 kubenswrapper[2573]: I1030 20:38:04.849506 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/743b90bd-3330-4751-a853-80b97fee0890-metrics-tls\") pod \"dns-default-8wwgv\" (UID: \"743b90bd-3330-4751-a853-80b97fee0890\") " pod="openshift-dns/dns-default-8wwgv" Oct 30 20:38:04.920281 ip-10-0-138-149 kubenswrapper[2573]: I1030 20:38:04.920254 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fwkdk\" (UniqueName: \"kubernetes.io/projected/59406b2a-a784-4d6e-aedf-96e11ff99f68-kube-api-access-fwkdk\") pod \"insights-runtime-extractor-q6mx7\" (UID: \"59406b2a-a784-4d6e-aedf-96e11ff99f68\") " pod="openshift-insights/insights-runtime-extractor-q6mx7" Oct 30 20:38:04.920363 ip-10-0-138-149 kubenswrapper[2573]: I1030 20:38:04.920256 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5x2gc\" (UniqueName: \"kubernetes.io/projected/743b90bd-3330-4751-a853-80b97fee0890-kube-api-access-5x2gc\") pod \"dns-default-8wwgv\" (UID: \"743b90bd-3330-4751-a853-80b97fee0890\") " pod="openshift-dns/dns-default-8wwgv" Oct 30 20:38:04.939116 ip-10-0-138-149 kubenswrapper[2573]: I1030 20:38:04.939097 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-brbhg" Oct 30 20:38:04.989868 ip-10-0-138-149 kubenswrapper[2573]: I1030 20:38:04.989843 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-q6mx7" Oct 30 20:38:05.035190 ip-10-0-138-149 kubenswrapper[2573]: I1030 20:38:05.035121 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-8wwgv" Oct 30 20:38:05.318053 ip-10-0-138-149 kubenswrapper[2573]: I1030 20:38:05.317984 2573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-brbhg"] Oct 30 20:38:05.321323 ip-10-0-138-149 kubenswrapper[2573]: W1030 20:38:05.321289 2573 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6a710ade_5a54_4595_89a8_636653c02fc5.slice/crio-8e02897398b972d7732ae296827399ab18c15cdc39c3d7dddec36ba3ebb2b7dc WatchSource:0}: Error finding container 8e02897398b972d7732ae296827399ab18c15cdc39c3d7dddec36ba3ebb2b7dc: Status 404 returned error can't find the container with id 8e02897398b972d7732ae296827399ab18c15cdc39c3d7dddec36ba3ebb2b7dc Oct 30 20:38:05.388957 ip-10-0-138-149 kubenswrapper[2573]: I1030 20:38:05.388931 2573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-q6mx7"] Oct 30 20:38:05.391879 ip-10-0-138-149 kubenswrapper[2573]: W1030 20:38:05.391857 2573 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod59406b2a_a784_4d6e_aedf_96e11ff99f68.slice/crio-9eae007fd35b17e1984bf02196ddc6a8933637392625a815cfb4747fde9e2909 WatchSource:0}: Error finding container 9eae007fd35b17e1984bf02196ddc6a8933637392625a815cfb4747fde9e2909: Status 404 returned error can't find the container with id 9eae007fd35b17e1984bf02196ddc6a8933637392625a815cfb4747fde9e2909 Oct 30 20:38:05.432286 ip-10-0-138-149 kubenswrapper[2573]: I1030 20:38:05.432265 2573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-8wwgv"] Oct 30 20:38:05.434451 ip-10-0-138-149 kubenswrapper[2573]: W1030 20:38:05.434430 2573 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod743b90bd_3330_4751_a853_80b97fee0890.slice/crio-b09ead73fb86a2392baeba9c87b84092951ed110487de47adab28bee7abb7826 WatchSource:0}: Error finding container b09ead73fb86a2392baeba9c87b84092951ed110487de47adab28bee7abb7826: Status 404 returned error can't find the container with id b09ead73fb86a2392baeba9c87b84092951ed110487de47adab28bee7abb7826 Oct 30 20:38:05.701843 ip-10-0-138-149 kubenswrapper[2573]: I1030 20:38:05.701797 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 20:38:05.733228 ip-10-0-138-149 kubenswrapper[2573]: I1030 20:38:05.733195 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-bcmwc" Oct 30 20:38:05.733404 ip-10-0-138-149 kubenswrapper[2573]: I1030 20:38:05.733375 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ws655" Oct 30 20:38:05.733545 ip-10-0-138-149 kubenswrapper[2573]: I1030 20:38:05.733516 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-nhs9k" Oct 30 20:38:05.786169 ip-10-0-138-149 kubenswrapper[2573]: I1030 20:38:05.786144 2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Oct 30 20:38:05.786169 ip-10-0-138-149 kubenswrapper[2573]: I1030 20:38:05.786152 2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-4lpn7\"" Oct 30 20:38:05.786361 ip-10-0-138-149 kubenswrapper[2573]: I1030 20:38:05.786181 2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Oct 30 20:38:05.786361 ip-10-0-138-149 kubenswrapper[2573]: I1030 20:38:05.786218 2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Oct 30 20:38:05.786361 ip-10-0-138-149 kubenswrapper[2573]: I1030 20:38:05.786314 2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-49f2v\"" Oct 30 20:38:05.786361 ip-10-0-138-149 kubenswrapper[2573]: I1030 20:38:05.786349 2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Oct 30 20:38:05.786361 ip-10-0-138-149 kubenswrapper[2573]: I1030 20:38:05.786350 2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-hjmcj\"" Oct 30 20:38:05.983713 ip-10-0-138-149 kubenswrapper[2573]: I1030 20:38:05.983631 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-brbhg" event={"ID":"6a710ade-5a54-4595-89a8-636653c02fc5","Type":"ContainerStarted","Data":"8e02897398b972d7732ae296827399ab18c15cdc39c3d7dddec36ba3ebb2b7dc"} Oct 30 20:38:05.984680 ip-10-0-138-149 kubenswrapper[2573]: I1030 20:38:05.984656 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-8wwgv" event={"ID":"743b90bd-3330-4751-a853-80b97fee0890","Type":"ContainerStarted","Data":"b09ead73fb86a2392baeba9c87b84092951ed110487de47adab28bee7abb7826"} Oct 30 20:38:05.986090 ip-10-0-138-149 kubenswrapper[2573]: I1030 20:38:05.986069 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-q6mx7" event={"ID":"59406b2a-a784-4d6e-aedf-96e11ff99f68","Type":"ContainerStarted","Data":"c2772a361c1432319ab050aec4d200f8f66f72f203dce9e8999895552dc785d9"} Oct 30 20:38:05.986179 ip-10-0-138-149 kubenswrapper[2573]: I1030 20:38:05.986118 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-q6mx7" event={"ID":"59406b2a-a784-4d6e-aedf-96e11ff99f68","Type":"ContainerStarted","Data":"9eae007fd35b17e1984bf02196ddc6a8933637392625a815cfb4747fde9e2909"} Oct 30 20:38:06.086741 ip-10-0-138-149 kubenswrapper[2573]: I1030 20:38:06.086712 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 20:38:06.253779 ip-10-0-138-149 kubenswrapper[2573]: I1030 20:38:06.253685 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 20:38:06.392965 ip-10-0-138-149 kubenswrapper[2573]: I1030 20:38:06.392937 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 20:38:06.524919 ip-10-0-138-149 kubenswrapper[2573]: I1030 20:38:06.524832 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 20:38:06.646216 ip-10-0-138-149 kubenswrapper[2573]: I1030 20:38:06.646187 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 20:38:06.792533 ip-10-0-138-149 kubenswrapper[2573]: I1030 20:38:06.792463 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 20:38:06.924940 ip-10-0-138-149 kubenswrapper[2573]: I1030 20:38:06.924906 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 20:38:06.991605 ip-10-0-138-149 kubenswrapper[2573]: I1030 20:38:06.991550 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-q6mx7" event={"ID":"59406b2a-a784-4d6e-aedf-96e11ff99f68","Type":"ContainerStarted","Data":"e61d8bfcb09b062241b3436052ba1f1a43864fb0f067eb16d3b13045fe0e13b7"} Oct 30 20:38:07.024088 ip-10-0-138-149 kubenswrapper[2573]: I1030 20:38:07.024056 2573 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-9cd89b74c-2xw4j"] Oct 30 20:38:07.027395 ip-10-0-138-149 kubenswrapper[2573]: I1030 20:38:07.027363 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-9cd89b74c-2xw4j" Oct 30 20:38:07.075316 ip-10-0-138-149 kubenswrapper[2573]: I1030 20:38:07.075244 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/5186676f-8c82-4667-a013-3efc22d2c4e7-prometheus-operator-tls\") pod \"prometheus-operator-9cd89b74c-2xw4j\" (UID: \"5186676f-8c82-4667-a013-3efc22d2c4e7\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-2xw4j" Oct 30 20:38:07.075316 ip-10-0-138-149 kubenswrapper[2573]: I1030 20:38:07.075293 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/5186676f-8c82-4667-a013-3efc22d2c4e7-metrics-client-ca\") pod \"prometheus-operator-9cd89b74c-2xw4j\" (UID: \"5186676f-8c82-4667-a013-3efc22d2c4e7\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-2xw4j" Oct 30 20:38:07.075459 ip-10-0-138-149 kubenswrapper[2573]: I1030 20:38:07.075315 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/5186676f-8c82-4667-a013-3efc22d2c4e7-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-9cd89b74c-2xw4j\" (UID: \"5186676f-8c82-4667-a013-3efc22d2c4e7\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-2xw4j" Oct 30 20:38:07.075459 ip-10-0-138-149 kubenswrapper[2573]: I1030 20:38:07.075347 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9d2s8\" (UniqueName: \"kubernetes.io/projected/5186676f-8c82-4667-a013-3efc22d2c4e7-kube-api-access-9d2s8\") pod \"prometheus-operator-9cd89b74c-2xw4j\" (UID: \"5186676f-8c82-4667-a013-3efc22d2c4e7\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-2xw4j" Oct 30 20:38:07.079872 ip-10-0-138-149 kubenswrapper[2573]: I1030 20:38:07.079646 2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Oct 30 20:38:07.079872 ip-10-0-138-149 kubenswrapper[2573]: I1030 20:38:07.079766 2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-kube-rbac-proxy-config\"" Oct 30 20:38:07.079872 ip-10-0-138-149 kubenswrapper[2573]: I1030 20:38:07.079795 2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Oct 30 20:38:07.079872 ip-10-0-138-149 kubenswrapper[2573]: I1030 20:38:07.079803 2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-dockercfg-dttss\"" Oct 30 20:38:07.079872 ip-10-0-138-149 kubenswrapper[2573]: I1030 20:38:07.079834 2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-tls\"" Oct 30 20:38:07.080177 ip-10-0-138-149 kubenswrapper[2573]: I1030 20:38:07.079903 2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" Oct 30 20:38:07.090039 ip-10-0-138-149 kubenswrapper[2573]: I1030 20:38:07.090012 2573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-9cd89b74c-2xw4j"] Oct 30 20:38:07.094805 ip-10-0-138-149 kubenswrapper[2573]: I1030 20:38:07.094766 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 20:38:07.176089 ip-10-0-138-149 kubenswrapper[2573]: I1030 20:38:07.176055 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/5186676f-8c82-4667-a013-3efc22d2c4e7-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-9cd89b74c-2xw4j\" (UID: \"5186676f-8c82-4667-a013-3efc22d2c4e7\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-2xw4j" Oct 30 20:38:07.176227 ip-10-0-138-149 kubenswrapper[2573]: I1030 20:38:07.176133 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9d2s8\" (UniqueName: \"kubernetes.io/projected/5186676f-8c82-4667-a013-3efc22d2c4e7-kube-api-access-9d2s8\") pod \"prometheus-operator-9cd89b74c-2xw4j\" (UID: \"5186676f-8c82-4667-a013-3efc22d2c4e7\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-2xw4j" Oct 30 20:38:07.176227 ip-10-0-138-149 kubenswrapper[2573]: I1030 20:38:07.176210 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/5186676f-8c82-4667-a013-3efc22d2c4e7-prometheus-operator-tls\") pod \"prometheus-operator-9cd89b74c-2xw4j\" (UID: \"5186676f-8c82-4667-a013-3efc22d2c4e7\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-2xw4j" Oct 30 20:38:07.176337 ip-10-0-138-149 kubenswrapper[2573]: I1030 20:38:07.176258 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/5186676f-8c82-4667-a013-3efc22d2c4e7-metrics-client-ca\") pod \"prometheus-operator-9cd89b74c-2xw4j\" (UID: \"5186676f-8c82-4667-a013-3efc22d2c4e7\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-2xw4j" Oct 30 20:38:07.177016 ip-10-0-138-149 kubenswrapper[2573]: I1030 20:38:07.176987 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/5186676f-8c82-4667-a013-3efc22d2c4e7-metrics-client-ca\") pod \"prometheus-operator-9cd89b74c-2xw4j\" (UID: \"5186676f-8c82-4667-a013-3efc22d2c4e7\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-2xw4j" Oct 30 20:38:07.178913 ip-10-0-138-149 kubenswrapper[2573]: I1030 20:38:07.178889 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/5186676f-8c82-4667-a013-3efc22d2c4e7-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-9cd89b74c-2xw4j\" (UID: \"5186676f-8c82-4667-a013-3efc22d2c4e7\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-2xw4j" Oct 30 20:38:07.179699 ip-10-0-138-149 kubenswrapper[2573]: I1030 20:38:07.179676 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/5186676f-8c82-4667-a013-3efc22d2c4e7-prometheus-operator-tls\") pod \"prometheus-operator-9cd89b74c-2xw4j\" (UID: \"5186676f-8c82-4667-a013-3efc22d2c4e7\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-2xw4j" Oct 30 20:38:07.233120 ip-10-0-138-149 kubenswrapper[2573]: I1030 20:38:07.233093 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9d2s8\" (UniqueName: \"kubernetes.io/projected/5186676f-8c82-4667-a013-3efc22d2c4e7-kube-api-access-9d2s8\") pod \"prometheus-operator-9cd89b74c-2xw4j\" (UID: \"5186676f-8c82-4667-a013-3efc22d2c4e7\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-2xw4j" Oct 30 20:38:07.343846 ip-10-0-138-149 kubenswrapper[2573]: I1030 20:38:07.343398 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-9cd89b74c-2xw4j" Oct 30 20:38:07.439010 ip-10-0-138-149 kubenswrapper[2573]: I1030 20:38:07.438977 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 20:38:07.566993 ip-10-0-138-149 kubenswrapper[2573]: I1030 20:38:07.566946 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 20:38:07.720457 ip-10-0-138-149 kubenswrapper[2573]: I1030 20:38:07.720397 2573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-9cd89b74c-2xw4j"] Oct 30 20:38:07.724982 ip-10-0-138-149 kubenswrapper[2573]: W1030 20:38:07.724937 2573 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5186676f_8c82_4667_a013_3efc22d2c4e7.slice/crio-45cc675d83bd08ae9d2107e90740daab9497b910e06df3435048aae121d8dcab WatchSource:0}: Error finding container 45cc675d83bd08ae9d2107e90740daab9497b910e06df3435048aae121d8dcab: Status 404 returned error can't find the container with id 45cc675d83bd08ae9d2107e90740daab9497b910e06df3435048aae121d8dcab Oct 30 20:38:07.850474 ip-10-0-138-149 kubenswrapper[2573]: I1030 20:38:07.850433 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 20:38:07.986432 ip-10-0-138-149 kubenswrapper[2573]: I1030 20:38:07.986361 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 20:38:07.996720 ip-10-0-138-149 kubenswrapper[2573]: I1030 20:38:07.996683 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-9cd89b74c-2xw4j" event={"ID":"5186676f-8c82-4667-a013-3efc22d2c4e7","Type":"ContainerStarted","Data":"45cc675d83bd08ae9d2107e90740daab9497b910e06df3435048aae121d8dcab"} Oct 30 20:38:08.846476 ip-10-0-138-149 kubenswrapper[2573]: I1030 20:38:08.846450 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 20:38:08.983746 ip-10-0-138-149 kubenswrapper[2573]: I1030 20:38:08.983722 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 20:38:09.000402 ip-10-0-138-149 kubenswrapper[2573]: I1030 20:38:09.000374 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-8wwgv" event={"ID":"743b90bd-3330-4751-a853-80b97fee0890","Type":"ContainerStarted","Data":"722497b95d0d4cb7322c6cc0150922d3db4c6dad83cb609533893da556cb0d64"} Oct 30 20:38:09.000557 ip-10-0-138-149 kubenswrapper[2573]: I1030 20:38:09.000427 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-8wwgv" event={"ID":"743b90bd-3330-4751-a853-80b97fee0890","Type":"ContainerStarted","Data":"f40d6fba5bb2ea274cafee4fb1b6b8aa5f57fa814794ec3dc357bf199f7774a4"} Oct 30 20:38:09.000557 ip-10-0-138-149 kubenswrapper[2573]: I1030 20:38:09.000490 2573 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-8wwgv" Oct 30 20:38:09.117667 ip-10-0-138-149 kubenswrapper[2573]: I1030 20:38:09.117619 2573 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-dns/dns-default-8wwgv" podStartSLOduration=2.043206549 podStartE2EDuration="5.117606243s" totalImagesPullingTime="3.074399694s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:38:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:38:09.116669538 +0000 UTC m=+53.929120845" watchObservedRunningTime="2026-10-30 20:38:09.117606243 +0000 UTC m=+53.930057571" Oct 30 20:38:09.261608 ip-10-0-138-149 kubenswrapper[2573]: I1030 20:38:09.261580 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 20:38:09.380933 ip-10-0-138-149 kubenswrapper[2573]: I1030 20:38:09.380824 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 20:38:10.004403 ip-10-0-138-149 kubenswrapper[2573]: I1030 20:38:10.004370 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-q6mx7" event={"ID":"59406b2a-a784-4d6e-aedf-96e11ff99f68","Type":"ContainerStarted","Data":"27db750c31807d32c36c15be90ced42d82fcd080ca0ab0e7cc3cb53f569e1af2"} Oct 30 20:38:10.005877 ip-10-0-138-149 kubenswrapper[2573]: I1030 20:38:10.005854 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-brbhg" event={"ID":"6a710ade-5a54-4595-89a8-636653c02fc5","Type":"ContainerStarted","Data":"fb1965a8e228b37bd96c6504208138932a2600b2b4f71b8a1c072f84c671adc8"} Oct 30 20:38:10.119339 ip-10-0-138-149 kubenswrapper[2573]: I1030 20:38:10.119289 2573 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-q6mx7" podStartSLOduration=2.486662074 podStartE2EDuration="6.119276491s" totalImagesPullingTime="3.632614417s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 20:38:04 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:38:10.119262892 +0000 UTC m=+54.931714199" watchObservedRunningTime="2026-10-30 20:38:10.119276491 +0000 UTC m=+54.931727799" Oct 30 20:38:10.235900 ip-10-0-138-149 kubenswrapper[2573]: I1030 20:38:10.235860 2573 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-brbhg" podStartSLOduration=2.387280846 podStartE2EDuration="6.23584821s" totalImagesPullingTime="3.848567364s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:38:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:38:10.235245631 +0000 UTC m=+55.047696939" watchObservedRunningTime="2026-10-30 20:38:10.23584821 +0000 UTC m=+55.048299564" Oct 30 20:38:10.256238 ip-10-0-138-149 kubenswrapper[2573]: I1030 20:38:10.256210 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 20:38:10.639077 ip-10-0-138-149 kubenswrapper[2573]: I1030 20:38:10.639042 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 20:38:10.770161 ip-10-0-138-149 kubenswrapper[2573]: I1030 20:38:10.770126 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 20:38:11.010946 ip-10-0-138-149 kubenswrapper[2573]: I1030 20:38:11.010861 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-9cd89b74c-2xw4j" event={"ID":"5186676f-8c82-4667-a013-3efc22d2c4e7","Type":"ContainerStarted","Data":"2b7eac87dc9a02f3e6da9d4c5b3ed86261a36fae5e00cd51d30de3505e35b157"} Oct 30 20:38:11.010946 ip-10-0-138-149 kubenswrapper[2573]: I1030 20:38:11.010935 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-9cd89b74c-2xw4j" event={"ID":"5186676f-8c82-4667-a013-3efc22d2c4e7","Type":"ContainerStarted","Data":"a05da39f62cd6031847dddf3ed955ee719b97fe8d2aba325cc464df72847b6a0"} Oct 30 20:38:11.124349 ip-10-0-138-149 kubenswrapper[2573]: I1030 20:38:11.124301 2573 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-9cd89b74c-2xw4j" podStartSLOduration=2.631024846 podStartE2EDuration="5.124287192s" totalImagesPullingTime="2.493262346s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:38:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:38:11.12339383 +0000 UTC m=+55.935845139" watchObservedRunningTime="2026-10-30 20:38:11.124287192 +0000 UTC m=+55.936738500" Oct 30 20:38:11.640018 ip-10-0-138-149 kubenswrapper[2573]: I1030 20:38:11.639978 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 20:38:12.063644 ip-10-0-138-149 kubenswrapper[2573]: I1030 20:38:12.063556 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 20:38:12.190756 ip-10-0-138-149 kubenswrapper[2573]: I1030 20:38:12.190714 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 20:38:12.347267 ip-10-0-138-149 kubenswrapper[2573]: I1030 20:38:12.347242 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 20:38:12.474499 ip-10-0-138-149 kubenswrapper[2573]: I1030 20:38:12.474462 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 20:38:12.932268 ip-10-0-138-149 kubenswrapper[2573]: I1030 20:38:12.932237 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 20:38:13.873516 ip-10-0-138-149 kubenswrapper[2573]: I1030 20:38:13.873477 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 20:38:14.094009 ip-10-0-138-149 kubenswrapper[2573]: I1030 20:38:14.093970 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 20:38:14.244727 ip-10-0-138-149 kubenswrapper[2573]: I1030 20:38:14.244644 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 20:38:14.390787 ip-10-0-138-149 kubenswrapper[2573]: I1030 20:38:14.390747 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 20:38:14.575971 ip-10-0-138-149 kubenswrapper[2573]: I1030 20:38:14.575876 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 20:38:14.764264 ip-10-0-138-149 kubenswrapper[2573]: I1030 20:38:14.764235 2573 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-jzxqq"] Oct 30 20:38:14.766245 ip-10-0-138-149 kubenswrapper[2573]: I1030 20:38:14.766224 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 20:38:14.767455 ip-10-0-138-149 kubenswrapper[2573]: I1030 20:38:14.767441 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-jzxqq" Oct 30 20:38:14.832839 ip-10-0-138-149 kubenswrapper[2573]: I1030 20:38:14.832770 2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" Oct 30 20:38:14.832954 ip-10-0-138-149 kubenswrapper[2573]: I1030 20:38:14.832877 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/7df66b3c-abe2-4d20-abda-45320cd4af1b-metrics-client-ca\") pod \"node-exporter-jzxqq\" (UID: \"7df66b3c-abe2-4d20-abda-45320cd4af1b\") " pod="openshift-monitoring/node-exporter-jzxqq" Oct 30 20:38:14.832954 ip-10-0-138-149 kubenswrapper[2573]: I1030 20:38:14.832914 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/7df66b3c-abe2-4d20-abda-45320cd4af1b-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-jzxqq\" (UID: \"7df66b3c-abe2-4d20-abda-45320cd4af1b\") " pod="openshift-monitoring/node-exporter-jzxqq" Oct 30 20:38:14.832954 ip-10-0-138-149 kubenswrapper[2573]: I1030 20:38:14.832932 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/7df66b3c-abe2-4d20-abda-45320cd4af1b-root\") pod \"node-exporter-jzxqq\" (UID: \"7df66b3c-abe2-4d20-abda-45320cd4af1b\") " pod="openshift-monitoring/node-exporter-jzxqq" Oct 30 20:38:14.832954 ip-10-0-138-149 kubenswrapper[2573]: I1030 20:38:14.832947 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/7df66b3c-abe2-4d20-abda-45320cd4af1b-node-exporter-accelerators-collector-config\") pod \"node-exporter-jzxqq\" (UID: \"7df66b3c-abe2-4d20-abda-45320cd4af1b\") " pod="openshift-monitoring/node-exporter-jzxqq" Oct 30 20:38:14.833133 ip-10-0-138-149 kubenswrapper[2573]: I1030 20:38:14.833122 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/7df66b3c-abe2-4d20-abda-45320cd4af1b-node-exporter-textfile\") pod \"node-exporter-jzxqq\" (UID: \"7df66b3c-abe2-4d20-abda-45320cd4af1b\") " pod="openshift-monitoring/node-exporter-jzxqq" Oct 30 20:38:14.833170 ip-10-0-138-149 kubenswrapper[2573]: I1030 20:38:14.833142 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/7df66b3c-abe2-4d20-abda-45320cd4af1b-node-exporter-tls\") pod \"node-exporter-jzxqq\" (UID: \"7df66b3c-abe2-4d20-abda-45320cd4af1b\") " pod="openshift-monitoring/node-exporter-jzxqq" Oct 30 20:38:14.833170 ip-10-0-138-149 kubenswrapper[2573]: I1030 20:38:14.833158 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xxc6m\" (UniqueName: \"kubernetes.io/projected/7df66b3c-abe2-4d20-abda-45320cd4af1b-kube-api-access-xxc6m\") pod \"node-exporter-jzxqq\" (UID: \"7df66b3c-abe2-4d20-abda-45320cd4af1b\") " pod="openshift-monitoring/node-exporter-jzxqq" Oct 30 20:38:14.833264 ip-10-0-138-149 kubenswrapper[2573]: I1030 20:38:14.833175 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/7df66b3c-abe2-4d20-abda-45320cd4af1b-node-exporter-wtmp\") pod \"node-exporter-jzxqq\" (UID: \"7df66b3c-abe2-4d20-abda-45320cd4af1b\") " pod="openshift-monitoring/node-exporter-jzxqq" Oct 30 20:38:14.833264 ip-10-0-138-149 kubenswrapper[2573]: I1030 20:38:14.833196 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/7df66b3c-abe2-4d20-abda-45320cd4af1b-sys\") pod \"node-exporter-jzxqq\" (UID: \"7df66b3c-abe2-4d20-abda-45320cd4af1b\") " pod="openshift-monitoring/node-exporter-jzxqq" Oct 30 20:38:14.870151 ip-10-0-138-149 kubenswrapper[2573]: I1030 20:38:14.870129 2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" Oct 30 20:38:14.870218 ip-10-0-138-149 kubenswrapper[2573]: I1030 20:38:14.870149 2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-dmr5l\"" Oct 30 20:38:14.870218 ip-10-0-138-149 kubenswrapper[2573]: I1030 20:38:14.870153 2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" Oct 30 20:38:14.905683 ip-10-0-138-149 kubenswrapper[2573]: I1030 20:38:14.905659 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 20:38:14.933723 ip-10-0-138-149 kubenswrapper[2573]: I1030 20:38:14.933698 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/7df66b3c-abe2-4d20-abda-45320cd4af1b-metrics-client-ca\") pod \"node-exporter-jzxqq\" (UID: \"7df66b3c-abe2-4d20-abda-45320cd4af1b\") " pod="openshift-monitoring/node-exporter-jzxqq" Oct 30 20:38:14.933819 ip-10-0-138-149 kubenswrapper[2573]: I1030 20:38:14.933738 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/7df66b3c-abe2-4d20-abda-45320cd4af1b-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-jzxqq\" (UID: \"7df66b3c-abe2-4d20-abda-45320cd4af1b\") " pod="openshift-monitoring/node-exporter-jzxqq" Oct 30 20:38:14.933819 ip-10-0-138-149 kubenswrapper[2573]: I1030 20:38:14.933761 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/7df66b3c-abe2-4d20-abda-45320cd4af1b-root\") pod \"node-exporter-jzxqq\" (UID: \"7df66b3c-abe2-4d20-abda-45320cd4af1b\") " pod="openshift-monitoring/node-exporter-jzxqq" Oct 30 20:38:14.933819 ip-10-0-138-149 kubenswrapper[2573]: I1030 20:38:14.933786 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/7df66b3c-abe2-4d20-abda-45320cd4af1b-node-exporter-accelerators-collector-config\") pod \"node-exporter-jzxqq\" (UID: \"7df66b3c-abe2-4d20-abda-45320cd4af1b\") " pod="openshift-monitoring/node-exporter-jzxqq" Oct 30 20:38:14.933956 ip-10-0-138-149 kubenswrapper[2573]: I1030 20:38:14.933836 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/7df66b3c-abe2-4d20-abda-45320cd4af1b-root\") pod \"node-exporter-jzxqq\" (UID: \"7df66b3c-abe2-4d20-abda-45320cd4af1b\") " pod="openshift-monitoring/node-exporter-jzxqq" Oct 30 20:38:14.933956 ip-10-0-138-149 kubenswrapper[2573]: I1030 20:38:14.933841 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/7df66b3c-abe2-4d20-abda-45320cd4af1b-node-exporter-textfile\") pod \"node-exporter-jzxqq\" (UID: \"7df66b3c-abe2-4d20-abda-45320cd4af1b\") " pod="openshift-monitoring/node-exporter-jzxqq" Oct 30 20:38:14.933956 ip-10-0-138-149 kubenswrapper[2573]: I1030 20:38:14.933883 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/7df66b3c-abe2-4d20-abda-45320cd4af1b-node-exporter-tls\") pod \"node-exporter-jzxqq\" (UID: \"7df66b3c-abe2-4d20-abda-45320cd4af1b\") " pod="openshift-monitoring/node-exporter-jzxqq" Oct 30 20:38:14.933956 ip-10-0-138-149 kubenswrapper[2573]: I1030 20:38:14.933904 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xxc6m\" (UniqueName: \"kubernetes.io/projected/7df66b3c-abe2-4d20-abda-45320cd4af1b-kube-api-access-xxc6m\") pod \"node-exporter-jzxqq\" (UID: \"7df66b3c-abe2-4d20-abda-45320cd4af1b\") " pod="openshift-monitoring/node-exporter-jzxqq" Oct 30 20:38:14.933956 ip-10-0-138-149 kubenswrapper[2573]: I1030 20:38:14.933923 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/7df66b3c-abe2-4d20-abda-45320cd4af1b-node-exporter-wtmp\") pod \"node-exporter-jzxqq\" (UID: \"7df66b3c-abe2-4d20-abda-45320cd4af1b\") " pod="openshift-monitoring/node-exporter-jzxqq" Oct 30 20:38:14.933956 ip-10-0-138-149 kubenswrapper[2573]: I1030 20:38:14.933947 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/7df66b3c-abe2-4d20-abda-45320cd4af1b-sys\") pod \"node-exporter-jzxqq\" (UID: \"7df66b3c-abe2-4d20-abda-45320cd4af1b\") " pod="openshift-monitoring/node-exporter-jzxqq" Oct 30 20:38:14.934232 ip-10-0-138-149 kubenswrapper[2573]: I1030 20:38:14.934030 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/7df66b3c-abe2-4d20-abda-45320cd4af1b-sys\") pod \"node-exporter-jzxqq\" (UID: \"7df66b3c-abe2-4d20-abda-45320cd4af1b\") " pod="openshift-monitoring/node-exporter-jzxqq" Oct 30 20:38:14.934232 ip-10-0-138-149 kubenswrapper[2573]: I1030 20:38:14.934133 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/7df66b3c-abe2-4d20-abda-45320cd4af1b-node-exporter-wtmp\") pod \"node-exporter-jzxqq\" (UID: \"7df66b3c-abe2-4d20-abda-45320cd4af1b\") " pod="openshift-monitoring/node-exporter-jzxqq" Oct 30 20:38:14.934339 ip-10-0-138-149 kubenswrapper[2573]: I1030 20:38:14.934249 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/7df66b3c-abe2-4d20-abda-45320cd4af1b-node-exporter-textfile\") pod \"node-exporter-jzxqq\" (UID: \"7df66b3c-abe2-4d20-abda-45320cd4af1b\") " pod="openshift-monitoring/node-exporter-jzxqq" Oct 30 20:38:14.934395 ip-10-0-138-149 kubenswrapper[2573]: I1030 20:38:14.934381 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/7df66b3c-abe2-4d20-abda-45320cd4af1b-metrics-client-ca\") pod \"node-exporter-jzxqq\" (UID: \"7df66b3c-abe2-4d20-abda-45320cd4af1b\") " pod="openshift-monitoring/node-exporter-jzxqq" Oct 30 20:38:14.934511 ip-10-0-138-149 kubenswrapper[2573]: I1030 20:38:14.934491 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/7df66b3c-abe2-4d20-abda-45320cd4af1b-node-exporter-accelerators-collector-config\") pod \"node-exporter-jzxqq\" (UID: \"7df66b3c-abe2-4d20-abda-45320cd4af1b\") " pod="openshift-monitoring/node-exporter-jzxqq" Oct 30 20:38:14.936182 ip-10-0-138-149 kubenswrapper[2573]: I1030 20:38:14.936165 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/7df66b3c-abe2-4d20-abda-45320cd4af1b-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-jzxqq\" (UID: \"7df66b3c-abe2-4d20-abda-45320cd4af1b\") " pod="openshift-monitoring/node-exporter-jzxqq" Oct 30 20:38:14.936255 ip-10-0-138-149 kubenswrapper[2573]: I1030 20:38:14.936247 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/7df66b3c-abe2-4d20-abda-45320cd4af1b-node-exporter-tls\") pod \"node-exporter-jzxqq\" (UID: \"7df66b3c-abe2-4d20-abda-45320cd4af1b\") " pod="openshift-monitoring/node-exporter-jzxqq" Oct 30 20:38:14.950703 ip-10-0-138-149 kubenswrapper[2573]: I1030 20:38:14.950682 2573 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-7x9jg" Oct 30 20:38:14.990738 ip-10-0-138-149 kubenswrapper[2573]: I1030 20:38:14.990715 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xxc6m\" (UniqueName: \"kubernetes.io/projected/7df66b3c-abe2-4d20-abda-45320cd4af1b-kube-api-access-xxc6m\") pod \"node-exporter-jzxqq\" (UID: \"7df66b3c-abe2-4d20-abda-45320cd4af1b\") " pod="openshift-monitoring/node-exporter-jzxqq" Oct 30 20:38:15.031788 ip-10-0-138-149 kubenswrapper[2573]: I1030 20:38:15.031763 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 20:38:15.078227 ip-10-0-138-149 kubenswrapper[2573]: I1030 20:38:15.078203 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-jzxqq" Oct 30 20:38:15.087887 ip-10-0-138-149 kubenswrapper[2573]: W1030 20:38:15.087861 2573 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7df66b3c_abe2_4d20_abda_45320cd4af1b.slice/crio-102ed47976de176fc09ed97a56fb62e603a1d5d576402ff178d3b2377f2145e4 WatchSource:0}: Error finding container 102ed47976de176fc09ed97a56fb62e603a1d5d576402ff178d3b2377f2145e4: Status 404 returned error can't find the container with id 102ed47976de176fc09ed97a56fb62e603a1d5d576402ff178d3b2377f2145e4 Oct 30 20:38:15.233080 ip-10-0-138-149 kubenswrapper[2573]: I1030 20:38:15.233054 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 20:38:15.365431 ip-10-0-138-149 kubenswrapper[2573]: I1030 20:38:15.365387 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 20:38:16.027614 ip-10-0-138-149 kubenswrapper[2573]: I1030 20:38:16.027559 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-jzxqq" event={"ID":"7df66b3c-abe2-4d20-abda-45320cd4af1b","Type":"ContainerStarted","Data":"102ed47976de176fc09ed97a56fb62e603a1d5d576402ff178d3b2377f2145e4"} Oct 30 20:38:16.041678 ip-10-0-138-149 kubenswrapper[2573]: I1030 20:38:16.041641 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 20:38:16.252316 ip-10-0-138-149 kubenswrapper[2573]: I1030 20:38:16.252275 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 20:38:16.910725 ip-10-0-138-149 kubenswrapper[2573]: I1030 20:38:16.910695 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 20:38:17.032885 ip-10-0-138-149 kubenswrapper[2573]: I1030 20:38:17.032852 2573 generic.go:356] "Generic (PLEG): container finished" podID="7df66b3c-abe2-4d20-abda-45320cd4af1b" containerID="47daf950889e16ba586bd60eca28cf7c6fd6f0789681ccd3d78f919841c63bae" exitCode=0 Oct 30 20:38:17.033252 ip-10-0-138-149 kubenswrapper[2573]: I1030 20:38:17.032922 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-jzxqq" event={"ID":"7df66b3c-abe2-4d20-abda-45320cd4af1b","Type":"ContainerDied","Data":"47daf950889e16ba586bd60eca28cf7c6fd6f0789681ccd3d78f919841c63bae"} Oct 30 20:38:17.281254 ip-10-0-138-149 kubenswrapper[2573]: I1030 20:38:17.281173 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 20:38:17.352446 ip-10-0-138-149 kubenswrapper[2573]: I1030 20:38:17.352413 2573 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/thanos-querier-7d8d77f945-zfsgx"] Oct 30 20:38:17.354505 ip-10-0-138-149 kubenswrapper[2573]: I1030 20:38:17.354490 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-7d8d77f945-zfsgx" Oct 30 20:38:17.411043 ip-10-0-138-149 kubenswrapper[2573]: I1030 20:38:17.411011 2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-metrics\"" Oct 30 20:38:17.411168 ip-10-0-138-149 kubenswrapper[2573]: I1030 20:38:17.411141 2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-grpc-tls-dvvj6jlp1e008\"" Oct 30 20:38:17.411223 ip-10-0-138-149 kubenswrapper[2573]: I1030 20:38:17.411176 2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-rules\"" Oct 30 20:38:17.411223 ip-10-0-138-149 kubenswrapper[2573]: I1030 20:38:17.411192 2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-tls\"" Oct 30 20:38:17.411321 ip-10-0-138-149 kubenswrapper[2573]: I1030 20:38:17.411243 2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-web\"" Oct 30 20:38:17.411321 ip-10-0-138-149 kubenswrapper[2573]: I1030 20:38:17.411285 2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy\"" Oct 30 20:38:17.411321 ip-10-0-138-149 kubenswrapper[2573]: I1030 20:38:17.411308 2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-dockercfg-ngvh2\"" Oct 30 20:38:17.426214 ip-10-0-138-149 kubenswrapper[2573]: I1030 20:38:17.426187 2573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-7d8d77f945-zfsgx"] Oct 30 20:38:17.453206 ip-10-0-138-149 kubenswrapper[2573]: I1030 20:38:17.453178 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/3bcf59e7-befb-4623-9a0f-f01b70963b3e-secret-thanos-querier-tls\") pod \"thanos-querier-7d8d77f945-zfsgx\" (UID: \"3bcf59e7-befb-4623-9a0f-f01b70963b3e\") " pod="openshift-monitoring/thanos-querier-7d8d77f945-zfsgx" Oct 30 20:38:17.453317 ip-10-0-138-149 kubenswrapper[2573]: I1030 20:38:17.453216 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/3bcf59e7-befb-4623-9a0f-f01b70963b3e-secret-grpc-tls\") pod \"thanos-querier-7d8d77f945-zfsgx\" (UID: \"3bcf59e7-befb-4623-9a0f-f01b70963b3e\") " pod="openshift-monitoring/thanos-querier-7d8d77f945-zfsgx" Oct 30 20:38:17.453317 ip-10-0-138-149 kubenswrapper[2573]: I1030 20:38:17.453253 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r9qpr\" (UniqueName: \"kubernetes.io/projected/3bcf59e7-befb-4623-9a0f-f01b70963b3e-kube-api-access-r9qpr\") pod \"thanos-querier-7d8d77f945-zfsgx\" (UID: \"3bcf59e7-befb-4623-9a0f-f01b70963b3e\") " pod="openshift-monitoring/thanos-querier-7d8d77f945-zfsgx" Oct 30 20:38:17.453389 ip-10-0-138-149 kubenswrapper[2573]: I1030 20:38:17.453329 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/3bcf59e7-befb-4623-9a0f-f01b70963b3e-tmp\") pod \"thanos-querier-7d8d77f945-zfsgx\" (UID: \"3bcf59e7-befb-4623-9a0f-f01b70963b3e\") " pod="openshift-monitoring/thanos-querier-7d8d77f945-zfsgx" Oct 30 20:38:17.453389 ip-10-0-138-149 kubenswrapper[2573]: I1030 20:38:17.453378 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/3bcf59e7-befb-4623-9a0f-f01b70963b3e-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-7d8d77f945-zfsgx\" (UID: \"3bcf59e7-befb-4623-9a0f-f01b70963b3e\") " pod="openshift-monitoring/thanos-querier-7d8d77f945-zfsgx" Oct 30 20:38:17.453466 ip-10-0-138-149 kubenswrapper[2573]: I1030 20:38:17.453395 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/3bcf59e7-befb-4623-9a0f-f01b70963b3e-metrics-client-ca\") pod \"thanos-querier-7d8d77f945-zfsgx\" (UID: \"3bcf59e7-befb-4623-9a0f-f01b70963b3e\") " pod="openshift-monitoring/thanos-querier-7d8d77f945-zfsgx" Oct 30 20:38:17.453495 ip-10-0-138-149 kubenswrapper[2573]: I1030 20:38:17.453477 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/3bcf59e7-befb-4623-9a0f-f01b70963b3e-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-7d8d77f945-zfsgx\" (UID: \"3bcf59e7-befb-4623-9a0f-f01b70963b3e\") " pod="openshift-monitoring/thanos-querier-7d8d77f945-zfsgx" Oct 30 20:38:17.453551 ip-10-0-138-149 kubenswrapper[2573]: I1030 20:38:17.453532 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/3bcf59e7-befb-4623-9a0f-f01b70963b3e-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-7d8d77f945-zfsgx\" (UID: \"3bcf59e7-befb-4623-9a0f-f01b70963b3e\") " pod="openshift-monitoring/thanos-querier-7d8d77f945-zfsgx" Oct 30 20:38:17.453610 ip-10-0-138-149 kubenswrapper[2573]: I1030 20:38:17.453594 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/3bcf59e7-befb-4623-9a0f-f01b70963b3e-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-7d8d77f945-zfsgx\" (UID: \"3bcf59e7-befb-4623-9a0f-f01b70963b3e\") " pod="openshift-monitoring/thanos-querier-7d8d77f945-zfsgx" Oct 30 20:38:17.554408 ip-10-0-138-149 kubenswrapper[2573]: I1030 20:38:17.554333 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/3bcf59e7-befb-4623-9a0f-f01b70963b3e-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-7d8d77f945-zfsgx\" (UID: \"3bcf59e7-befb-4623-9a0f-f01b70963b3e\") " pod="openshift-monitoring/thanos-querier-7d8d77f945-zfsgx" Oct 30 20:38:17.554408 ip-10-0-138-149 kubenswrapper[2573]: I1030 20:38:17.554366 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/3bcf59e7-befb-4623-9a0f-f01b70963b3e-metrics-client-ca\") pod \"thanos-querier-7d8d77f945-zfsgx\" (UID: \"3bcf59e7-befb-4623-9a0f-f01b70963b3e\") " pod="openshift-monitoring/thanos-querier-7d8d77f945-zfsgx" Oct 30 20:38:17.554408 ip-10-0-138-149 kubenswrapper[2573]: I1030 20:38:17.554403 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/3bcf59e7-befb-4623-9a0f-f01b70963b3e-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-7d8d77f945-zfsgx\" (UID: \"3bcf59e7-befb-4623-9a0f-f01b70963b3e\") " pod="openshift-monitoring/thanos-querier-7d8d77f945-zfsgx" Oct 30 20:38:17.554666 ip-10-0-138-149 kubenswrapper[2573]: I1030 20:38:17.554438 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/3bcf59e7-befb-4623-9a0f-f01b70963b3e-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-7d8d77f945-zfsgx\" (UID: \"3bcf59e7-befb-4623-9a0f-f01b70963b3e\") " pod="openshift-monitoring/thanos-querier-7d8d77f945-zfsgx" Oct 30 20:38:17.554666 ip-10-0-138-149 kubenswrapper[2573]: I1030 20:38:17.554477 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/3bcf59e7-befb-4623-9a0f-f01b70963b3e-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-7d8d77f945-zfsgx\" (UID: \"3bcf59e7-befb-4623-9a0f-f01b70963b3e\") " pod="openshift-monitoring/thanos-querier-7d8d77f945-zfsgx" Oct 30 20:38:17.554666 ip-10-0-138-149 kubenswrapper[2573]: I1030 20:38:17.554617 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/3bcf59e7-befb-4623-9a0f-f01b70963b3e-secret-thanos-querier-tls\") pod \"thanos-querier-7d8d77f945-zfsgx\" (UID: \"3bcf59e7-befb-4623-9a0f-f01b70963b3e\") " pod="openshift-monitoring/thanos-querier-7d8d77f945-zfsgx" Oct 30 20:38:17.554816 ip-10-0-138-149 kubenswrapper[2573]: I1030 20:38:17.554670 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/3bcf59e7-befb-4623-9a0f-f01b70963b3e-secret-grpc-tls\") pod \"thanos-querier-7d8d77f945-zfsgx\" (UID: \"3bcf59e7-befb-4623-9a0f-f01b70963b3e\") " pod="openshift-monitoring/thanos-querier-7d8d77f945-zfsgx" Oct 30 20:38:17.554816 ip-10-0-138-149 kubenswrapper[2573]: I1030 20:38:17.554712 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r9qpr\" (UniqueName: \"kubernetes.io/projected/3bcf59e7-befb-4623-9a0f-f01b70963b3e-kube-api-access-r9qpr\") pod \"thanos-querier-7d8d77f945-zfsgx\" (UID: \"3bcf59e7-befb-4623-9a0f-f01b70963b3e\") " pod="openshift-monitoring/thanos-querier-7d8d77f945-zfsgx" Oct 30 20:38:17.554816 ip-10-0-138-149 kubenswrapper[2573]: I1030 20:38:17.554742 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/3bcf59e7-befb-4623-9a0f-f01b70963b3e-tmp\") pod \"thanos-querier-7d8d77f945-zfsgx\" (UID: \"3bcf59e7-befb-4623-9a0f-f01b70963b3e\") " pod="openshift-monitoring/thanos-querier-7d8d77f945-zfsgx" Oct 30 20:38:17.555395 ip-10-0-138-149 kubenswrapper[2573]: I1030 20:38:17.555073 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/3bcf59e7-befb-4623-9a0f-f01b70963b3e-tmp\") pod \"thanos-querier-7d8d77f945-zfsgx\" (UID: \"3bcf59e7-befb-4623-9a0f-f01b70963b3e\") " pod="openshift-monitoring/thanos-querier-7d8d77f945-zfsgx" Oct 30 20:38:17.555395 ip-10-0-138-149 kubenswrapper[2573]: I1030 20:38:17.555177 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/3bcf59e7-befb-4623-9a0f-f01b70963b3e-metrics-client-ca\") pod \"thanos-querier-7d8d77f945-zfsgx\" (UID: \"3bcf59e7-befb-4623-9a0f-f01b70963b3e\") " pod="openshift-monitoring/thanos-querier-7d8d77f945-zfsgx" Oct 30 20:38:17.557127 ip-10-0-138-149 kubenswrapper[2573]: I1030 20:38:17.557103 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/3bcf59e7-befb-4623-9a0f-f01b70963b3e-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-7d8d77f945-zfsgx\" (UID: \"3bcf59e7-befb-4623-9a0f-f01b70963b3e\") " pod="openshift-monitoring/thanos-querier-7d8d77f945-zfsgx" Oct 30 20:38:17.557447 ip-10-0-138-149 kubenswrapper[2573]: I1030 20:38:17.557424 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/3bcf59e7-befb-4623-9a0f-f01b70963b3e-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-7d8d77f945-zfsgx\" (UID: \"3bcf59e7-befb-4623-9a0f-f01b70963b3e\") " pod="openshift-monitoring/thanos-querier-7d8d77f945-zfsgx" Oct 30 20:38:17.557531 ip-10-0-138-149 kubenswrapper[2573]: I1030 20:38:17.557425 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/3bcf59e7-befb-4623-9a0f-f01b70963b3e-secret-thanos-querier-tls\") pod \"thanos-querier-7d8d77f945-zfsgx\" (UID: \"3bcf59e7-befb-4623-9a0f-f01b70963b3e\") " pod="openshift-monitoring/thanos-querier-7d8d77f945-zfsgx" Oct 30 20:38:17.557531 ip-10-0-138-149 kubenswrapper[2573]: I1030 20:38:17.557485 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/3bcf59e7-befb-4623-9a0f-f01b70963b3e-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-7d8d77f945-zfsgx\" (UID: \"3bcf59e7-befb-4623-9a0f-f01b70963b3e\") " pod="openshift-monitoring/thanos-querier-7d8d77f945-zfsgx" Oct 30 20:38:17.557531 ip-10-0-138-149 kubenswrapper[2573]: I1030 20:38:17.557512 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/3bcf59e7-befb-4623-9a0f-f01b70963b3e-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-7d8d77f945-zfsgx\" (UID: \"3bcf59e7-befb-4623-9a0f-f01b70963b3e\") " pod="openshift-monitoring/thanos-querier-7d8d77f945-zfsgx" Oct 30 20:38:17.557699 ip-10-0-138-149 kubenswrapper[2573]: I1030 20:38:17.557636 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/3bcf59e7-befb-4623-9a0f-f01b70963b3e-secret-grpc-tls\") pod \"thanos-querier-7d8d77f945-zfsgx\" (UID: \"3bcf59e7-befb-4623-9a0f-f01b70963b3e\") " pod="openshift-monitoring/thanos-querier-7d8d77f945-zfsgx" Oct 30 20:38:17.622501 ip-10-0-138-149 kubenswrapper[2573]: I1030 20:38:17.622470 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r9qpr\" (UniqueName: \"kubernetes.io/projected/3bcf59e7-befb-4623-9a0f-f01b70963b3e-kube-api-access-r9qpr\") pod \"thanos-querier-7d8d77f945-zfsgx\" (UID: \"3bcf59e7-befb-4623-9a0f-f01b70963b3e\") " pod="openshift-monitoring/thanos-querier-7d8d77f945-zfsgx" Oct 30 20:38:17.668614 ip-10-0-138-149 kubenswrapper[2573]: I1030 20:38:17.668584 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-7d8d77f945-zfsgx" Oct 30 20:38:17.676057 ip-10-0-138-149 kubenswrapper[2573]: I1030 20:38:17.676007 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 20:38:17.760776 ip-10-0-138-149 kubenswrapper[2573]: I1030 20:38:17.760744 2573 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console/downloads-6dd789c8b7-5pslw"] Oct 30 20:38:17.763146 ip-10-0-138-149 kubenswrapper[2573]: I1030 20:38:17.763123 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-6dd789c8b7-5pslw" Oct 30 20:38:17.817288 ip-10-0-138-149 kubenswrapper[2573]: I1030 20:38:17.817228 2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\"" Oct 30 20:38:17.819589 ip-10-0-138-149 kubenswrapper[2573]: I1030 20:38:17.819554 2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\"" Oct 30 20:38:17.833762 ip-10-0-138-149 kubenswrapper[2573]: I1030 20:38:17.833734 2573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-6dd789c8b7-5pslw"] Oct 30 20:38:17.857643 ip-10-0-138-149 kubenswrapper[2573]: I1030 20:38:17.857623 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/59046ab6-2f4c-4031-ba8c-21fb322a9d07-tmp\") pod \"downloads-6dd789c8b7-5pslw\" (UID: \"59046ab6-2f4c-4031-ba8c-21fb322a9d07\") " pod="openshift-console/downloads-6dd789c8b7-5pslw" Oct 30 20:38:17.857745 ip-10-0-138-149 kubenswrapper[2573]: I1030 20:38:17.857662 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jmxp5\" (UniqueName: \"kubernetes.io/projected/59046ab6-2f4c-4031-ba8c-21fb322a9d07-kube-api-access-jmxp5\") pod \"downloads-6dd789c8b7-5pslw\" (UID: \"59046ab6-2f4c-4031-ba8c-21fb322a9d07\") " pod="openshift-console/downloads-6dd789c8b7-5pslw" Oct 30 20:38:17.894969 ip-10-0-138-149 kubenswrapper[2573]: I1030 20:38:17.894938 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 20:38:17.958946 ip-10-0-138-149 kubenswrapper[2573]: I1030 20:38:17.958921 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jmxp5\" (UniqueName: \"kubernetes.io/projected/59046ab6-2f4c-4031-ba8c-21fb322a9d07-kube-api-access-jmxp5\") pod \"downloads-6dd789c8b7-5pslw\" (UID: \"59046ab6-2f4c-4031-ba8c-21fb322a9d07\") " pod="openshift-console/downloads-6dd789c8b7-5pslw" Oct 30 20:38:17.959058 ip-10-0-138-149 kubenswrapper[2573]: I1030 20:38:17.959012 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/59046ab6-2f4c-4031-ba8c-21fb322a9d07-tmp\") pod \"downloads-6dd789c8b7-5pslw\" (UID: \"59046ab6-2f4c-4031-ba8c-21fb322a9d07\") " pod="openshift-console/downloads-6dd789c8b7-5pslw" Oct 30 20:38:17.959350 ip-10-0-138-149 kubenswrapper[2573]: I1030 20:38:17.959334 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/59046ab6-2f4c-4031-ba8c-21fb322a9d07-tmp\") pod \"downloads-6dd789c8b7-5pslw\" (UID: \"59046ab6-2f4c-4031-ba8c-21fb322a9d07\") " pod="openshift-console/downloads-6dd789c8b7-5pslw" Oct 30 20:38:18.015387 ip-10-0-138-149 kubenswrapper[2573]: I1030 20:38:18.015365 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jmxp5\" (UniqueName: \"kubernetes.io/projected/59046ab6-2f4c-4031-ba8c-21fb322a9d07-kube-api-access-jmxp5\") pod \"downloads-6dd789c8b7-5pslw\" (UID: \"59046ab6-2f4c-4031-ba8c-21fb322a9d07\") " pod="openshift-console/downloads-6dd789c8b7-5pslw" Oct 30 20:38:18.038132 ip-10-0-138-149 kubenswrapper[2573]: I1030 20:38:18.038106 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-jzxqq" event={"ID":"7df66b3c-abe2-4d20-abda-45320cd4af1b","Type":"ContainerStarted","Data":"4c3b4bbda632e6b099cd45ae5235e1c8241a265b48e92597272e314f589be2f8"} Oct 30 20:38:18.038553 ip-10-0-138-149 kubenswrapper[2573]: I1030 20:38:18.038151 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-jzxqq" event={"ID":"7df66b3c-abe2-4d20-abda-45320cd4af1b","Type":"ContainerStarted","Data":"dbb952abd9ded54c05b63db8bc9aa87b5b6ce9b1054f1aa09b0d8c8c70de44f2"} Oct 30 20:38:18.073114 ip-10-0-138-149 kubenswrapper[2573]: I1030 20:38:18.073059 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-6dd789c8b7-5pslw" Oct 30 20:38:18.091390 ip-10-0-138-149 kubenswrapper[2573]: I1030 20:38:18.091367 2573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-7d8d77f945-zfsgx"] Oct 30 20:38:18.094067 ip-10-0-138-149 kubenswrapper[2573]: W1030 20:38:18.094045 2573 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3bcf59e7_befb_4623_9a0f_f01b70963b3e.slice/crio-3ad45e738307d25ff119e50ab25a4df89fe53b2944f14cdbaadd57c8a04859a7 WatchSource:0}: Error finding container 3ad45e738307d25ff119e50ab25a4df89fe53b2944f14cdbaadd57c8a04859a7: Status 404 returned error can't find the container with id 3ad45e738307d25ff119e50ab25a4df89fe53b2944f14cdbaadd57c8a04859a7 Oct 30 20:38:18.158138 ip-10-0-138-149 kubenswrapper[2573]: I1030 20:38:18.158094 2573 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-jzxqq" podStartSLOduration=2.715886266 podStartE2EDuration="4.158079644s" totalImagesPullingTime="1.37604419s" totalInitContainerRuntime="66.149188ms" isStatefulPod=true podCreationTimestamp="2026-10-30 20:38:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:38:18.157397212 +0000 UTC m=+62.969848531" watchObservedRunningTime="2026-10-30 20:38:18.158079644 +0000 UTC m=+62.970530949" Oct 30 20:38:18.258290 ip-10-0-138-149 kubenswrapper[2573]: I1030 20:38:18.258255 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 20:38:18.413271 ip-10-0-138-149 kubenswrapper[2573]: I1030 20:38:18.413238 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 20:38:18.442486 ip-10-0-138-149 kubenswrapper[2573]: I1030 20:38:18.442458 2573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-6dd789c8b7-5pslw"] Oct 30 20:38:18.445229 ip-10-0-138-149 kubenswrapper[2573]: W1030 20:38:18.445205 2573 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod59046ab6_2f4c_4031_ba8c_21fb322a9d07.slice/crio-001f8ab63fe326590b9b7ea7f701d9a74e60cb5e83eaf3d6cace90bbf287965b WatchSource:0}: Error finding container 001f8ab63fe326590b9b7ea7f701d9a74e60cb5e83eaf3d6cace90bbf287965b: Status 404 returned error can't find the container with id 001f8ab63fe326590b9b7ea7f701d9a74e60cb5e83eaf3d6cace90bbf287965b Oct 30 20:38:19.008621 ip-10-0-138-149 kubenswrapper[2573]: I1030 20:38:19.008594 2573 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-8wwgv" Oct 30 20:38:19.043706 ip-10-0-138-149 kubenswrapper[2573]: I1030 20:38:19.043673 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-6dd789c8b7-5pslw" event={"ID":"59046ab6-2f4c-4031-ba8c-21fb322a9d07","Type":"ContainerStarted","Data":"001f8ab63fe326590b9b7ea7f701d9a74e60cb5e83eaf3d6cace90bbf287965b"} Oct 30 20:38:19.045039 ip-10-0-138-149 kubenswrapper[2573]: I1030 20:38:19.045018 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-7d8d77f945-zfsgx" event={"ID":"3bcf59e7-befb-4623-9a0f-f01b70963b3e","Type":"ContainerStarted","Data":"3ad45e738307d25ff119e50ab25a4df89fe53b2944f14cdbaadd57c8a04859a7"} Oct 30 20:38:19.228333 ip-10-0-138-149 kubenswrapper[2573]: I1030 20:38:19.228287 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 20:38:19.363822 ip-10-0-138-149 kubenswrapper[2573]: I1030 20:38:19.363787 2573 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/monitoring-plugin-77f4f76594-64rbk"] Oct 30 20:38:19.365928 ip-10-0-138-149 kubenswrapper[2573]: I1030 20:38:19.365903 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-77f4f76594-64rbk" Oct 30 20:38:19.377400 ip-10-0-138-149 kubenswrapper[2573]: I1030 20:38:19.377377 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 20:38:19.433113 ip-10-0-138-149 kubenswrapper[2573]: I1030 20:38:19.433080 2573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-77f4f76594-64rbk"] Oct 30 20:38:19.443646 ip-10-0-138-149 kubenswrapper[2573]: I1030 20:38:19.443613 2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-cert\"" Oct 30 20:38:19.443888 ip-10-0-138-149 kubenswrapper[2573]: I1030 20:38:19.443868 2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-dockercfg-dfkzz\"" Oct 30 20:38:19.469976 ip-10-0-138-149 kubenswrapper[2573]: I1030 20:38:19.469946 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wk62v\" (UniqueName: \"kubernetes.io/projected/42765893-c3e3-41ab-811f-3135d4c74b07-kube-api-access-wk62v\") pod \"monitoring-plugin-77f4f76594-64rbk\" (UID: \"42765893-c3e3-41ab-811f-3135d4c74b07\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-64rbk" Oct 30 20:38:19.470121 ip-10-0-138-149 kubenswrapper[2573]: I1030 20:38:19.470010 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/42765893-c3e3-41ab-811f-3135d4c74b07-monitoring-plugin-cert\") pod \"monitoring-plugin-77f4f76594-64rbk\" (UID: \"42765893-c3e3-41ab-811f-3135d4c74b07\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-64rbk" Oct 30 20:38:19.517244 ip-10-0-138-149 kubenswrapper[2573]: I1030 20:38:19.517215 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 20:38:19.571444 ip-10-0-138-149 kubenswrapper[2573]: I1030 20:38:19.571409 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/42765893-c3e3-41ab-811f-3135d4c74b07-monitoring-plugin-cert\") pod \"monitoring-plugin-77f4f76594-64rbk\" (UID: \"42765893-c3e3-41ab-811f-3135d4c74b07\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-64rbk" Oct 30 20:38:19.571614 ip-10-0-138-149 kubenswrapper[2573]: I1030 20:38:19.571592 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wk62v\" (UniqueName: \"kubernetes.io/projected/42765893-c3e3-41ab-811f-3135d4c74b07-kube-api-access-wk62v\") pod \"monitoring-plugin-77f4f76594-64rbk\" (UID: \"42765893-c3e3-41ab-811f-3135d4c74b07\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-64rbk" Oct 30 20:38:19.574598 ip-10-0-138-149 kubenswrapper[2573]: I1030 20:38:19.574531 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/42765893-c3e3-41ab-811f-3135d4c74b07-monitoring-plugin-cert\") pod \"monitoring-plugin-77f4f76594-64rbk\" (UID: \"42765893-c3e3-41ab-811f-3135d4c74b07\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-64rbk" Oct 30 20:38:19.629861 ip-10-0-138-149 kubenswrapper[2573]: I1030 20:38:19.629798 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wk62v\" (UniqueName: \"kubernetes.io/projected/42765893-c3e3-41ab-811f-3135d4c74b07-kube-api-access-wk62v\") pod \"monitoring-plugin-77f4f76594-64rbk\" (UID: \"42765893-c3e3-41ab-811f-3135d4c74b07\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-64rbk" Oct 30 20:38:19.649720 ip-10-0-138-149 kubenswrapper[2573]: I1030 20:38:19.649687 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 20:38:19.678703 ip-10-0-138-149 kubenswrapper[2573]: I1030 20:38:19.678670 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-77f4f76594-64rbk" Oct 30 20:38:20.060310 ip-10-0-138-149 kubenswrapper[2573]: I1030 20:38:20.060238 2573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-77f4f76594-64rbk"] Oct 30 20:38:20.068392 ip-10-0-138-149 kubenswrapper[2573]: W1030 20:38:20.063672 2573 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod42765893_c3e3_41ab_811f_3135d4c74b07.slice/crio-bc1c0047dc770cd5bcd474cbf88b4fde068c541eb9976869312899e1ea43f700 WatchSource:0}: Error finding container bc1c0047dc770cd5bcd474cbf88b4fde068c541eb9976869312899e1ea43f700: Status 404 returned error can't find the container with id bc1c0047dc770cd5bcd474cbf88b4fde068c541eb9976869312899e1ea43f700 Oct 30 20:38:20.705156 ip-10-0-138-149 kubenswrapper[2573]: I1030 20:38:20.705092 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-8wwgv_743b90bd-3330-4751-a853-80b97fee0890/dns/0.log" Oct 30 20:38:20.810792 ip-10-0-138-149 kubenswrapper[2573]: I1030 20:38:20.810714 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-8wwgv_743b90bd-3330-4751-a853-80b97fee0890/kube-rbac-proxy/0.log" Oct 30 20:38:21.055903 ip-10-0-138-149 kubenswrapper[2573]: I1030 20:38:21.055859 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-77f4f76594-64rbk" event={"ID":"42765893-c3e3-41ab-811f-3135d4c74b07","Type":"ContainerStarted","Data":"bc1c0047dc770cd5bcd474cbf88b4fde068c541eb9976869312899e1ea43f700"} Oct 30 20:38:21.058370 ip-10-0-138-149 kubenswrapper[2573]: I1030 20:38:21.058336 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-7d8d77f945-zfsgx" event={"ID":"3bcf59e7-befb-4623-9a0f-f01b70963b3e","Type":"ContainerStarted","Data":"165f6661e7cfae20f17eea344318c40a197f0e7efbf890dc92ebac609c48f933"} Oct 30 20:38:21.058503 ip-10-0-138-149 kubenswrapper[2573]: I1030 20:38:21.058410 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-7d8d77f945-zfsgx" event={"ID":"3bcf59e7-befb-4623-9a0f-f01b70963b3e","Type":"ContainerStarted","Data":"7e103abd4c9f5d22eb0731e94c3edf0a34284823d484293439bd38b8c065cf9e"} Oct 30 20:38:21.058503 ip-10-0-138-149 kubenswrapper[2573]: I1030 20:38:21.058453 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-7d8d77f945-zfsgx" event={"ID":"3bcf59e7-befb-4623-9a0f-f01b70963b3e","Type":"ContainerStarted","Data":"dff9891be9777b7e99ee6b71ec31c3b320c12096a9492253529a2715d9fe238d"} Oct 30 20:38:21.149414 ip-10-0-138-149 kubenswrapper[2573]: I1030 20:38:21.149370 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 20:38:21.274033 ip-10-0-138-149 kubenswrapper[2573]: I1030 20:38:21.273985 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 20:38:21.396832 ip-10-0-138-149 kubenswrapper[2573]: I1030 20:38:21.396802 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-nv2rk_666b6e91-4a60-45d9-9e3e-877867ded7fc/dns-node-resolver/0.log" Oct 30 20:38:21.492305 ip-10-0-138-149 kubenswrapper[2573]: I1030 20:38:21.492232 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e886897a-c6a3-414f-875f-aa112541b608-metrics-certs\") pod \"network-metrics-daemon-nhs9k\" (UID: \"e886897a-c6a3-414f-875f-aa112541b608\") " pod="openshift-multus/network-metrics-daemon-nhs9k" Oct 30 20:38:21.543578 ip-10-0-138-149 kubenswrapper[2573]: I1030 20:38:21.543534 2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Oct 30 20:38:21.555182 ip-10-0-138-149 kubenswrapper[2573]: I1030 20:38:21.555157 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e886897a-c6a3-414f-875f-aa112541b608-metrics-certs\") pod \"network-metrics-daemon-nhs9k\" (UID: \"e886897a-c6a3-414f-875f-aa112541b608\") " pod="openshift-multus/network-metrics-daemon-nhs9k" Oct 30 20:38:21.593070 ip-10-0-138-149 kubenswrapper[2573]: I1030 20:38:21.593033 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/0d6769f4-dc38-45ac-863b-c75fa672866a-original-pull-secret\") pod \"global-pull-secret-syncer-bcmwc\" (UID: \"0d6769f4-dc38-45ac-863b-c75fa672866a\") " pod="kube-system/global-pull-secret-syncer-bcmwc" Oct 30 20:38:21.593222 ip-10-0-138-149 kubenswrapper[2573]: I1030 20:38:21.593081 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bc9kz\" (UniqueName: \"kubernetes.io/projected/567a72f6-035d-4c61-acaf-f46008029168-kube-api-access-bc9kz\") pod \"network-check-target-ws655\" (UID: \"567a72f6-035d-4c61-acaf-f46008029168\") " pod="openshift-network-diagnostics/network-check-target-ws655" Oct 30 20:38:21.644366 ip-10-0-138-149 kubenswrapper[2573]: I1030 20:38:21.644335 2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Oct 30 20:38:21.644366 ip-10-0-138-149 kubenswrapper[2573]: I1030 20:38:21.644336 2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Oct 30 20:38:21.656347 ip-10-0-138-149 kubenswrapper[2573]: I1030 20:38:21.656315 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/0d6769f4-dc38-45ac-863b-c75fa672866a-original-pull-secret\") pod \"global-pull-secret-syncer-bcmwc\" (UID: \"0d6769f4-dc38-45ac-863b-c75fa672866a\") " pod="kube-system/global-pull-secret-syncer-bcmwc" Oct 30 20:38:21.704348 ip-10-0-138-149 kubenswrapper[2573]: I1030 20:38:21.704318 2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Oct 30 20:38:21.711453 ip-10-0-138-149 kubenswrapper[2573]: I1030 20:38:21.711426 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bc9kz\" (UniqueName: \"kubernetes.io/projected/567a72f6-035d-4c61-acaf-f46008029168-kube-api-access-bc9kz\") pod \"network-check-target-ws655\" (UID: \"567a72f6-035d-4c61-acaf-f46008029168\") " pod="openshift-network-diagnostics/network-check-target-ws655" Oct 30 20:38:21.719050 ip-10-0-138-149 kubenswrapper[2573]: I1030 20:38:21.719027 2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-4lpn7\"" Oct 30 20:38:21.729260 ip-10-0-138-149 kubenswrapper[2573]: I1030 20:38:21.729238 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-nhs9k" Oct 30 20:38:21.997171 ip-10-0-138-149 kubenswrapper[2573]: I1030 20:38:21.997137 2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-49f2v\"" Oct 30 20:38:21.997706 ip-10-0-138-149 kubenswrapper[2573]: I1030 20:38:21.997675 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-brbhg_6a710ade-5a54-4595-89a8-636653c02fc5/serve-healthcheck-canary/0.log" Oct 30 20:38:22.002177 ip-10-0-138-149 kubenswrapper[2573]: I1030 20:38:22.002147 2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-hjmcj\"" Oct 30 20:38:22.006741 ip-10-0-138-149 kubenswrapper[2573]: I1030 20:38:22.006527 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-bcmwc" Oct 30 20:38:22.011848 ip-10-0-138-149 kubenswrapper[2573]: I1030 20:38:22.011812 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ws655" Oct 30 20:38:22.102135 ip-10-0-138-149 kubenswrapper[2573]: I1030 20:38:22.102058 2573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-nhs9k"] Oct 30 20:38:22.110097 ip-10-0-138-149 kubenswrapper[2573]: W1030 20:38:22.110059 2573 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode886897a_c6a3_414f_875f_aa112541b608.slice/crio-5b61271db530e05551d571fbdb0978efcada0909ef41a847b1b2ee25ed94f87d WatchSource:0}: Error finding container 5b61271db530e05551d571fbdb0978efcada0909ef41a847b1b2ee25ed94f87d: Status 404 returned error can't find the container with id 5b61271db530e05551d571fbdb0978efcada0909ef41a847b1b2ee25ed94f87d Oct 30 20:38:22.396411 ip-10-0-138-149 kubenswrapper[2573]: I1030 20:38:22.396380 2573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-bcmwc"] Oct 30 20:38:22.402403 ip-10-0-138-149 kubenswrapper[2573]: W1030 20:38:22.402377 2573 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0d6769f4_dc38_45ac_863b_c75fa672866a.slice/crio-5cc6dd4df47e4cf12f0856338314d254fce6069ae6f0c51a2d22fa0cb408aad3 WatchSource:0}: Error finding container 5cc6dd4df47e4cf12f0856338314d254fce6069ae6f0c51a2d22fa0cb408aad3: Status 404 returned error can't find the container with id 5cc6dd4df47e4cf12f0856338314d254fce6069ae6f0c51a2d22fa0cb408aad3 Oct 30 20:38:22.443401 ip-10-0-138-149 kubenswrapper[2573]: I1030 20:38:22.443377 2573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-ws655"] Oct 30 20:38:22.446751 ip-10-0-138-149 kubenswrapper[2573]: W1030 20:38:22.446724 2573 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod567a72f6_035d_4c61_acaf_f46008029168.slice/crio-220896563d049256ce2118869de7b04d501299d99a0a4a523e35d142fc735a63 WatchSource:0}: Error finding container 220896563d049256ce2118869de7b04d501299d99a0a4a523e35d142fc735a63: Status 404 returned error can't find the container with id 220896563d049256ce2118869de7b04d501299d99a0a4a523e35d142fc735a63 Oct 30 20:38:22.604418 ip-10-0-138-149 kubenswrapper[2573]: I1030 20:38:22.604382 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 20:38:22.725082 ip-10-0-138-149 kubenswrapper[2573]: I1030 20:38:22.725003 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 20:38:23.077501 ip-10-0-138-149 kubenswrapper[2573]: I1030 20:38:23.077393 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-77f4f76594-64rbk" event={"ID":"42765893-c3e3-41ab-811f-3135d4c74b07","Type":"ContainerStarted","Data":"0ab9be19a8d111e4c013e834b6cdd183e69d44f545f5f0d37af369dc3c4a948b"} Oct 30 20:38:23.078457 ip-10-0-138-149 kubenswrapper[2573]: I1030 20:38:23.078435 2573 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/monitoring-plugin-77f4f76594-64rbk" Oct 30 20:38:23.088403 ip-10-0-138-149 kubenswrapper[2573]: I1030 20:38:23.088375 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-7d8d77f945-zfsgx" event={"ID":"3bcf59e7-befb-4623-9a0f-f01b70963b3e","Type":"ContainerStarted","Data":"4ded1695066e4ff41e0619a2bd6e86d5a4e78825bdc3b0e8673132b921c5aaa9"} Oct 30 20:38:23.088518 ip-10-0-138-149 kubenswrapper[2573]: I1030 20:38:23.088439 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-7d8d77f945-zfsgx" event={"ID":"3bcf59e7-befb-4623-9a0f-f01b70963b3e","Type":"ContainerStarted","Data":"30dc4e43ceff2f012686a4ebcb8b70872bc1d8900d15abf2e08eb9a974991805"} Oct 30 20:38:23.088518 ip-10-0-138-149 kubenswrapper[2573]: I1030 20:38:23.088480 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-7d8d77f945-zfsgx" event={"ID":"3bcf59e7-befb-4623-9a0f-f01b70963b3e","Type":"ContainerStarted","Data":"55e6cd3a255224b5ace4dce2670a68353e7ec2eb08d17ae06cecf6bef97730fb"} Oct 30 20:38:23.088840 ip-10-0-138-149 kubenswrapper[2573]: I1030 20:38:23.088812 2573 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/thanos-querier-7d8d77f945-zfsgx" Oct 30 20:38:23.089222 ip-10-0-138-149 kubenswrapper[2573]: I1030 20:38:23.089205 2573 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/monitoring-plugin-77f4f76594-64rbk" Oct 30 20:38:23.090096 ip-10-0-138-149 kubenswrapper[2573]: I1030 20:38:23.090046 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-ws655" event={"ID":"567a72f6-035d-4c61-acaf-f46008029168","Type":"ContainerStarted","Data":"220896563d049256ce2118869de7b04d501299d99a0a4a523e35d142fc735a63"} Oct 30 20:38:23.091273 ip-10-0-138-149 kubenswrapper[2573]: I1030 20:38:23.091249 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-nhs9k" event={"ID":"e886897a-c6a3-414f-875f-aa112541b608","Type":"ContainerStarted","Data":"5b61271db530e05551d571fbdb0978efcada0909ef41a847b1b2ee25ed94f87d"} Oct 30 20:38:23.096712 ip-10-0-138-149 kubenswrapper[2573]: I1030 20:38:23.096685 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-bcmwc" event={"ID":"0d6769f4-dc38-45ac-863b-c75fa672866a","Type":"ContainerStarted","Data":"5cc6dd4df47e4cf12f0856338314d254fce6069ae6f0c51a2d22fa0cb408aad3"} Oct 30 20:38:23.198113 ip-10-0-138-149 kubenswrapper[2573]: I1030 20:38:23.198027 2573 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/monitoring-plugin-77f4f76594-64rbk" podStartSLOduration=2.100577412 podStartE2EDuration="4.198008924s" totalImagesPullingTime="2.097431512s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:38:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:38:23.196597369 +0000 UTC m=+68.009048679" watchObservedRunningTime="2026-10-30 20:38:23.198008924 +0000 UTC m=+68.010460234" Oct 30 20:38:23.430090 ip-10-0-138-149 kubenswrapper[2573]: I1030 20:38:23.430030 2573 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/thanos-querier-7d8d77f945-zfsgx" podStartSLOduration=2.663023376 podStartE2EDuration="6.42976645s" totalImagesPullingTime="3.766743074s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:38:17 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:38:23.316942609 +0000 UTC m=+68.129393924" watchObservedRunningTime="2026-10-30 20:38:23.42976645 +0000 UTC m=+68.242217759" Oct 30 20:38:24.104898 ip-10-0-138-149 kubenswrapper[2573]: I1030 20:38:24.104862 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-nhs9k" event={"ID":"e886897a-c6a3-414f-875f-aa112541b608","Type":"ContainerStarted","Data":"6a22cd5360a33628b9e6c21b599879826067fa78e6af1bd91ddaac689440e955"} Oct 30 20:38:24.105070 ip-10-0-138-149 kubenswrapper[2573]: I1030 20:38:24.104921 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-nhs9k" event={"ID":"e886897a-c6a3-414f-875f-aa112541b608","Type":"ContainerStarted","Data":"2eda3ecef61fdf9d28cf0ac715361d5b9b196070f346518bd19a60011c95b54c"} Oct 30 20:38:24.218502 ip-10-0-138-149 kubenswrapper[2573]: I1030 20:38:24.218445 2573 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-nhs9k" podStartSLOduration=66.633246063 podStartE2EDuration="1m8.21842471s" totalImagesPullingTime="1.585178647s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:37:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:38:24.217119391 +0000 UTC m=+69.029570700" watchObservedRunningTime="2026-10-30 20:38:24.21842471 +0000 UTC m=+69.030876019" Oct 30 20:38:24.271785 ip-10-0-138-149 kubenswrapper[2573]: I1030 20:38:24.271749 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 20:38:24.407535 ip-10-0-138-149 kubenswrapper[2573]: I1030 20:38:24.407453 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 20:38:26.299894 ip-10-0-138-149 kubenswrapper[2573]: I1030 20:38:26.299859 2573 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console/console-6bfdc6f474-hj4hq"] Oct 30 20:38:26.303038 ip-10-0-138-149 kubenswrapper[2573]: I1030 20:38:26.303016 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6bfdc6f474-hj4hq" Oct 30 20:38:26.404891 ip-10-0-138-149 kubenswrapper[2573]: I1030 20:38:26.404838 2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\"" Oct 30 20:38:26.404891 ip-10-0-138-149 kubenswrapper[2573]: I1030 20:38:26.404850 2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\"" Oct 30 20:38:26.404891 ip-10-0-138-149 kubenswrapper[2573]: I1030 20:38:26.404850 2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\"" Oct 30 20:38:26.404891 ip-10-0-138-149 kubenswrapper[2573]: I1030 20:38:26.404870 2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\"" Oct 30 20:38:26.404891 ip-10-0-138-149 kubenswrapper[2573]: I1030 20:38:26.404918 2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\"" Oct 30 20:38:26.405241 ip-10-0-138-149 kubenswrapper[2573]: I1030 20:38:26.404985 2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-rd5vq\"" Oct 30 20:38:26.410965 ip-10-0-138-149 kubenswrapper[2573]: I1030 20:38:26.410939 2573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-6bfdc6f474-hj4hq"] Oct 30 20:38:26.431463 ip-10-0-138-149 kubenswrapper[2573]: I1030 20:38:26.431433 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/6646b935-fe10-415a-be4b-a62a64ea45ec-console-config\") pod \"console-6bfdc6f474-hj4hq\" (UID: \"6646b935-fe10-415a-be4b-a62a64ea45ec\") " pod="openshift-console/console-6bfdc6f474-hj4hq" Oct 30 20:38:26.431635 ip-10-0-138-149 kubenswrapper[2573]: I1030 20:38:26.431482 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/6646b935-fe10-415a-be4b-a62a64ea45ec-tmp\") pod \"console-6bfdc6f474-hj4hq\" (UID: \"6646b935-fe10-415a-be4b-a62a64ea45ec\") " pod="openshift-console/console-6bfdc6f474-hj4hq" Oct 30 20:38:26.431635 ip-10-0-138-149 kubenswrapper[2573]: I1030 20:38:26.431596 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/6646b935-fe10-415a-be4b-a62a64ea45ec-console-serving-cert\") pod \"console-6bfdc6f474-hj4hq\" (UID: \"6646b935-fe10-415a-be4b-a62a64ea45ec\") " pod="openshift-console/console-6bfdc6f474-hj4hq" Oct 30 20:38:26.431635 ip-10-0-138-149 kubenswrapper[2573]: I1030 20:38:26.431633 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fqlw7\" (UniqueName: \"kubernetes.io/projected/6646b935-fe10-415a-be4b-a62a64ea45ec-kube-api-access-fqlw7\") pod \"console-6bfdc6f474-hj4hq\" (UID: \"6646b935-fe10-415a-be4b-a62a64ea45ec\") " pod="openshift-console/console-6bfdc6f474-hj4hq" Oct 30 20:38:26.431785 ip-10-0-138-149 kubenswrapper[2573]: I1030 20:38:26.431679 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/6646b935-fe10-415a-be4b-a62a64ea45ec-console-oauth-config\") pod \"console-6bfdc6f474-hj4hq\" (UID: \"6646b935-fe10-415a-be4b-a62a64ea45ec\") " pod="openshift-console/console-6bfdc6f474-hj4hq" Oct 30 20:38:26.431785 ip-10-0-138-149 kubenswrapper[2573]: I1030 20:38:26.431707 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/6646b935-fe10-415a-be4b-a62a64ea45ec-oauth-serving-cert\") pod \"console-6bfdc6f474-hj4hq\" (UID: \"6646b935-fe10-415a-be4b-a62a64ea45ec\") " pod="openshift-console/console-6bfdc6f474-hj4hq" Oct 30 20:38:26.431785 ip-10-0-138-149 kubenswrapper[2573]: I1030 20:38:26.431733 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/6646b935-fe10-415a-be4b-a62a64ea45ec-service-ca\") pod \"console-6bfdc6f474-hj4hq\" (UID: \"6646b935-fe10-415a-be4b-a62a64ea45ec\") " pod="openshift-console/console-6bfdc6f474-hj4hq" Oct 30 20:38:26.532118 ip-10-0-138-149 kubenswrapper[2573]: I1030 20:38:26.532081 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/6646b935-fe10-415a-be4b-a62a64ea45ec-oauth-serving-cert\") pod \"console-6bfdc6f474-hj4hq\" (UID: \"6646b935-fe10-415a-be4b-a62a64ea45ec\") " pod="openshift-console/console-6bfdc6f474-hj4hq" Oct 30 20:38:26.532314 ip-10-0-138-149 kubenswrapper[2573]: I1030 20:38:26.532139 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/6646b935-fe10-415a-be4b-a62a64ea45ec-service-ca\") pod \"console-6bfdc6f474-hj4hq\" (UID: \"6646b935-fe10-415a-be4b-a62a64ea45ec\") " pod="openshift-console/console-6bfdc6f474-hj4hq" Oct 30 20:38:26.532314 ip-10-0-138-149 kubenswrapper[2573]: I1030 20:38:26.532169 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/6646b935-fe10-415a-be4b-a62a64ea45ec-console-config\") pod \"console-6bfdc6f474-hj4hq\" (UID: \"6646b935-fe10-415a-be4b-a62a64ea45ec\") " pod="openshift-console/console-6bfdc6f474-hj4hq" Oct 30 20:38:26.532314 ip-10-0-138-149 kubenswrapper[2573]: I1030 20:38:26.532199 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/6646b935-fe10-415a-be4b-a62a64ea45ec-tmp\") pod \"console-6bfdc6f474-hj4hq\" (UID: \"6646b935-fe10-415a-be4b-a62a64ea45ec\") " pod="openshift-console/console-6bfdc6f474-hj4hq" Oct 30 20:38:26.537165 ip-10-0-138-149 kubenswrapper[2573]: I1030 20:38:26.536578 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/6646b935-fe10-415a-be4b-a62a64ea45ec-console-serving-cert\") pod \"console-6bfdc6f474-hj4hq\" (UID: \"6646b935-fe10-415a-be4b-a62a64ea45ec\") " pod="openshift-console/console-6bfdc6f474-hj4hq" Oct 30 20:38:26.537165 ip-10-0-138-149 kubenswrapper[2573]: I1030 20:38:26.536630 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fqlw7\" (UniqueName: \"kubernetes.io/projected/6646b935-fe10-415a-be4b-a62a64ea45ec-kube-api-access-fqlw7\") pod \"console-6bfdc6f474-hj4hq\" (UID: \"6646b935-fe10-415a-be4b-a62a64ea45ec\") " pod="openshift-console/console-6bfdc6f474-hj4hq" Oct 30 20:38:26.537165 ip-10-0-138-149 kubenswrapper[2573]: I1030 20:38:26.536726 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/6646b935-fe10-415a-be4b-a62a64ea45ec-console-oauth-config\") pod \"console-6bfdc6f474-hj4hq\" (UID: \"6646b935-fe10-415a-be4b-a62a64ea45ec\") " pod="openshift-console/console-6bfdc6f474-hj4hq" Oct 30 20:38:26.537165 ip-10-0-138-149 kubenswrapper[2573]: I1030 20:38:26.537126 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/6646b935-fe10-415a-be4b-a62a64ea45ec-oauth-serving-cert\") pod \"console-6bfdc6f474-hj4hq\" (UID: \"6646b935-fe10-415a-be4b-a62a64ea45ec\") " pod="openshift-console/console-6bfdc6f474-hj4hq" Oct 30 20:38:26.537867 ip-10-0-138-149 kubenswrapper[2573]: I1030 20:38:26.537847 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/6646b935-fe10-415a-be4b-a62a64ea45ec-console-config\") pod \"console-6bfdc6f474-hj4hq\" (UID: \"6646b935-fe10-415a-be4b-a62a64ea45ec\") " pod="openshift-console/console-6bfdc6f474-hj4hq" Oct 30 20:38:26.538346 ip-10-0-138-149 kubenswrapper[2573]: I1030 20:38:26.538298 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/6646b935-fe10-415a-be4b-a62a64ea45ec-service-ca\") pod \"console-6bfdc6f474-hj4hq\" (UID: \"6646b935-fe10-415a-be4b-a62a64ea45ec\") " pod="openshift-console/console-6bfdc6f474-hj4hq" Oct 30 20:38:26.539878 ip-10-0-138-149 kubenswrapper[2573]: I1030 20:38:26.539851 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/6646b935-fe10-415a-be4b-a62a64ea45ec-tmp\") pod \"console-6bfdc6f474-hj4hq\" (UID: \"6646b935-fe10-415a-be4b-a62a64ea45ec\") " pod="openshift-console/console-6bfdc6f474-hj4hq" Oct 30 20:38:26.544379 ip-10-0-138-149 kubenswrapper[2573]: I1030 20:38:26.544352 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/6646b935-fe10-415a-be4b-a62a64ea45ec-console-oauth-config\") pod \"console-6bfdc6f474-hj4hq\" (UID: \"6646b935-fe10-415a-be4b-a62a64ea45ec\") " pod="openshift-console/console-6bfdc6f474-hj4hq" Oct 30 20:38:26.544521 ip-10-0-138-149 kubenswrapper[2573]: I1030 20:38:26.544489 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/6646b935-fe10-415a-be4b-a62a64ea45ec-console-serving-cert\") pod \"console-6bfdc6f474-hj4hq\" (UID: \"6646b935-fe10-415a-be4b-a62a64ea45ec\") " pod="openshift-console/console-6bfdc6f474-hj4hq" Oct 30 20:38:26.595033 ip-10-0-138-149 kubenswrapper[2573]: I1030 20:38:26.594997 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fqlw7\" (UniqueName: \"kubernetes.io/projected/6646b935-fe10-415a-be4b-a62a64ea45ec-kube-api-access-fqlw7\") pod \"console-6bfdc6f474-hj4hq\" (UID: \"6646b935-fe10-415a-be4b-a62a64ea45ec\") " pod="openshift-console/console-6bfdc6f474-hj4hq" Oct 30 20:38:26.622652 ip-10-0-138-149 kubenswrapper[2573]: I1030 20:38:26.622625 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6bfdc6f474-hj4hq" Oct 30 20:38:27.012661 ip-10-0-138-149 kubenswrapper[2573]: I1030 20:38:27.012636 2573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-6bfdc6f474-hj4hq"] Oct 30 20:38:27.024117 ip-10-0-138-149 kubenswrapper[2573]: W1030 20:38:27.024082 2573 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6646b935_fe10_415a_be4b_a62a64ea45ec.slice/crio-c1c69985245a3e90dd11045025945aa9e230f6a0cddc66aa01c9131cf8ce0d3b WatchSource:0}: Error finding container c1c69985245a3e90dd11045025945aa9e230f6a0cddc66aa01c9131cf8ce0d3b: Status 404 returned error can't find the container with id c1c69985245a3e90dd11045025945aa9e230f6a0cddc66aa01c9131cf8ce0d3b Oct 30 20:38:27.119190 ip-10-0-138-149 kubenswrapper[2573]: I1030 20:38:27.119149 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-ws655" event={"ID":"567a72f6-035d-4c61-acaf-f46008029168","Type":"ContainerStarted","Data":"f8b6e3fd7f3c936728eb660b57c48b29df320f2d1b1bf92759b599afc7c19896"} Oct 30 20:38:27.119938 ip-10-0-138-149 kubenswrapper[2573]: I1030 20:38:27.119917 2573 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-ws655" Oct 30 20:38:27.121817 ip-10-0-138-149 kubenswrapper[2573]: I1030 20:38:27.121786 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6bfdc6f474-hj4hq" event={"ID":"6646b935-fe10-415a-be4b-a62a64ea45ec","Type":"ContainerStarted","Data":"c1c69985245a3e90dd11045025945aa9e230f6a0cddc66aa01c9131cf8ce0d3b"} Oct 30 20:38:27.233117 ip-10-0-138-149 kubenswrapper[2573]: I1030 20:38:27.233012 2573 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-ws655" podStartSLOduration=66.765476267 podStartE2EDuration="1m11.232993863s" totalImagesPullingTime="4.467517596s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:37:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:38:27.232862921 +0000 UTC m=+72.045314232" watchObservedRunningTime="2026-10-30 20:38:27.232993863 +0000 UTC m=+72.045445172" Oct 30 20:38:27.927323 ip-10-0-138-149 kubenswrapper[2573]: I1030 20:38:27.927286 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 20:38:28.050945 ip-10-0-138-149 kubenswrapper[2573]: I1030 20:38:28.050914 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 20:38:28.216606 ip-10-0-138-149 kubenswrapper[2573]: I1030 20:38:28.216511 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 20:38:28.342361 ip-10-0-138-149 kubenswrapper[2573]: I1030 20:38:28.342292 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 20:38:28.987285 ip-10-0-138-149 kubenswrapper[2573]: I1030 20:38:28.987247 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 20:38:29.111848 ip-10-0-138-149 kubenswrapper[2573]: I1030 20:38:29.111822 2573 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/thanos-querier-7d8d77f945-zfsgx" Oct 30 20:38:29.127514 ip-10-0-138-149 kubenswrapper[2573]: I1030 20:38:29.127491 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 20:38:29.482555 ip-10-0-138-149 systemd[1]: Stopping Kubernetes Kubelet... Oct 30 20:38:29.495046 ip-10-0-138-149 systemd[1]: kubelet.service: Deactivated successfully. Oct 30 20:38:29.495251 ip-10-0-138-149 systemd[1]: Stopped Kubernetes Kubelet. Oct 30 20:38:29.495300 ip-10-0-138-149 systemd[1]: kubelet.service: Consumed 3.908s CPU time. Oct 30 20:38:29.515437 ip-10-0-138-149 systemd[1]: Starting Kubernetes Kubelet... Oct 30 20:38:29.619790 ip-10-0-138-149 kubenswrapper[5378]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Oct 30 20:38:29.619790 ip-10-0-138-149 kubenswrapper[5378]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Oct 30 20:38:29.619790 ip-10-0-138-149 kubenswrapper[5378]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Oct 30 20:38:29.619790 ip-10-0-138-149 kubenswrapper[5378]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Oct 30 20:38:29.624613 ip-10-0-138-149 kubenswrapper[5378]: W1030 20:38:29.622111 5378 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 20:38:29.624613 ip-10-0-138-149 kubenswrapper[5378]: W1030 20:38:29.622215 5378 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 20:38:29.624613 ip-10-0-138-149 kubenswrapper[5378]: W1030 20:38:29.622220 5378 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 20:38:29.624613 ip-10-0-138-149 kubenswrapper[5378]: W1030 20:38:29.622224 5378 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 20:38:29.624613 ip-10-0-138-149 kubenswrapper[5378]: W1030 20:38:29.622229 5378 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 20:38:29.624613 ip-10-0-138-149 kubenswrapper[5378]: W1030 20:38:29.622234 5378 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 20:38:29.624613 ip-10-0-138-149 kubenswrapper[5378]: W1030 20:38:29.622238 5378 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 20:38:29.624613 ip-10-0-138-149 kubenswrapper[5378]: W1030 20:38:29.622245 5378 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 20:38:29.624613 ip-10-0-138-149 kubenswrapper[5378]: W1030 20:38:29.622252 5378 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 20:38:29.624613 ip-10-0-138-149 kubenswrapper[5378]: W1030 20:38:29.622257 5378 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 20:38:29.624613 ip-10-0-138-149 kubenswrapper[5378]: W1030 20:38:29.622262 5378 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 20:38:29.624613 ip-10-0-138-149 kubenswrapper[5378]: W1030 20:38:29.622267 5378 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 20:38:29.624613 ip-10-0-138-149 kubenswrapper[5378]: W1030 20:38:29.622271 5378 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 20:38:29.624613 ip-10-0-138-149 kubenswrapper[5378]: W1030 20:38:29.622275 5378 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 20:38:29.624613 ip-10-0-138-149 kubenswrapper[5378]: W1030 20:38:29.622290 5378 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 20:38:29.624613 ip-10-0-138-149 kubenswrapper[5378]: W1030 20:38:29.622295 5378 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 20:38:29.624613 ip-10-0-138-149 kubenswrapper[5378]: W1030 20:38:29.622299 5378 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 20:38:29.624613 ip-10-0-138-149 kubenswrapper[5378]: W1030 20:38:29.622303 5378 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 20:38:29.624613 ip-10-0-138-149 kubenswrapper[5378]: W1030 20:38:29.622307 5378 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 20:38:29.626498 ip-10-0-138-149 kubenswrapper[5378]: W1030 20:38:29.622311 5378 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 20:38:29.626498 ip-10-0-138-149 kubenswrapper[5378]: W1030 20:38:29.622315 5378 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 20:38:29.626498 ip-10-0-138-149 kubenswrapper[5378]: W1030 20:38:29.622320 5378 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 20:38:29.626498 ip-10-0-138-149 kubenswrapper[5378]: W1030 20:38:29.622325 5378 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 20:38:29.626498 ip-10-0-138-149 kubenswrapper[5378]: W1030 20:38:29.622329 5378 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 20:38:29.626498 ip-10-0-138-149 kubenswrapper[5378]: W1030 20:38:29.622333 5378 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 20:38:29.626498 ip-10-0-138-149 kubenswrapper[5378]: W1030 20:38:29.622337 5378 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 20:38:29.626498 ip-10-0-138-149 kubenswrapper[5378]: W1030 20:38:29.622342 5378 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 20:38:29.626498 ip-10-0-138-149 kubenswrapper[5378]: W1030 20:38:29.622353 5378 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 20:38:29.626498 ip-10-0-138-149 kubenswrapper[5378]: W1030 20:38:29.622359 5378 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 20:38:29.626498 ip-10-0-138-149 kubenswrapper[5378]: W1030 20:38:29.622363 5378 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 20:38:29.626498 ip-10-0-138-149 kubenswrapper[5378]: W1030 20:38:29.622367 5378 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 20:38:29.626498 ip-10-0-138-149 kubenswrapper[5378]: W1030 20:38:29.622372 5378 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 20:38:29.626498 ip-10-0-138-149 kubenswrapper[5378]: W1030 20:38:29.622376 5378 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 20:38:29.626498 ip-10-0-138-149 kubenswrapper[5378]: W1030 20:38:29.622380 5378 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 20:38:29.626498 ip-10-0-138-149 kubenswrapper[5378]: W1030 20:38:29.622385 5378 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 20:38:29.626498 ip-10-0-138-149 kubenswrapper[5378]: W1030 20:38:29.622389 5378 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 20:38:29.626498 ip-10-0-138-149 kubenswrapper[5378]: W1030 20:38:29.622393 5378 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 20:38:29.626498 ip-10-0-138-149 kubenswrapper[5378]: W1030 20:38:29.622398 5378 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 20:38:29.627392 ip-10-0-138-149 kubenswrapper[5378]: W1030 20:38:29.622402 5378 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 20:38:29.627392 ip-10-0-138-149 kubenswrapper[5378]: W1030 20:38:29.622406 5378 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 20:38:29.627392 ip-10-0-138-149 kubenswrapper[5378]: W1030 20:38:29.622411 5378 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 20:38:29.627392 ip-10-0-138-149 kubenswrapper[5378]: W1030 20:38:29.622415 5378 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 20:38:29.627392 ip-10-0-138-149 kubenswrapper[5378]: W1030 20:38:29.622419 5378 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 20:38:29.627392 ip-10-0-138-149 kubenswrapper[5378]: W1030 20:38:29.622423 5378 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 20:38:29.627392 ip-10-0-138-149 kubenswrapper[5378]: W1030 20:38:29.622427 5378 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 20:38:29.627392 ip-10-0-138-149 kubenswrapper[5378]: W1030 20:38:29.622431 5378 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 20:38:29.627392 ip-10-0-138-149 kubenswrapper[5378]: W1030 20:38:29.622436 5378 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 20:38:29.627392 ip-10-0-138-149 kubenswrapper[5378]: W1030 20:38:29.622441 5378 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 20:38:29.627392 ip-10-0-138-149 kubenswrapper[5378]: W1030 20:38:29.622445 5378 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 20:38:29.627392 ip-10-0-138-149 kubenswrapper[5378]: W1030 20:38:29.622449 5378 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 20:38:29.627392 ip-10-0-138-149 kubenswrapper[5378]: W1030 20:38:29.622453 5378 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 20:38:29.627392 ip-10-0-138-149 kubenswrapper[5378]: W1030 20:38:29.622457 5378 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 20:38:29.627392 ip-10-0-138-149 kubenswrapper[5378]: W1030 20:38:29.622461 5378 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 20:38:29.627392 ip-10-0-138-149 kubenswrapper[5378]: W1030 20:38:29.622467 5378 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 20:38:29.627392 ip-10-0-138-149 kubenswrapper[5378]: W1030 20:38:29.622471 5378 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 20:38:29.627392 ip-10-0-138-149 kubenswrapper[5378]: W1030 20:38:29.622475 5378 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 20:38:29.627392 ip-10-0-138-149 kubenswrapper[5378]: W1030 20:38:29.622480 5378 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 20:38:29.628253 ip-10-0-138-149 kubenswrapper[5378]: W1030 20:38:29.622485 5378 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 20:38:29.628253 ip-10-0-138-149 kubenswrapper[5378]: W1030 20:38:29.622489 5378 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 20:38:29.628253 ip-10-0-138-149 kubenswrapper[5378]: W1030 20:38:29.622493 5378 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 20:38:29.628253 ip-10-0-138-149 kubenswrapper[5378]: W1030 20:38:29.622497 5378 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 20:38:29.628253 ip-10-0-138-149 kubenswrapper[5378]: W1030 20:38:29.622502 5378 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 20:38:29.628253 ip-10-0-138-149 kubenswrapper[5378]: W1030 20:38:29.622506 5378 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 20:38:29.628253 ip-10-0-138-149 kubenswrapper[5378]: W1030 20:38:29.622510 5378 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 20:38:29.628253 ip-10-0-138-149 kubenswrapper[5378]: W1030 20:38:29.622515 5378 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 20:38:29.628253 ip-10-0-138-149 kubenswrapper[5378]: W1030 20:38:29.622520 5378 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 20:38:29.628253 ip-10-0-138-149 kubenswrapper[5378]: W1030 20:38:29.622524 5378 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 20:38:29.628253 ip-10-0-138-149 kubenswrapper[5378]: W1030 20:38:29.622528 5378 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 20:38:29.628253 ip-10-0-138-149 kubenswrapper[5378]: W1030 20:38:29.622533 5378 feature_gate.go:386] unrecognized feature gate: Example Oct 30 20:38:29.628253 ip-10-0-138-149 kubenswrapper[5378]: W1030 20:38:29.622536 5378 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 20:38:29.628253 ip-10-0-138-149 kubenswrapper[5378]: W1030 20:38:29.622541 5378 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 20:38:29.628253 ip-10-0-138-149 kubenswrapper[5378]: W1030 20:38:29.622546 5378 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 20:38:29.628253 ip-10-0-138-149 kubenswrapper[5378]: W1030 20:38:29.622550 5378 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 20:38:29.628253 ip-10-0-138-149 kubenswrapper[5378]: W1030 20:38:29.622554 5378 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 20:38:29.628253 ip-10-0-138-149 kubenswrapper[5378]: W1030 20:38:29.622558 5378 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 20:38:29.628253 ip-10-0-138-149 kubenswrapper[5378]: W1030 20:38:29.622578 5378 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 20:38:29.628253 ip-10-0-138-149 kubenswrapper[5378]: W1030 20:38:29.622582 5378 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 20:38:29.629359 ip-10-0-138-149 kubenswrapper[5378]: W1030 20:38:29.622586 5378 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 20:38:29.629359 ip-10-0-138-149 kubenswrapper[5378]: W1030 20:38:29.622591 5378 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 20:38:29.629359 ip-10-0-138-149 kubenswrapper[5378]: W1030 20:38:29.622598 5378 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 20:38:29.629359 ip-10-0-138-149 kubenswrapper[5378]: W1030 20:38:29.622605 5378 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 20:38:29.629359 ip-10-0-138-149 kubenswrapper[5378]: W1030 20:38:29.622610 5378 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 20:38:29.629359 ip-10-0-138-149 kubenswrapper[5378]: W1030 20:38:29.622614 5378 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 20:38:29.629359 ip-10-0-138-149 kubenswrapper[5378]: W1030 20:38:29.622618 5378 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 20:38:29.629359 ip-10-0-138-149 kubenswrapper[5378]: W1030 20:38:29.622622 5378 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 20:38:29.629359 ip-10-0-138-149 kubenswrapper[5378]: W1030 20:38:29.622626 5378 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 20:38:29.629359 ip-10-0-138-149 kubenswrapper[5378]: W1030 20:38:29.622630 5378 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 20:38:29.629359 ip-10-0-138-149 kubenswrapper[5378]: W1030 20:38:29.622634 5378 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 20:38:29.629359 ip-10-0-138-149 kubenswrapper[5378]: W1030 20:38:29.622639 5378 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 20:38:29.629359 ip-10-0-138-149 kubenswrapper[5378]: W1030 20:38:29.622643 5378 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 20:38:29.629359 ip-10-0-138-149 kubenswrapper[5378]: W1030 20:38:29.622647 5378 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 20:38:29.629359 ip-10-0-138-149 kubenswrapper[5378]: W1030 20:38:29.622651 5378 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 20:38:29.629359 ip-10-0-138-149 kubenswrapper[5378]: W1030 20:38:29.622655 5378 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 20:38:29.629359 ip-10-0-138-149 kubenswrapper[5378]: W1030 20:38:29.622660 5378 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 20:38:29.629359 ip-10-0-138-149 kubenswrapper[5378]: W1030 20:38:29.622665 5378 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 20:38:29.629359 ip-10-0-138-149 kubenswrapper[5378]: W1030 20:38:29.622669 5378 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 20:38:29.630266 ip-10-0-138-149 kubenswrapper[5378]: W1030 20:38:29.622673 5378 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 20:38:29.630266 ip-10-0-138-149 kubenswrapper[5378]: W1030 20:38:29.622677 5378 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 20:38:29.630266 ip-10-0-138-149 kubenswrapper[5378]: W1030 20:38:29.622681 5378 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 20:38:29.630266 ip-10-0-138-149 kubenswrapper[5378]: W1030 20:38:29.622686 5378 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 20:38:29.630266 ip-10-0-138-149 kubenswrapper[5378]: W1030 20:38:29.622690 5378 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 20:38:29.630266 ip-10-0-138-149 kubenswrapper[5378]: W1030 20:38:29.622694 5378 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 20:38:29.630266 ip-10-0-138-149 kubenswrapper[5378]: W1030 20:38:29.622698 5378 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 20:38:29.630266 ip-10-0-138-149 kubenswrapper[5378]: W1030 20:38:29.622702 5378 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 20:38:29.630266 ip-10-0-138-149 kubenswrapper[5378]: W1030 20:38:29.622706 5378 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 20:38:29.630266 ip-10-0-138-149 kubenswrapper[5378]: W1030 20:38:29.622710 5378 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 20:38:29.630266 ip-10-0-138-149 kubenswrapper[5378]: W1030 20:38:29.623884 5378 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 20:38:29.630266 ip-10-0-138-149 kubenswrapper[5378]: W1030 20:38:29.623896 5378 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 20:38:29.630266 ip-10-0-138-149 kubenswrapper[5378]: W1030 20:38:29.623901 5378 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 20:38:29.630266 ip-10-0-138-149 kubenswrapper[5378]: W1030 20:38:29.623906 5378 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 20:38:29.630266 ip-10-0-138-149 kubenswrapper[5378]: W1030 20:38:29.623911 5378 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 20:38:29.630266 ip-10-0-138-149 kubenswrapper[5378]: W1030 20:38:29.623915 5378 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 20:38:29.630266 ip-10-0-138-149 kubenswrapper[5378]: W1030 20:38:29.623920 5378 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 20:38:29.630266 ip-10-0-138-149 kubenswrapper[5378]: W1030 20:38:29.623924 5378 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 20:38:29.630266 ip-10-0-138-149 kubenswrapper[5378]: W1030 20:38:29.623930 5378 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 20:38:29.630266 ip-10-0-138-149 kubenswrapper[5378]: W1030 20:38:29.623934 5378 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 20:38:29.631151 ip-10-0-138-149 kubenswrapper[5378]: W1030 20:38:29.623938 5378 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 20:38:29.631151 ip-10-0-138-149 kubenswrapper[5378]: W1030 20:38:29.623944 5378 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 20:38:29.631151 ip-10-0-138-149 kubenswrapper[5378]: W1030 20:38:29.623948 5378 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 20:38:29.631151 ip-10-0-138-149 kubenswrapper[5378]: W1030 20:38:29.623952 5378 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 20:38:29.631151 ip-10-0-138-149 kubenswrapper[5378]: W1030 20:38:29.623957 5378 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 20:38:29.631151 ip-10-0-138-149 kubenswrapper[5378]: W1030 20:38:29.623961 5378 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 20:38:29.631151 ip-10-0-138-149 kubenswrapper[5378]: W1030 20:38:29.623965 5378 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 20:38:29.631151 ip-10-0-138-149 kubenswrapper[5378]: W1030 20:38:29.623969 5378 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 20:38:29.631151 ip-10-0-138-149 kubenswrapper[5378]: W1030 20:38:29.623974 5378 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 20:38:29.631151 ip-10-0-138-149 kubenswrapper[5378]: W1030 20:38:29.623978 5378 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 20:38:29.631151 ip-10-0-138-149 kubenswrapper[5378]: W1030 20:38:29.623982 5378 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 20:38:29.631151 ip-10-0-138-149 kubenswrapper[5378]: W1030 20:38:29.623987 5378 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 20:38:29.631151 ip-10-0-138-149 kubenswrapper[5378]: W1030 20:38:29.623991 5378 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 20:38:29.631151 ip-10-0-138-149 kubenswrapper[5378]: W1030 20:38:29.623995 5378 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 20:38:29.631151 ip-10-0-138-149 kubenswrapper[5378]: W1030 20:38:29.623999 5378 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 20:38:29.631151 ip-10-0-138-149 kubenswrapper[5378]: W1030 20:38:29.624003 5378 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 20:38:29.631151 ip-10-0-138-149 kubenswrapper[5378]: W1030 20:38:29.624007 5378 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 20:38:29.631151 ip-10-0-138-149 kubenswrapper[5378]: W1030 20:38:29.624011 5378 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 20:38:29.631151 ip-10-0-138-149 kubenswrapper[5378]: W1030 20:38:29.624015 5378 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 20:38:29.631151 ip-10-0-138-149 kubenswrapper[5378]: W1030 20:38:29.624019 5378 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 20:38:29.632093 ip-10-0-138-149 kubenswrapper[5378]: W1030 20:38:29.624023 5378 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 20:38:29.632093 ip-10-0-138-149 kubenswrapper[5378]: W1030 20:38:29.624027 5378 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 20:38:29.632093 ip-10-0-138-149 kubenswrapper[5378]: W1030 20:38:29.624031 5378 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 20:38:29.632093 ip-10-0-138-149 kubenswrapper[5378]: W1030 20:38:29.624036 5378 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 20:38:29.632093 ip-10-0-138-149 kubenswrapper[5378]: W1030 20:38:29.624044 5378 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 20:38:29.632093 ip-10-0-138-149 kubenswrapper[5378]: W1030 20:38:29.624048 5378 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 20:38:29.632093 ip-10-0-138-149 kubenswrapper[5378]: W1030 20:38:29.624052 5378 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 20:38:29.632093 ip-10-0-138-149 kubenswrapper[5378]: W1030 20:38:29.624057 5378 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 20:38:29.632093 ip-10-0-138-149 kubenswrapper[5378]: W1030 20:38:29.624061 5378 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 20:38:29.632093 ip-10-0-138-149 kubenswrapper[5378]: W1030 20:38:29.624066 5378 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 20:38:29.632093 ip-10-0-138-149 kubenswrapper[5378]: W1030 20:38:29.624074 5378 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 20:38:29.632093 ip-10-0-138-149 kubenswrapper[5378]: W1030 20:38:29.624078 5378 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 20:38:29.632093 ip-10-0-138-149 kubenswrapper[5378]: W1030 20:38:29.624082 5378 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 20:38:29.632093 ip-10-0-138-149 kubenswrapper[5378]: W1030 20:38:29.624086 5378 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 20:38:29.632093 ip-10-0-138-149 kubenswrapper[5378]: W1030 20:38:29.624091 5378 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 20:38:29.632093 ip-10-0-138-149 kubenswrapper[5378]: W1030 20:38:29.624095 5378 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 20:38:29.632093 ip-10-0-138-149 kubenswrapper[5378]: W1030 20:38:29.624099 5378 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 20:38:29.632093 ip-10-0-138-149 kubenswrapper[5378]: W1030 20:38:29.624103 5378 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 20:38:29.632093 ip-10-0-138-149 kubenswrapper[5378]: W1030 20:38:29.624107 5378 feature_gate.go:386] unrecognized feature gate: Example Oct 30 20:38:29.632923 ip-10-0-138-149 kubenswrapper[5378]: W1030 20:38:29.624111 5378 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 20:38:29.632923 ip-10-0-138-149 kubenswrapper[5378]: W1030 20:38:29.624115 5378 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 20:38:29.632923 ip-10-0-138-149 kubenswrapper[5378]: W1030 20:38:29.624119 5378 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 20:38:29.632923 ip-10-0-138-149 kubenswrapper[5378]: W1030 20:38:29.624123 5378 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 20:38:29.632923 ip-10-0-138-149 kubenswrapper[5378]: W1030 20:38:29.624128 5378 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 20:38:29.632923 ip-10-0-138-149 kubenswrapper[5378]: W1030 20:38:29.624132 5378 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 20:38:29.632923 ip-10-0-138-149 kubenswrapper[5378]: W1030 20:38:29.624137 5378 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 20:38:29.632923 ip-10-0-138-149 kubenswrapper[5378]: W1030 20:38:29.624141 5378 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 20:38:29.632923 ip-10-0-138-149 kubenswrapper[5378]: W1030 20:38:29.624145 5378 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 20:38:29.632923 ip-10-0-138-149 kubenswrapper[5378]: W1030 20:38:29.624149 5378 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 20:38:29.632923 ip-10-0-138-149 kubenswrapper[5378]: W1030 20:38:29.624153 5378 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 20:38:29.632923 ip-10-0-138-149 kubenswrapper[5378]: W1030 20:38:29.624157 5378 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 20:38:29.632923 ip-10-0-138-149 kubenswrapper[5378]: W1030 20:38:29.624161 5378 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 20:38:29.632923 ip-10-0-138-149 kubenswrapper[5378]: W1030 20:38:29.624165 5378 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 20:38:29.632923 ip-10-0-138-149 kubenswrapper[5378]: W1030 20:38:29.624169 5378 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 20:38:29.632923 ip-10-0-138-149 kubenswrapper[5378]: W1030 20:38:29.624173 5378 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 20:38:29.632923 ip-10-0-138-149 kubenswrapper[5378]: W1030 20:38:29.624177 5378 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 20:38:29.632923 ip-10-0-138-149 kubenswrapper[5378]: W1030 20:38:29.624183 5378 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 20:38:29.632923 ip-10-0-138-149 kubenswrapper[5378]: W1030 20:38:29.624188 5378 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 20:38:29.633779 ip-10-0-138-149 kubenswrapper[5378]: W1030 20:38:29.624192 5378 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 20:38:29.633779 ip-10-0-138-149 kubenswrapper[5378]: W1030 20:38:29.624196 5378 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 20:38:29.633779 ip-10-0-138-149 kubenswrapper[5378]: W1030 20:38:29.624200 5378 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 20:38:29.633779 ip-10-0-138-149 kubenswrapper[5378]: W1030 20:38:29.624204 5378 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 20:38:29.633779 ip-10-0-138-149 kubenswrapper[5378]: W1030 20:38:29.624210 5378 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 20:38:29.633779 ip-10-0-138-149 kubenswrapper[5378]: W1030 20:38:29.624214 5378 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 20:38:29.633779 ip-10-0-138-149 kubenswrapper[5378]: W1030 20:38:29.624218 5378 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 20:38:29.633779 ip-10-0-138-149 kubenswrapper[5378]: W1030 20:38:29.624222 5378 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 20:38:29.633779 ip-10-0-138-149 kubenswrapper[5378]: W1030 20:38:29.624227 5378 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 20:38:29.633779 ip-10-0-138-149 kubenswrapper[5378]: W1030 20:38:29.624231 5378 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 20:38:29.633779 ip-10-0-138-149 kubenswrapper[5378]: W1030 20:38:29.624235 5378 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 20:38:29.633779 ip-10-0-138-149 kubenswrapper[5378]: W1030 20:38:29.624239 5378 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 20:38:29.633779 ip-10-0-138-149 kubenswrapper[5378]: W1030 20:38:29.624243 5378 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 20:38:29.633779 ip-10-0-138-149 kubenswrapper[5378]: W1030 20:38:29.624247 5378 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 20:38:29.633779 ip-10-0-138-149 kubenswrapper[5378]: W1030 20:38:29.624254 5378 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 20:38:29.633779 ip-10-0-138-149 kubenswrapper[5378]: W1030 20:38:29.624259 5378 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 20:38:29.633779 ip-10-0-138-149 kubenswrapper[5378]: W1030 20:38:29.624264 5378 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 20:38:29.633779 ip-10-0-138-149 kubenswrapper[5378]: W1030 20:38:29.624268 5378 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 20:38:29.633779 ip-10-0-138-149 kubenswrapper[5378]: W1030 20:38:29.624272 5378 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 20:38:29.634622 ip-10-0-138-149 kubenswrapper[5378]: W1030 20:38:29.624276 5378 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 20:38:29.634622 ip-10-0-138-149 kubenswrapper[5378]: W1030 20:38:29.624280 5378 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 20:38:29.634622 ip-10-0-138-149 kubenswrapper[5378]: W1030 20:38:29.624284 5378 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 20:38:29.634622 ip-10-0-138-149 kubenswrapper[5378]: W1030 20:38:29.624288 5378 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 20:38:29.634622 ip-10-0-138-149 kubenswrapper[5378]: W1030 20:38:29.624292 5378 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 20:38:29.634622 ip-10-0-138-149 kubenswrapper[5378]: W1030 20:38:29.624296 5378 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 20:38:29.634622 ip-10-0-138-149 kubenswrapper[5378]: W1030 20:38:29.624300 5378 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 20:38:29.634622 ip-10-0-138-149 kubenswrapper[5378]: W1030 20:38:29.624304 5378 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 20:38:29.634622 ip-10-0-138-149 kubenswrapper[5378]: W1030 20:38:29.624308 5378 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 20:38:29.634622 ip-10-0-138-149 kubenswrapper[5378]: W1030 20:38:29.624312 5378 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 20:38:29.634622 ip-10-0-138-149 kubenswrapper[5378]: W1030 20:38:29.624316 5378 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 20:38:29.634622 ip-10-0-138-149 kubenswrapper[5378]: W1030 20:38:29.624321 5378 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 20:38:29.634622 ip-10-0-138-149 kubenswrapper[5378]: W1030 20:38:29.624325 5378 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 20:38:29.634622 ip-10-0-138-149 kubenswrapper[5378]: W1030 20:38:29.624329 5378 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 20:38:29.634622 ip-10-0-138-149 kubenswrapper[5378]: W1030 20:38:29.624335 5378 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 20:38:29.634622 ip-10-0-138-149 kubenswrapper[5378]: W1030 20:38:29.624341 5378 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 20:38:29.634622 ip-10-0-138-149 kubenswrapper[5378]: W1030 20:38:29.624346 5378 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 20:38:29.634622 ip-10-0-138-149 kubenswrapper[5378]: W1030 20:38:29.624354 5378 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 20:38:29.634622 ip-10-0-138-149 kubenswrapper[5378]: W1030 20:38:29.624360 5378 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 20:38:29.634622 ip-10-0-138-149 kubenswrapper[5378]: W1030 20:38:29.624944 5378 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 20:38:29.635557 ip-10-0-138-149 kubenswrapper[5378]: W1030 20:38:29.624952 5378 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 20:38:29.635557 ip-10-0-138-149 kubenswrapper[5378]: W1030 20:38:29.624955 5378 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 20:38:29.635557 ip-10-0-138-149 kubenswrapper[5378]: W1030 20:38:29.624958 5378 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 20:38:29.635557 ip-10-0-138-149 kubenswrapper[5378]: W1030 20:38:29.624961 5378 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 20:38:29.635557 ip-10-0-138-149 kubenswrapper[5378]: W1030 20:38:29.624963 5378 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 20:38:29.635557 ip-10-0-138-149 kubenswrapper[5378]: W1030 20:38:29.624966 5378 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 20:38:29.635557 ip-10-0-138-149 kubenswrapper[5378]: W1030 20:38:29.624969 5378 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 20:38:29.635557 ip-10-0-138-149 kubenswrapper[5378]: W1030 20:38:29.624971 5378 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 20:38:29.635557 ip-10-0-138-149 kubenswrapper[5378]: W1030 20:38:29.624974 5378 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 20:38:29.635557 ip-10-0-138-149 kubenswrapper[5378]: W1030 20:38:29.624976 5378 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 20:38:29.635557 ip-10-0-138-149 kubenswrapper[5378]: W1030 20:38:29.624979 5378 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 20:38:29.635557 ip-10-0-138-149 kubenswrapper[5378]: W1030 20:38:29.624982 5378 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 20:38:29.635557 ip-10-0-138-149 kubenswrapper[5378]: W1030 20:38:29.624984 5378 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 20:38:29.635557 ip-10-0-138-149 kubenswrapper[5378]: W1030 20:38:29.624987 5378 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 20:38:29.635557 ip-10-0-138-149 kubenswrapper[5378]: W1030 20:38:29.624990 5378 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 20:38:29.635557 ip-10-0-138-149 kubenswrapper[5378]: W1030 20:38:29.624994 5378 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 20:38:29.635557 ip-10-0-138-149 kubenswrapper[5378]: W1030 20:38:29.624998 5378 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 20:38:29.635557 ip-10-0-138-149 kubenswrapper[5378]: W1030 20:38:29.625002 5378 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 20:38:29.635557 ip-10-0-138-149 kubenswrapper[5378]: W1030 20:38:29.625006 5378 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 20:38:29.636374 ip-10-0-138-149 kubenswrapper[5378]: W1030 20:38:29.625010 5378 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 20:38:29.636374 ip-10-0-138-149 kubenswrapper[5378]: W1030 20:38:29.625014 5378 feature_gate.go:386] unrecognized feature gate: Example Oct 30 20:38:29.636374 ip-10-0-138-149 kubenswrapper[5378]: W1030 20:38:29.625019 5378 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 20:38:29.636374 ip-10-0-138-149 kubenswrapper[5378]: W1030 20:38:29.625023 5378 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 20:38:29.636374 ip-10-0-138-149 kubenswrapper[5378]: W1030 20:38:29.625031 5378 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 20:38:29.636374 ip-10-0-138-149 kubenswrapper[5378]: W1030 20:38:29.625035 5378 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 20:38:29.636374 ip-10-0-138-149 kubenswrapper[5378]: W1030 20:38:29.625041 5378 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 20:38:29.636374 ip-10-0-138-149 kubenswrapper[5378]: W1030 20:38:29.625046 5378 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 20:38:29.636374 ip-10-0-138-149 kubenswrapper[5378]: W1030 20:38:29.625051 5378 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 20:38:29.636374 ip-10-0-138-149 kubenswrapper[5378]: W1030 20:38:29.625055 5378 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 20:38:29.636374 ip-10-0-138-149 kubenswrapper[5378]: W1030 20:38:29.625062 5378 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 20:38:29.636374 ip-10-0-138-149 kubenswrapper[5378]: W1030 20:38:29.625067 5378 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 20:38:29.636374 ip-10-0-138-149 kubenswrapper[5378]: W1030 20:38:29.625072 5378 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 20:38:29.636374 ip-10-0-138-149 kubenswrapper[5378]: W1030 20:38:29.625076 5378 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 20:38:29.636374 ip-10-0-138-149 kubenswrapper[5378]: W1030 20:38:29.625080 5378 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 20:38:29.636374 ip-10-0-138-149 kubenswrapper[5378]: W1030 20:38:29.625084 5378 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 20:38:29.636374 ip-10-0-138-149 kubenswrapper[5378]: W1030 20:38:29.625089 5378 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 20:38:29.636374 ip-10-0-138-149 kubenswrapper[5378]: W1030 20:38:29.625093 5378 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 20:38:29.636374 ip-10-0-138-149 kubenswrapper[5378]: W1030 20:38:29.625097 5378 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 20:38:29.636374 ip-10-0-138-149 kubenswrapper[5378]: W1030 20:38:29.625101 5378 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 20:38:29.637084 ip-10-0-138-149 kubenswrapper[5378]: W1030 20:38:29.625106 5378 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 20:38:29.637084 ip-10-0-138-149 kubenswrapper[5378]: W1030 20:38:29.625108 5378 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 20:38:29.637084 ip-10-0-138-149 kubenswrapper[5378]: W1030 20:38:29.625113 5378 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 20:38:29.637084 ip-10-0-138-149 kubenswrapper[5378]: W1030 20:38:29.625117 5378 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 20:38:29.637084 ip-10-0-138-149 kubenswrapper[5378]: W1030 20:38:29.625121 5378 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 20:38:29.637084 ip-10-0-138-149 kubenswrapper[5378]: W1030 20:38:29.625126 5378 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 20:38:29.637084 ip-10-0-138-149 kubenswrapper[5378]: W1030 20:38:29.625130 5378 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 20:38:29.637084 ip-10-0-138-149 kubenswrapper[5378]: W1030 20:38:29.625133 5378 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 20:38:29.637084 ip-10-0-138-149 kubenswrapper[5378]: W1030 20:38:29.625136 5378 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 20:38:29.637084 ip-10-0-138-149 kubenswrapper[5378]: W1030 20:38:29.625138 5378 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 20:38:29.637084 ip-10-0-138-149 kubenswrapper[5378]: W1030 20:38:29.625141 5378 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 20:38:29.637084 ip-10-0-138-149 kubenswrapper[5378]: W1030 20:38:29.625143 5378 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 20:38:29.637084 ip-10-0-138-149 kubenswrapper[5378]: W1030 20:38:29.625146 5378 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 20:38:29.637084 ip-10-0-138-149 kubenswrapper[5378]: W1030 20:38:29.625149 5378 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 20:38:29.637084 ip-10-0-138-149 kubenswrapper[5378]: W1030 20:38:29.625152 5378 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 20:38:29.637084 ip-10-0-138-149 kubenswrapper[5378]: W1030 20:38:29.625154 5378 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 20:38:29.637084 ip-10-0-138-149 kubenswrapper[5378]: W1030 20:38:29.625159 5378 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 20:38:29.637084 ip-10-0-138-149 kubenswrapper[5378]: W1030 20:38:29.625161 5378 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 20:38:29.637084 ip-10-0-138-149 kubenswrapper[5378]: W1030 20:38:29.625164 5378 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 20:38:29.637084 ip-10-0-138-149 kubenswrapper[5378]: W1030 20:38:29.625166 5378 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 20:38:29.637782 ip-10-0-138-149 kubenswrapper[5378]: W1030 20:38:29.625169 5378 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 20:38:29.637782 ip-10-0-138-149 kubenswrapper[5378]: W1030 20:38:29.625171 5378 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 20:38:29.637782 ip-10-0-138-149 kubenswrapper[5378]: W1030 20:38:29.625175 5378 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 20:38:29.637782 ip-10-0-138-149 kubenswrapper[5378]: W1030 20:38:29.625178 5378 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 20:38:29.637782 ip-10-0-138-149 kubenswrapper[5378]: W1030 20:38:29.625183 5378 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 20:38:29.637782 ip-10-0-138-149 kubenswrapper[5378]: W1030 20:38:29.625187 5378 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 20:38:29.637782 ip-10-0-138-149 kubenswrapper[5378]: W1030 20:38:29.625192 5378 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 20:38:29.637782 ip-10-0-138-149 kubenswrapper[5378]: W1030 20:38:29.625196 5378 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 20:38:29.637782 ip-10-0-138-149 kubenswrapper[5378]: W1030 20:38:29.625200 5378 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 20:38:29.637782 ip-10-0-138-149 kubenswrapper[5378]: W1030 20:38:29.625205 5378 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 20:38:29.637782 ip-10-0-138-149 kubenswrapper[5378]: W1030 20:38:29.625209 5378 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 20:38:29.637782 ip-10-0-138-149 kubenswrapper[5378]: W1030 20:38:29.625213 5378 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 20:38:29.637782 ip-10-0-138-149 kubenswrapper[5378]: W1030 20:38:29.625218 5378 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 20:38:29.637782 ip-10-0-138-149 kubenswrapper[5378]: W1030 20:38:29.625222 5378 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 20:38:29.637782 ip-10-0-138-149 kubenswrapper[5378]: W1030 20:38:29.625227 5378 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 20:38:29.637782 ip-10-0-138-149 kubenswrapper[5378]: W1030 20:38:29.625231 5378 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 20:38:29.637782 ip-10-0-138-149 kubenswrapper[5378]: W1030 20:38:29.625235 5378 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 20:38:29.637782 ip-10-0-138-149 kubenswrapper[5378]: W1030 20:38:29.625238 5378 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 20:38:29.637782 ip-10-0-138-149 kubenswrapper[5378]: W1030 20:38:29.625241 5378 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 20:38:29.638379 ip-10-0-138-149 kubenswrapper[5378]: W1030 20:38:29.625244 5378 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 20:38:29.638379 ip-10-0-138-149 kubenswrapper[5378]: W1030 20:38:29.625246 5378 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 20:38:29.638379 ip-10-0-138-149 kubenswrapper[5378]: W1030 20:38:29.625249 5378 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 20:38:29.638379 ip-10-0-138-149 kubenswrapper[5378]: W1030 20:38:29.625251 5378 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 20:38:29.638379 ip-10-0-138-149 kubenswrapper[5378]: W1030 20:38:29.625254 5378 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 20:38:29.638379 ip-10-0-138-149 kubenswrapper[5378]: W1030 20:38:29.625257 5378 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 20:38:29.638379 ip-10-0-138-149 kubenswrapper[5378]: W1030 20:38:29.625259 5378 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 20:38:29.638379 ip-10-0-138-149 kubenswrapper[5378]: W1030 20:38:29.625262 5378 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 20:38:29.638379 ip-10-0-138-149 kubenswrapper[5378]: W1030 20:38:29.625264 5378 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 20:38:29.638379 ip-10-0-138-149 kubenswrapper[5378]: W1030 20:38:29.625269 5378 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 20:38:29.638379 ip-10-0-138-149 kubenswrapper[5378]: W1030 20:38:29.625271 5378 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 20:38:29.638379 ip-10-0-138-149 kubenswrapper[5378]: W1030 20:38:29.625274 5378 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 20:38:29.638379 ip-10-0-138-149 kubenswrapper[5378]: W1030 20:38:29.625276 5378 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 20:38:29.638379 ip-10-0-138-149 kubenswrapper[5378]: W1030 20:38:29.625279 5378 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 20:38:29.638379 ip-10-0-138-149 kubenswrapper[5378]: W1030 20:38:29.625282 5378 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 20:38:29.638379 ip-10-0-138-149 kubenswrapper[5378]: W1030 20:38:29.625286 5378 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 20:38:29.638379 ip-10-0-138-149 kubenswrapper[5378]: W1030 20:38:29.625288 5378 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 20:38:29.638379 ip-10-0-138-149 kubenswrapper[5378]: W1030 20:38:29.625291 5378 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 20:38:29.638379 ip-10-0-138-149 kubenswrapper[5378]: W1030 20:38:29.625294 5378 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 20:38:29.639197 ip-10-0-138-149 kubenswrapper[5378]: W1030 20:38:29.625297 5378 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 20:38:29.639197 ip-10-0-138-149 kubenswrapper[5378]: W1030 20:38:29.625300 5378 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 20:38:29.639197 ip-10-0-138-149 kubenswrapper[5378]: W1030 20:38:29.625302 5378 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 20:38:29.639197 ip-10-0-138-149 kubenswrapper[5378]: W1030 20:38:29.625305 5378 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 20:38:29.639197 ip-10-0-138-149 kubenswrapper[5378]: W1030 20:38:29.625308 5378 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 20:38:29.639197 ip-10-0-138-149 kubenswrapper[5378]: W1030 20:38:29.625312 5378 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 20:38:29.639197 ip-10-0-138-149 kubenswrapper[5378]: W1030 20:38:29.625320 5378 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 20:38:29.639197 ip-10-0-138-149 kubenswrapper[5378]: W1030 20:38:29.625326 5378 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 20:38:29.639197 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:38:29.625440 5378 flags.go:64] FLAG: --address="0.0.0.0" Oct 30 20:38:29.639197 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:38:29.625457 5378 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Oct 30 20:38:29.639197 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:38:29.625467 5378 flags.go:64] FLAG: --anonymous-auth="true" Oct 30 20:38:29.639197 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:38:29.625472 5378 flags.go:64] FLAG: --application-metrics-count-limit="100" Oct 30 20:38:29.639197 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:38:29.625477 5378 flags.go:64] FLAG: --authentication-token-webhook="false" Oct 30 20:38:29.639197 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:38:29.625482 5378 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Oct 30 20:38:29.639197 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:38:29.625489 5378 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Oct 30 20:38:29.639197 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:38:29.625500 5378 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Oct 30 20:38:29.639197 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:38:29.625506 5378 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Oct 30 20:38:29.639197 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:38:29.625509 5378 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Oct 30 20:38:29.639197 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:38:29.625513 5378 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Oct 30 20:38:29.639197 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:38:29.625516 5378 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Oct 30 20:38:29.639197 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:38:29.625520 5378 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Oct 30 20:38:29.639897 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:38:29.625523 5378 flags.go:64] FLAG: --cgroup-root="" Oct 30 20:38:29.639897 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:38:29.625527 5378 flags.go:64] FLAG: --cgroups-per-qos="true" Oct 30 20:38:29.639897 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:38:29.625532 5378 flags.go:64] FLAG: --client-ca-file="" Oct 30 20:38:29.639897 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:38:29.625537 5378 flags.go:64] FLAG: --cloud-provider="external" Oct 30 20:38:29.639897 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:38:29.625542 5378 flags.go:64] FLAG: --cluster-dns="[]" Oct 30 20:38:29.639897 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:38:29.625547 5378 flags.go:64] FLAG: --cluster-domain="" Oct 30 20:38:29.639897 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:38:29.625550 5378 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Oct 30 20:38:29.639897 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:38:29.625555 5378 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Oct 30 20:38:29.639897 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:38:29.625559 5378 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Oct 30 20:38:29.639897 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:38:29.625574 5378 flags.go:64] FLAG: --container-log-max-files="5" Oct 30 20:38:29.639897 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:38:29.625578 5378 flags.go:64] FLAG: --container-log-max-size="10Mi" Oct 30 20:38:29.639897 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:38:29.625582 5378 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Oct 30 20:38:29.639897 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:38:29.625586 5378 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Oct 30 20:38:29.639897 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:38:29.625589 5378 flags.go:64] FLAG: --containerd-namespace="k8s.io" Oct 30 20:38:29.639897 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:38:29.625592 5378 flags.go:64] FLAG: --contention-profiling="false" Oct 30 20:38:29.639897 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:38:29.625595 5378 flags.go:64] FLAG: --cpu-cfs-quota="true" Oct 30 20:38:29.639897 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:38:29.625598 5378 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Oct 30 20:38:29.639897 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:38:29.625601 5378 flags.go:64] FLAG: --cpu-manager-policy="none" Oct 30 20:38:29.639897 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:38:29.625605 5378 flags.go:64] FLAG: --cpu-manager-policy-options="" Oct 30 20:38:29.639897 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:38:29.625611 5378 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Oct 30 20:38:29.639897 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:38:29.625617 5378 flags.go:64] FLAG: --enable-controller-attach-detach="true" Oct 30 20:38:29.639897 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:38:29.625621 5378 flags.go:64] FLAG: --enable-debugging-handlers="true" Oct 30 20:38:29.639897 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:38:29.625626 5378 flags.go:64] FLAG: --enable-load-reader="false" Oct 30 20:38:29.639897 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:38:29.625629 5378 flags.go:64] FLAG: --enable-server="true" Oct 30 20:38:29.639897 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:38:29.625632 5378 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Oct 30 20:38:29.640739 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:38:29.625641 5378 flags.go:64] FLAG: --event-burst="100" Oct 30 20:38:29.640739 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:38:29.625646 5378 flags.go:64] FLAG: --event-qps="50" Oct 30 20:38:29.640739 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:38:29.625651 5378 flags.go:64] FLAG: --event-storage-age-limit="default=0" Oct 30 20:38:29.640739 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:38:29.625656 5378 flags.go:64] FLAG: --event-storage-event-limit="default=0" Oct 30 20:38:29.640739 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:38:29.625659 5378 flags.go:64] FLAG: --eviction-hard="" Oct 30 20:38:29.640739 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:38:29.625664 5378 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Oct 30 20:38:29.640739 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:38:29.625667 5378 flags.go:64] FLAG: --eviction-minimum-reclaim="" Oct 30 20:38:29.640739 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:38:29.625671 5378 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Oct 30 20:38:29.640739 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:38:29.625676 5378 flags.go:64] FLAG: --eviction-soft="" Oct 30 20:38:29.640739 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:38:29.625679 5378 flags.go:64] FLAG: --eviction-soft-grace-period="" Oct 30 20:38:29.640739 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:38:29.625682 5378 flags.go:64] FLAG: --exit-on-lock-contention="false" Oct 30 20:38:29.640739 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:38:29.625685 5378 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Oct 30 20:38:29.640739 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:38:29.625690 5378 flags.go:64] FLAG: --experimental-mounter-path="" Oct 30 20:38:29.640739 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:38:29.625695 5378 flags.go:64] FLAG: --fail-cgroupv1="true" Oct 30 20:38:29.640739 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:38:29.625702 5378 flags.go:64] FLAG: --fail-swap-on="true" Oct 30 20:38:29.640739 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:38:29.625706 5378 flags.go:64] FLAG: --feature-gates="" Oct 30 20:38:29.640739 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:38:29.625711 5378 flags.go:64] FLAG: --file-check-frequency="20s" Oct 30 20:38:29.640739 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:38:29.625717 5378 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Oct 30 20:38:29.640739 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:38:29.625722 5378 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Oct 30 20:38:29.640739 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:38:29.625727 5378 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Oct 30 20:38:29.640739 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:38:29.625730 5378 flags.go:64] FLAG: --healthz-port="10248" Oct 30 20:38:29.640739 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:38:29.625733 5378 flags.go:64] FLAG: --help="false" Oct 30 20:38:29.640739 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:38:29.625736 5378 flags.go:64] FLAG: --hostname-override="ip-10-0-138-149.us-west-1.compute.internal" Oct 30 20:38:29.640739 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:38:29.625740 5378 flags.go:64] FLAG: --housekeeping-interval="10s" Oct 30 20:38:29.640739 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:38:29.625743 5378 flags.go:64] FLAG: --http-check-frequency="20s" Oct 30 20:38:29.641582 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:38:29.625746 5378 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Oct 30 20:38:29.641582 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:38:29.625749 5378 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Oct 30 20:38:29.641582 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:38:29.625753 5378 flags.go:64] FLAG: --image-gc-high-threshold="85" Oct 30 20:38:29.641582 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:38:29.625756 5378 flags.go:64] FLAG: --image-gc-low-threshold="80" Oct 30 20:38:29.641582 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:38:29.625758 5378 flags.go:64] FLAG: --image-service-endpoint="" Oct 30 20:38:29.641582 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:38:29.625761 5378 flags.go:64] FLAG: --kernel-memcg-notification="false" Oct 30 20:38:29.641582 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:38:29.625764 5378 flags.go:64] FLAG: --kube-api-burst="100" Oct 30 20:38:29.641582 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:38:29.625767 5378 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Oct 30 20:38:29.641582 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:38:29.625771 5378 flags.go:64] FLAG: --kube-api-qps="50" Oct 30 20:38:29.641582 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:38:29.625774 5378 flags.go:64] FLAG: --kube-reserved="" Oct 30 20:38:29.641582 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:38:29.625777 5378 flags.go:64] FLAG: --kube-reserved-cgroup="" Oct 30 20:38:29.641582 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:38:29.625780 5378 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Oct 30 20:38:29.641582 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:38:29.625783 5378 flags.go:64] FLAG: --kubelet-cgroups="" Oct 30 20:38:29.641582 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:38:29.625788 5378 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Oct 30 20:38:29.641582 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:38:29.625793 5378 flags.go:64] FLAG: --lock-file="" Oct 30 20:38:29.641582 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:38:29.625799 5378 flags.go:64] FLAG: --log-cadvisor-usage="false" Oct 30 20:38:29.641582 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:38:29.625802 5378 flags.go:64] FLAG: --log-flush-frequency="5s" Oct 30 20:38:29.641582 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:38:29.625805 5378 flags.go:64] FLAG: --log-json-info-buffer-size="0" Oct 30 20:38:29.641582 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:38:29.625811 5378 flags.go:64] FLAG: --log-json-split-stream="false" Oct 30 20:38:29.641582 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:38:29.625814 5378 flags.go:64] FLAG: --log-text-info-buffer-size="0" Oct 30 20:38:29.641582 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:38:29.625817 5378 flags.go:64] FLAG: --log-text-split-stream="false" Oct 30 20:38:29.641582 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:38:29.625822 5378 flags.go:64] FLAG: --logging-format="text" Oct 30 20:38:29.641582 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:38:29.625825 5378 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Oct 30 20:38:29.642339 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:38:29.625828 5378 flags.go:64] FLAG: --make-iptables-util-chains="true" Oct 30 20:38:29.642339 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:38:29.625831 5378 flags.go:64] FLAG: --manifest-url="" Oct 30 20:38:29.642339 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:38:29.625834 5378 flags.go:64] FLAG: --manifest-url-header="" Oct 30 20:38:29.642339 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:38:29.625839 5378 flags.go:64] FLAG: --max-housekeeping-interval="15s" Oct 30 20:38:29.642339 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:38:29.625844 5378 flags.go:64] FLAG: --max-open-files="1000000" Oct 30 20:38:29.642339 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:38:29.625850 5378 flags.go:64] FLAG: --max-pods="110" Oct 30 20:38:29.642339 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:38:29.625855 5378 flags.go:64] FLAG: --maximum-dead-containers="-1" Oct 30 20:38:29.642339 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:38:29.625860 5378 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Oct 30 20:38:29.642339 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:38:29.625866 5378 flags.go:64] FLAG: --memory-manager-policy="None" Oct 30 20:38:29.642339 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:38:29.625871 5378 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Oct 30 20:38:29.642339 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:38:29.625876 5378 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Oct 30 20:38:29.642339 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:38:29.625880 5378 flags.go:64] FLAG: --node-ip="0.0.0.0" Oct 30 20:38:29.642339 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:38:29.625883 5378 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Oct 30 20:38:29.642339 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:38:29.625891 5378 flags.go:64] FLAG: --node-status-max-images="50" Oct 30 20:38:29.642339 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:38:29.625894 5378 flags.go:64] FLAG: --node-status-update-frequency="10s" Oct 30 20:38:29.642339 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:38:29.625897 5378 flags.go:64] FLAG: --oom-score-adj="-999" Oct 30 20:38:29.642339 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:38:29.625900 5378 flags.go:64] FLAG: --pod-cidr="" Oct 30 20:38:29.642339 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:38:29.625903 5378 flags.go:64] FLAG: --pod-manifest-path="" Oct 30 20:38:29.642339 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:38:29.625906 5378 flags.go:64] FLAG: --pod-max-pids="-1" Oct 30 20:38:29.642339 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:38:29.625909 5378 flags.go:64] FLAG: --pods-per-core="0" Oct 30 20:38:29.642339 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:38:29.625912 5378 flags.go:64] FLAG: --port="10250" Oct 30 20:38:29.642339 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:38:29.625917 5378 flags.go:64] FLAG: --protect-kernel-defaults="false" Oct 30 20:38:29.642339 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:38:29.625921 5378 flags.go:64] FLAG: --provider-id="aws:///us-west-1a/i-0668a941f40a9593e" Oct 30 20:38:29.642339 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:38:29.625927 5378 flags.go:64] FLAG: --qos-reserved="" Oct 30 20:38:29.642339 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:38:29.625933 5378 flags.go:64] FLAG: --read-only-port="10255" Oct 30 20:38:29.642339 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:38:29.625936 5378 flags.go:64] FLAG: --register-node="true" Oct 30 20:38:29.643458 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:38:29.625939 5378 flags.go:64] FLAG: --register-with-taints="" Oct 30 20:38:29.643458 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:38:29.625943 5378 flags.go:64] FLAG: --registry-burst="10" Oct 30 20:38:29.643458 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:38:29.625946 5378 flags.go:64] FLAG: --registry-qps="5" Oct 30 20:38:29.643458 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:38:29.625949 5378 flags.go:64] FLAG: --reserved-cpus="" Oct 30 20:38:29.643458 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:38:29.625954 5378 flags.go:64] FLAG: --reserved-memory="" Oct 30 20:38:29.643458 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:38:29.625957 5378 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Oct 30 20:38:29.643458 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:38:29.625960 5378 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Oct 30 20:38:29.643458 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:38:29.625964 5378 flags.go:64] FLAG: --rotate-certificates="false" Oct 30 20:38:29.643458 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:38:29.625967 5378 flags.go:64] FLAG: --rotate-server-certificates="false" Oct 30 20:38:29.643458 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:38:29.625970 5378 flags.go:64] FLAG: --runonce="false" Oct 30 20:38:29.643458 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:38:29.625973 5378 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Oct 30 20:38:29.643458 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:38:29.625977 5378 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Oct 30 20:38:29.643458 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:38:29.625980 5378 flags.go:64] FLAG: --seccomp-default="false" Oct 30 20:38:29.643458 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:38:29.625983 5378 flags.go:64] FLAG: --serialize-image-pulls="true" Oct 30 20:38:29.643458 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:38:29.625986 5378 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Oct 30 20:38:29.643458 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:38:29.625989 5378 flags.go:64] FLAG: --storage-driver-db="cadvisor" Oct 30 20:38:29.643458 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:38:29.625993 5378 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Oct 30 20:38:29.643458 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:38:29.625998 5378 flags.go:64] FLAG: --storage-driver-password="root" Oct 30 20:38:29.643458 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:38:29.626002 5378 flags.go:64] FLAG: --storage-driver-secure="false" Oct 30 20:38:29.643458 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:38:29.626007 5378 flags.go:64] FLAG: --storage-driver-table="stats" Oct 30 20:38:29.643458 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:38:29.626011 5378 flags.go:64] FLAG: --storage-driver-user="root" Oct 30 20:38:29.643458 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:38:29.626015 5378 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Oct 30 20:38:29.643458 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:38:29.626021 5378 flags.go:64] FLAG: --sync-frequency="1m0s" Oct 30 20:38:29.643458 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:38:29.626027 5378 flags.go:64] FLAG: --system-cgroups="" Oct 30 20:38:29.643458 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:38:29.626032 5378 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Oct 30 20:38:29.644519 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:38:29.626039 5378 flags.go:64] FLAG: --system-reserved-cgroup="" Oct 30 20:38:29.644519 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:38:29.626043 5378 flags.go:64] FLAG: --tls-cert-file="" Oct 30 20:38:29.644519 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:38:29.626048 5378 flags.go:64] FLAG: --tls-cipher-suites="[]" Oct 30 20:38:29.644519 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:38:29.626054 5378 flags.go:64] FLAG: --tls-min-version="" Oct 30 20:38:29.644519 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:38:29.626058 5378 flags.go:64] FLAG: --tls-private-key-file="" Oct 30 20:38:29.644519 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:38:29.626062 5378 flags.go:64] FLAG: --topology-manager-policy="none" Oct 30 20:38:29.644519 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:38:29.626066 5378 flags.go:64] FLAG: --topology-manager-policy-options="" Oct 30 20:38:29.644519 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:38:29.626069 5378 flags.go:64] FLAG: --topology-manager-scope="container" Oct 30 20:38:29.644519 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:38:29.626073 5378 flags.go:64] FLAG: --v="2" Oct 30 20:38:29.644519 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:38:29.626079 5378 flags.go:64] FLAG: --version="false" Oct 30 20:38:29.644519 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:38:29.626086 5378 flags.go:64] FLAG: --vmodule="" Oct 30 20:38:29.644519 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:38:29.626094 5378 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Oct 30 20:38:29.644519 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:38:29.626098 5378 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Oct 30 20:38:29.644519 ip-10-0-138-149 kubenswrapper[5378]: W1030 20:38:29.626256 5378 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 20:38:29.644519 ip-10-0-138-149 kubenswrapper[5378]: W1030 20:38:29.626261 5378 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 20:38:29.644519 ip-10-0-138-149 kubenswrapper[5378]: W1030 20:38:29.626264 5378 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 20:38:29.644519 ip-10-0-138-149 kubenswrapper[5378]: W1030 20:38:29.626267 5378 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 20:38:29.644519 ip-10-0-138-149 kubenswrapper[5378]: W1030 20:38:29.626270 5378 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 20:38:29.644519 ip-10-0-138-149 kubenswrapper[5378]: W1030 20:38:29.626273 5378 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 20:38:29.644519 ip-10-0-138-149 kubenswrapper[5378]: W1030 20:38:29.626277 5378 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 20:38:29.644519 ip-10-0-138-149 kubenswrapper[5378]: W1030 20:38:29.626280 5378 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 20:38:29.644519 ip-10-0-138-149 kubenswrapper[5378]: W1030 20:38:29.626283 5378 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 20:38:29.644519 ip-10-0-138-149 kubenswrapper[5378]: W1030 20:38:29.626286 5378 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 20:38:29.645348 ip-10-0-138-149 kubenswrapper[5378]: W1030 20:38:29.626288 5378 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 20:38:29.645348 ip-10-0-138-149 kubenswrapper[5378]: W1030 20:38:29.626291 5378 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 20:38:29.645348 ip-10-0-138-149 kubenswrapper[5378]: W1030 20:38:29.626294 5378 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 20:38:29.645348 ip-10-0-138-149 kubenswrapper[5378]: W1030 20:38:29.626297 5378 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 20:38:29.645348 ip-10-0-138-149 kubenswrapper[5378]: W1030 20:38:29.626301 5378 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 20:38:29.645348 ip-10-0-138-149 kubenswrapper[5378]: W1030 20:38:29.626306 5378 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 20:38:29.645348 ip-10-0-138-149 kubenswrapper[5378]: W1030 20:38:29.626311 5378 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 20:38:29.645348 ip-10-0-138-149 kubenswrapper[5378]: W1030 20:38:29.626315 5378 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 20:38:29.645348 ip-10-0-138-149 kubenswrapper[5378]: W1030 20:38:29.626320 5378 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 20:38:29.645348 ip-10-0-138-149 kubenswrapper[5378]: W1030 20:38:29.626325 5378 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 20:38:29.645348 ip-10-0-138-149 kubenswrapper[5378]: W1030 20:38:29.626329 5378 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 20:38:29.645348 ip-10-0-138-149 kubenswrapper[5378]: W1030 20:38:29.626332 5378 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 20:38:29.645348 ip-10-0-138-149 kubenswrapper[5378]: W1030 20:38:29.626335 5378 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 20:38:29.645348 ip-10-0-138-149 kubenswrapper[5378]: W1030 20:38:29.626337 5378 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 20:38:29.645348 ip-10-0-138-149 kubenswrapper[5378]: W1030 20:38:29.626342 5378 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 20:38:29.645348 ip-10-0-138-149 kubenswrapper[5378]: W1030 20:38:29.626345 5378 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 20:38:29.645348 ip-10-0-138-149 kubenswrapper[5378]: W1030 20:38:29.626354 5378 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 20:38:29.645348 ip-10-0-138-149 kubenswrapper[5378]: W1030 20:38:29.626357 5378 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 20:38:29.645348 ip-10-0-138-149 kubenswrapper[5378]: W1030 20:38:29.626360 5378 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 20:38:29.645948 ip-10-0-138-149 kubenswrapper[5378]: W1030 20:38:29.626364 5378 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 20:38:29.645948 ip-10-0-138-149 kubenswrapper[5378]: W1030 20:38:29.626370 5378 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 20:38:29.645948 ip-10-0-138-149 kubenswrapper[5378]: W1030 20:38:29.626374 5378 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 20:38:29.645948 ip-10-0-138-149 kubenswrapper[5378]: W1030 20:38:29.626379 5378 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 20:38:29.645948 ip-10-0-138-149 kubenswrapper[5378]: W1030 20:38:29.626382 5378 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 20:38:29.645948 ip-10-0-138-149 kubenswrapper[5378]: W1030 20:38:29.626385 5378 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 20:38:29.645948 ip-10-0-138-149 kubenswrapper[5378]: W1030 20:38:29.626388 5378 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 20:38:29.645948 ip-10-0-138-149 kubenswrapper[5378]: W1030 20:38:29.626390 5378 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 20:38:29.645948 ip-10-0-138-149 kubenswrapper[5378]: W1030 20:38:29.626393 5378 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 20:38:29.645948 ip-10-0-138-149 kubenswrapper[5378]: W1030 20:38:29.626396 5378 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 20:38:29.645948 ip-10-0-138-149 kubenswrapper[5378]: W1030 20:38:29.626399 5378 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 20:38:29.645948 ip-10-0-138-149 kubenswrapper[5378]: W1030 20:38:29.626402 5378 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 20:38:29.645948 ip-10-0-138-149 kubenswrapper[5378]: W1030 20:38:29.626405 5378 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 20:38:29.645948 ip-10-0-138-149 kubenswrapper[5378]: W1030 20:38:29.626407 5378 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 20:38:29.645948 ip-10-0-138-149 kubenswrapper[5378]: W1030 20:38:29.626410 5378 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 20:38:29.645948 ip-10-0-138-149 kubenswrapper[5378]: W1030 20:38:29.626412 5378 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 20:38:29.645948 ip-10-0-138-149 kubenswrapper[5378]: W1030 20:38:29.626415 5378 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 20:38:29.645948 ip-10-0-138-149 kubenswrapper[5378]: W1030 20:38:29.626418 5378 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 20:38:29.645948 ip-10-0-138-149 kubenswrapper[5378]: W1030 20:38:29.626420 5378 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 20:38:29.646512 ip-10-0-138-149 kubenswrapper[5378]: W1030 20:38:29.626423 5378 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 20:38:29.646512 ip-10-0-138-149 kubenswrapper[5378]: W1030 20:38:29.626426 5378 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 20:38:29.646512 ip-10-0-138-149 kubenswrapper[5378]: W1030 20:38:29.626428 5378 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 20:38:29.646512 ip-10-0-138-149 kubenswrapper[5378]: W1030 20:38:29.626432 5378 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 20:38:29.646512 ip-10-0-138-149 kubenswrapper[5378]: W1030 20:38:29.626435 5378 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 20:38:29.646512 ip-10-0-138-149 kubenswrapper[5378]: W1030 20:38:29.626438 5378 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 20:38:29.646512 ip-10-0-138-149 kubenswrapper[5378]: W1030 20:38:29.626442 5378 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 20:38:29.646512 ip-10-0-138-149 kubenswrapper[5378]: W1030 20:38:29.626446 5378 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 20:38:29.646512 ip-10-0-138-149 kubenswrapper[5378]: W1030 20:38:29.626452 5378 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 20:38:29.646512 ip-10-0-138-149 kubenswrapper[5378]: W1030 20:38:29.626457 5378 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 20:38:29.646512 ip-10-0-138-149 kubenswrapper[5378]: W1030 20:38:29.626459 5378 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 20:38:29.646512 ip-10-0-138-149 kubenswrapper[5378]: W1030 20:38:29.626462 5378 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 20:38:29.646512 ip-10-0-138-149 kubenswrapper[5378]: W1030 20:38:29.626464 5378 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 20:38:29.646512 ip-10-0-138-149 kubenswrapper[5378]: W1030 20:38:29.626467 5378 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 20:38:29.646512 ip-10-0-138-149 kubenswrapper[5378]: W1030 20:38:29.626471 5378 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 20:38:29.646512 ip-10-0-138-149 kubenswrapper[5378]: W1030 20:38:29.626474 5378 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 20:38:29.646512 ip-10-0-138-149 kubenswrapper[5378]: W1030 20:38:29.626476 5378 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 20:38:29.646512 ip-10-0-138-149 kubenswrapper[5378]: W1030 20:38:29.626479 5378 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 20:38:29.646512 ip-10-0-138-149 kubenswrapper[5378]: W1030 20:38:29.626482 5378 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 20:38:29.647085 ip-10-0-138-149 kubenswrapper[5378]: W1030 20:38:29.626486 5378 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 20:38:29.647085 ip-10-0-138-149 kubenswrapper[5378]: W1030 20:38:29.626491 5378 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 20:38:29.647085 ip-10-0-138-149 kubenswrapper[5378]: W1030 20:38:29.626496 5378 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 20:38:29.647085 ip-10-0-138-149 kubenswrapper[5378]: W1030 20:38:29.626502 5378 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 20:38:29.647085 ip-10-0-138-149 kubenswrapper[5378]: W1030 20:38:29.626507 5378 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 20:38:29.647085 ip-10-0-138-149 kubenswrapper[5378]: W1030 20:38:29.626513 5378 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 20:38:29.647085 ip-10-0-138-149 kubenswrapper[5378]: W1030 20:38:29.626517 5378 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 20:38:29.647085 ip-10-0-138-149 kubenswrapper[5378]: W1030 20:38:29.626521 5378 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 20:38:29.647085 ip-10-0-138-149 kubenswrapper[5378]: W1030 20:38:29.626525 5378 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 20:38:29.647085 ip-10-0-138-149 kubenswrapper[5378]: W1030 20:38:29.626530 5378 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 20:38:29.647085 ip-10-0-138-149 kubenswrapper[5378]: W1030 20:38:29.626534 5378 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 20:38:29.647085 ip-10-0-138-149 kubenswrapper[5378]: W1030 20:38:29.626538 5378 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 20:38:29.647085 ip-10-0-138-149 kubenswrapper[5378]: W1030 20:38:29.626542 5378 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 20:38:29.647085 ip-10-0-138-149 kubenswrapper[5378]: W1030 20:38:29.626547 5378 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 20:38:29.647085 ip-10-0-138-149 kubenswrapper[5378]: W1030 20:38:29.626551 5378 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 20:38:29.647085 ip-10-0-138-149 kubenswrapper[5378]: W1030 20:38:29.626555 5378 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 20:38:29.647085 ip-10-0-138-149 kubenswrapper[5378]: W1030 20:38:29.626573 5378 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 20:38:29.647085 ip-10-0-138-149 kubenswrapper[5378]: W1030 20:38:29.626577 5378 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 20:38:29.647085 ip-10-0-138-149 kubenswrapper[5378]: W1030 20:38:29.626580 5378 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 20:38:29.647748 ip-10-0-138-149 kubenswrapper[5378]: W1030 20:38:29.626582 5378 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 20:38:29.647748 ip-10-0-138-149 kubenswrapper[5378]: W1030 20:38:29.626585 5378 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 20:38:29.647748 ip-10-0-138-149 kubenswrapper[5378]: W1030 20:38:29.626592 5378 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 20:38:29.647748 ip-10-0-138-149 kubenswrapper[5378]: W1030 20:38:29.626597 5378 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 20:38:29.647748 ip-10-0-138-149 kubenswrapper[5378]: W1030 20:38:29.626601 5378 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 20:38:29.647748 ip-10-0-138-149 kubenswrapper[5378]: W1030 20:38:29.626604 5378 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 20:38:29.647748 ip-10-0-138-149 kubenswrapper[5378]: W1030 20:38:29.626607 5378 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 20:38:29.647748 ip-10-0-138-149 kubenswrapper[5378]: W1030 20:38:29.626609 5378 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 20:38:29.647748 ip-10-0-138-149 kubenswrapper[5378]: W1030 20:38:29.626613 5378 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 20:38:29.647748 ip-10-0-138-149 kubenswrapper[5378]: W1030 20:38:29.626616 5378 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 20:38:29.647748 ip-10-0-138-149 kubenswrapper[5378]: W1030 20:38:29.626618 5378 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 20:38:29.647748 ip-10-0-138-149 kubenswrapper[5378]: W1030 20:38:29.626621 5378 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 20:38:29.647748 ip-10-0-138-149 kubenswrapper[5378]: W1030 20:38:29.626624 5378 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 20:38:29.647748 ip-10-0-138-149 kubenswrapper[5378]: W1030 20:38:29.626626 5378 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 20:38:29.647748 ip-10-0-138-149 kubenswrapper[5378]: W1030 20:38:29.626629 5378 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 20:38:29.647748 ip-10-0-138-149 kubenswrapper[5378]: W1030 20:38:29.626632 5378 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 20:38:29.647748 ip-10-0-138-149 kubenswrapper[5378]: W1030 20:38:29.626634 5378 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 20:38:29.647748 ip-10-0-138-149 kubenswrapper[5378]: W1030 20:38:29.626637 5378 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 20:38:29.647748 ip-10-0-138-149 kubenswrapper[5378]: W1030 20:38:29.626639 5378 feature_gate.go:386] unrecognized feature gate: Example Oct 30 20:38:29.647748 ip-10-0-138-149 kubenswrapper[5378]: W1030 20:38:29.626643 5378 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 20:38:29.648408 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:38:29.626678 5378 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Oct 30 20:38:29.648408 ip-10-0-138-149 kubenswrapper[5378]: W1030 20:38:29.626726 5378 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 20:38:29.648408 ip-10-0-138-149 kubenswrapper[5378]: W1030 20:38:29.626732 5378 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 20:38:29.648408 ip-10-0-138-149 kubenswrapper[5378]: W1030 20:38:29.626736 5378 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 20:38:29.648408 ip-10-0-138-149 kubenswrapper[5378]: W1030 20:38:29.626739 5378 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 20:38:29.648408 ip-10-0-138-149 kubenswrapper[5378]: W1030 20:38:29.626743 5378 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 20:38:29.648408 ip-10-0-138-149 kubenswrapper[5378]: W1030 20:38:29.626747 5378 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 20:38:29.648408 ip-10-0-138-149 kubenswrapper[5378]: W1030 20:38:29.626752 5378 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 20:38:29.648408 ip-10-0-138-149 kubenswrapper[5378]: W1030 20:38:29.626756 5378 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 20:38:29.648408 ip-10-0-138-149 kubenswrapper[5378]: W1030 20:38:29.626760 5378 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 20:38:29.648408 ip-10-0-138-149 kubenswrapper[5378]: W1030 20:38:29.626765 5378 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 20:38:29.648408 ip-10-0-138-149 kubenswrapper[5378]: W1030 20:38:29.626770 5378 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 20:38:29.648408 ip-10-0-138-149 kubenswrapper[5378]: W1030 20:38:29.626774 5378 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 20:38:29.648408 ip-10-0-138-149 kubenswrapper[5378]: W1030 20:38:29.626778 5378 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 20:38:29.648408 ip-10-0-138-149 kubenswrapper[5378]: W1030 20:38:29.626785 5378 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 20:38:29.648408 ip-10-0-138-149 kubenswrapper[5378]: W1030 20:38:29.626789 5378 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 20:38:29.648898 ip-10-0-138-149 kubenswrapper[5378]: W1030 20:38:29.626793 5378 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 20:38:29.648898 ip-10-0-138-149 kubenswrapper[5378]: W1030 20:38:29.626797 5378 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 20:38:29.648898 ip-10-0-138-149 kubenswrapper[5378]: W1030 20:38:29.626802 5378 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 20:38:29.648898 ip-10-0-138-149 kubenswrapper[5378]: W1030 20:38:29.626806 5378 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 20:38:29.648898 ip-10-0-138-149 kubenswrapper[5378]: W1030 20:38:29.626811 5378 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 20:38:29.648898 ip-10-0-138-149 kubenswrapper[5378]: W1030 20:38:29.626816 5378 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 20:38:29.648898 ip-10-0-138-149 kubenswrapper[5378]: W1030 20:38:29.626819 5378 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 20:38:29.648898 ip-10-0-138-149 kubenswrapper[5378]: W1030 20:38:29.626821 5378 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 20:38:29.648898 ip-10-0-138-149 kubenswrapper[5378]: W1030 20:38:29.626824 5378 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 20:38:29.648898 ip-10-0-138-149 kubenswrapper[5378]: W1030 20:38:29.626827 5378 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 20:38:29.648898 ip-10-0-138-149 kubenswrapper[5378]: W1030 20:38:29.626830 5378 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 20:38:29.648898 ip-10-0-138-149 kubenswrapper[5378]: W1030 20:38:29.626832 5378 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 20:38:29.648898 ip-10-0-138-149 kubenswrapper[5378]: W1030 20:38:29.626835 5378 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 20:38:29.648898 ip-10-0-138-149 kubenswrapper[5378]: W1030 20:38:29.626838 5378 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 20:38:29.648898 ip-10-0-138-149 kubenswrapper[5378]: W1030 20:38:29.626840 5378 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 20:38:29.648898 ip-10-0-138-149 kubenswrapper[5378]: W1030 20:38:29.626843 5378 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 20:38:29.648898 ip-10-0-138-149 kubenswrapper[5378]: W1030 20:38:29.626846 5378 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 20:38:29.648898 ip-10-0-138-149 kubenswrapper[5378]: W1030 20:38:29.626849 5378 feature_gate.go:386] unrecognized feature gate: Example Oct 30 20:38:29.648898 ip-10-0-138-149 kubenswrapper[5378]: W1030 20:38:29.626852 5378 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 20:38:29.648898 ip-10-0-138-149 kubenswrapper[5378]: W1030 20:38:29.626855 5378 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 20:38:29.649480 ip-10-0-138-149 kubenswrapper[5378]: W1030 20:38:29.626858 5378 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 20:38:29.649480 ip-10-0-138-149 kubenswrapper[5378]: W1030 20:38:29.626861 5378 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 20:38:29.649480 ip-10-0-138-149 kubenswrapper[5378]: W1030 20:38:29.626863 5378 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 20:38:29.649480 ip-10-0-138-149 kubenswrapper[5378]: W1030 20:38:29.626866 5378 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 20:38:29.649480 ip-10-0-138-149 kubenswrapper[5378]: W1030 20:38:29.626868 5378 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 20:38:29.649480 ip-10-0-138-149 kubenswrapper[5378]: W1030 20:38:29.626871 5378 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 20:38:29.649480 ip-10-0-138-149 kubenswrapper[5378]: W1030 20:38:29.626874 5378 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 20:38:29.649480 ip-10-0-138-149 kubenswrapper[5378]: W1030 20:38:29.626877 5378 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 20:38:29.649480 ip-10-0-138-149 kubenswrapper[5378]: W1030 20:38:29.626880 5378 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 20:38:29.649480 ip-10-0-138-149 kubenswrapper[5378]: W1030 20:38:29.626882 5378 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 20:38:29.649480 ip-10-0-138-149 kubenswrapper[5378]: W1030 20:38:29.626886 5378 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 20:38:29.649480 ip-10-0-138-149 kubenswrapper[5378]: W1030 20:38:29.626889 5378 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 20:38:29.649480 ip-10-0-138-149 kubenswrapper[5378]: W1030 20:38:29.626891 5378 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 20:38:29.649480 ip-10-0-138-149 kubenswrapper[5378]: W1030 20:38:29.626894 5378 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 20:38:29.649480 ip-10-0-138-149 kubenswrapper[5378]: W1030 20:38:29.626897 5378 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 20:38:29.649480 ip-10-0-138-149 kubenswrapper[5378]: W1030 20:38:29.626899 5378 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 20:38:29.649480 ip-10-0-138-149 kubenswrapper[5378]: W1030 20:38:29.626902 5378 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 20:38:29.649480 ip-10-0-138-149 kubenswrapper[5378]: W1030 20:38:29.626908 5378 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 20:38:29.649480 ip-10-0-138-149 kubenswrapper[5378]: W1030 20:38:29.626912 5378 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 20:38:29.649480 ip-10-0-138-149 kubenswrapper[5378]: W1030 20:38:29.626917 5378 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 20:38:29.650155 ip-10-0-138-149 kubenswrapper[5378]: W1030 20:38:29.626922 5378 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 20:38:29.650155 ip-10-0-138-149 kubenswrapper[5378]: W1030 20:38:29.626927 5378 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 20:38:29.650155 ip-10-0-138-149 kubenswrapper[5378]: W1030 20:38:29.626932 5378 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 20:38:29.650155 ip-10-0-138-149 kubenswrapper[5378]: W1030 20:38:29.626936 5378 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 20:38:29.650155 ip-10-0-138-149 kubenswrapper[5378]: W1030 20:38:29.626939 5378 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 20:38:29.650155 ip-10-0-138-149 kubenswrapper[5378]: W1030 20:38:29.626941 5378 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 20:38:29.650155 ip-10-0-138-149 kubenswrapper[5378]: W1030 20:38:29.626944 5378 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 20:38:29.650155 ip-10-0-138-149 kubenswrapper[5378]: W1030 20:38:29.626947 5378 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 20:38:29.650155 ip-10-0-138-149 kubenswrapper[5378]: W1030 20:38:29.626949 5378 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 20:38:29.650155 ip-10-0-138-149 kubenswrapper[5378]: W1030 20:38:29.626953 5378 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 20:38:29.650155 ip-10-0-138-149 kubenswrapper[5378]: W1030 20:38:29.626956 5378 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 20:38:29.650155 ip-10-0-138-149 kubenswrapper[5378]: W1030 20:38:29.626958 5378 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 20:38:29.650155 ip-10-0-138-149 kubenswrapper[5378]: W1030 20:38:29.626961 5378 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 20:38:29.650155 ip-10-0-138-149 kubenswrapper[5378]: W1030 20:38:29.626967 5378 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 20:38:29.650155 ip-10-0-138-149 kubenswrapper[5378]: W1030 20:38:29.626971 5378 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 20:38:29.650155 ip-10-0-138-149 kubenswrapper[5378]: W1030 20:38:29.626975 5378 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 20:38:29.650155 ip-10-0-138-149 kubenswrapper[5378]: W1030 20:38:29.626979 5378 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 20:38:29.650155 ip-10-0-138-149 kubenswrapper[5378]: W1030 20:38:29.626984 5378 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 20:38:29.650155 ip-10-0-138-149 kubenswrapper[5378]: W1030 20:38:29.626987 5378 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 20:38:29.650773 ip-10-0-138-149 kubenswrapper[5378]: W1030 20:38:29.626989 5378 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 20:38:29.650773 ip-10-0-138-149 kubenswrapper[5378]: W1030 20:38:29.626992 5378 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 20:38:29.650773 ip-10-0-138-149 kubenswrapper[5378]: W1030 20:38:29.626995 5378 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 20:38:29.650773 ip-10-0-138-149 kubenswrapper[5378]: W1030 20:38:29.626999 5378 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 20:38:29.650773 ip-10-0-138-149 kubenswrapper[5378]: W1030 20:38:29.627001 5378 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 20:38:29.650773 ip-10-0-138-149 kubenswrapper[5378]: W1030 20:38:29.627004 5378 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 20:38:29.650773 ip-10-0-138-149 kubenswrapper[5378]: W1030 20:38:29.627007 5378 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 20:38:29.650773 ip-10-0-138-149 kubenswrapper[5378]: W1030 20:38:29.627010 5378 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 20:38:29.650773 ip-10-0-138-149 kubenswrapper[5378]: W1030 20:38:29.627013 5378 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 20:38:29.650773 ip-10-0-138-149 kubenswrapper[5378]: W1030 20:38:29.627015 5378 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 20:38:29.650773 ip-10-0-138-149 kubenswrapper[5378]: W1030 20:38:29.627019 5378 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 20:38:29.650773 ip-10-0-138-149 kubenswrapper[5378]: W1030 20:38:29.627023 5378 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 20:38:29.650773 ip-10-0-138-149 kubenswrapper[5378]: W1030 20:38:29.627027 5378 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 20:38:29.650773 ip-10-0-138-149 kubenswrapper[5378]: W1030 20:38:29.627030 5378 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 20:38:29.650773 ip-10-0-138-149 kubenswrapper[5378]: W1030 20:38:29.627032 5378 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 20:38:29.650773 ip-10-0-138-149 kubenswrapper[5378]: W1030 20:38:29.627035 5378 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 20:38:29.650773 ip-10-0-138-149 kubenswrapper[5378]: W1030 20:38:29.627039 5378 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 20:38:29.650773 ip-10-0-138-149 kubenswrapper[5378]: W1030 20:38:29.627041 5378 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 20:38:29.650773 ip-10-0-138-149 kubenswrapper[5378]: W1030 20:38:29.627044 5378 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 20:38:29.651320 ip-10-0-138-149 kubenswrapper[5378]: W1030 20:38:29.627046 5378 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 20:38:29.651320 ip-10-0-138-149 kubenswrapper[5378]: W1030 20:38:29.627049 5378 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 20:38:29.651320 ip-10-0-138-149 kubenswrapper[5378]: W1030 20:38:29.627052 5378 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 20:38:29.651320 ip-10-0-138-149 kubenswrapper[5378]: W1030 20:38:29.627054 5378 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 20:38:29.651320 ip-10-0-138-149 kubenswrapper[5378]: W1030 20:38:29.627057 5378 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 20:38:29.651320 ip-10-0-138-149 kubenswrapper[5378]: W1030 20:38:29.627060 5378 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 20:38:29.651320 ip-10-0-138-149 kubenswrapper[5378]: W1030 20:38:29.627063 5378 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 20:38:29.651320 ip-10-0-138-149 kubenswrapper[5378]: W1030 20:38:29.627066 5378 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 20:38:29.651320 ip-10-0-138-149 kubenswrapper[5378]: W1030 20:38:29.627070 5378 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 20:38:29.651320 ip-10-0-138-149 kubenswrapper[5378]: W1030 20:38:29.627074 5378 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 20:38:29.651320 ip-10-0-138-149 kubenswrapper[5378]: W1030 20:38:29.627076 5378 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 20:38:29.651320 ip-10-0-138-149 kubenswrapper[5378]: W1030 20:38:29.627079 5378 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 20:38:29.651320 ip-10-0-138-149 kubenswrapper[5378]: W1030 20:38:29.627097 5378 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 20:38:29.651320 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:38:29.634154 5378 server.go:525] "Kubelet version" kubeletVersion="v1.35.9" Oct 30 20:38:29.651320 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:38:29.634178 5378 server.go:527] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Oct 30 20:38:29.651320 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:38:29.634210 5378 watchdog_linux.go:95] "Systemd watchdog is not enabled" Oct 30 20:38:29.651320 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:38:29.634218 5378 watchdog_linux.go:138] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Oct 30 20:38:29.651320 ip-10-0-138-149 kubenswrapper[5378]: W1030 20:38:29.634284 5378 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 20:38:29.651320 ip-10-0-138-149 kubenswrapper[5378]: W1030 20:38:29.634292 5378 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 20:38:29.651906 ip-10-0-138-149 kubenswrapper[5378]: W1030 20:38:29.634298 5378 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 20:38:29.651906 ip-10-0-138-149 kubenswrapper[5378]: W1030 20:38:29.634303 5378 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 20:38:29.651906 ip-10-0-138-149 kubenswrapper[5378]: W1030 20:38:29.634309 5378 feature_gate.go:386] unrecognized feature gate: Example Oct 30 20:38:29.651906 ip-10-0-138-149 kubenswrapper[5378]: W1030 20:38:29.634314 5378 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 20:38:29.651906 ip-10-0-138-149 kubenswrapper[5378]: W1030 20:38:29.634318 5378 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 20:38:29.651906 ip-10-0-138-149 kubenswrapper[5378]: W1030 20:38:29.634323 5378 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 20:38:29.651906 ip-10-0-138-149 kubenswrapper[5378]: W1030 20:38:29.634329 5378 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 20:38:29.651906 ip-10-0-138-149 kubenswrapper[5378]: W1030 20:38:29.634333 5378 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 20:38:29.651906 ip-10-0-138-149 kubenswrapper[5378]: W1030 20:38:29.634337 5378 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 20:38:29.651906 ip-10-0-138-149 kubenswrapper[5378]: W1030 20:38:29.634342 5378 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 20:38:29.651906 ip-10-0-138-149 kubenswrapper[5378]: W1030 20:38:29.634352 5378 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 20:38:29.651906 ip-10-0-138-149 kubenswrapper[5378]: W1030 20:38:29.634357 5378 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 20:38:29.651906 ip-10-0-138-149 kubenswrapper[5378]: W1030 20:38:29.634361 5378 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 20:38:29.651906 ip-10-0-138-149 kubenswrapper[5378]: W1030 20:38:29.634365 5378 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 20:38:29.651906 ip-10-0-138-149 kubenswrapper[5378]: W1030 20:38:29.634367 5378 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 20:38:29.651906 ip-10-0-138-149 kubenswrapper[5378]: W1030 20:38:29.634370 5378 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 20:38:29.651906 ip-10-0-138-149 kubenswrapper[5378]: W1030 20:38:29.634373 5378 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 20:38:29.651906 ip-10-0-138-149 kubenswrapper[5378]: W1030 20:38:29.634378 5378 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 20:38:29.651906 ip-10-0-138-149 kubenswrapper[5378]: W1030 20:38:29.634383 5378 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 20:38:29.651906 ip-10-0-138-149 kubenswrapper[5378]: W1030 20:38:29.634386 5378 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 20:38:29.652411 ip-10-0-138-149 kubenswrapper[5378]: W1030 20:38:29.634389 5378 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 20:38:29.652411 ip-10-0-138-149 kubenswrapper[5378]: W1030 20:38:29.634391 5378 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 20:38:29.652411 ip-10-0-138-149 kubenswrapper[5378]: W1030 20:38:29.634394 5378 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 20:38:29.652411 ip-10-0-138-149 kubenswrapper[5378]: W1030 20:38:29.634397 5378 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 20:38:29.652411 ip-10-0-138-149 kubenswrapper[5378]: W1030 20:38:29.634400 5378 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 20:38:29.652411 ip-10-0-138-149 kubenswrapper[5378]: W1030 20:38:29.634402 5378 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 20:38:29.652411 ip-10-0-138-149 kubenswrapper[5378]: W1030 20:38:29.634406 5378 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 20:38:29.652411 ip-10-0-138-149 kubenswrapper[5378]: W1030 20:38:29.634410 5378 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 20:38:29.652411 ip-10-0-138-149 kubenswrapper[5378]: W1030 20:38:29.634413 5378 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 20:38:29.652411 ip-10-0-138-149 kubenswrapper[5378]: W1030 20:38:29.634415 5378 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 20:38:29.652411 ip-10-0-138-149 kubenswrapper[5378]: W1030 20:38:29.634418 5378 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 20:38:29.652411 ip-10-0-138-149 kubenswrapper[5378]: W1030 20:38:29.634421 5378 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 20:38:29.652411 ip-10-0-138-149 kubenswrapper[5378]: W1030 20:38:29.634423 5378 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 20:38:29.652411 ip-10-0-138-149 kubenswrapper[5378]: W1030 20:38:29.634426 5378 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 20:38:29.652411 ip-10-0-138-149 kubenswrapper[5378]: W1030 20:38:29.634429 5378 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 20:38:29.652411 ip-10-0-138-149 kubenswrapper[5378]: W1030 20:38:29.634431 5378 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 20:38:29.652411 ip-10-0-138-149 kubenswrapper[5378]: W1030 20:38:29.634435 5378 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 20:38:29.652411 ip-10-0-138-149 kubenswrapper[5378]: W1030 20:38:29.634440 5378 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 20:38:29.652411 ip-10-0-138-149 kubenswrapper[5378]: W1030 20:38:29.634444 5378 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 20:38:29.652411 ip-10-0-138-149 kubenswrapper[5378]: W1030 20:38:29.634449 5378 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 20:38:29.653005 ip-10-0-138-149 kubenswrapper[5378]: W1030 20:38:29.634453 5378 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 20:38:29.653005 ip-10-0-138-149 kubenswrapper[5378]: W1030 20:38:29.634457 5378 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 20:38:29.653005 ip-10-0-138-149 kubenswrapper[5378]: W1030 20:38:29.634462 5378 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 20:38:29.653005 ip-10-0-138-149 kubenswrapper[5378]: W1030 20:38:29.634467 5378 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 20:38:29.653005 ip-10-0-138-149 kubenswrapper[5378]: W1030 20:38:29.634471 5378 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 20:38:29.653005 ip-10-0-138-149 kubenswrapper[5378]: W1030 20:38:29.634476 5378 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 20:38:29.653005 ip-10-0-138-149 kubenswrapper[5378]: W1030 20:38:29.634480 5378 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 20:38:29.653005 ip-10-0-138-149 kubenswrapper[5378]: W1030 20:38:29.634484 5378 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 20:38:29.653005 ip-10-0-138-149 kubenswrapper[5378]: W1030 20:38:29.634488 5378 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 20:38:29.653005 ip-10-0-138-149 kubenswrapper[5378]: W1030 20:38:29.634492 5378 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 20:38:29.653005 ip-10-0-138-149 kubenswrapper[5378]: W1030 20:38:29.634497 5378 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 20:38:29.653005 ip-10-0-138-149 kubenswrapper[5378]: W1030 20:38:29.634504 5378 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 20:38:29.653005 ip-10-0-138-149 kubenswrapper[5378]: W1030 20:38:29.634509 5378 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 20:38:29.653005 ip-10-0-138-149 kubenswrapper[5378]: W1030 20:38:29.634512 5378 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 20:38:29.653005 ip-10-0-138-149 kubenswrapper[5378]: W1030 20:38:29.634515 5378 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 20:38:29.653005 ip-10-0-138-149 kubenswrapper[5378]: W1030 20:38:29.634518 5378 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 20:38:29.653005 ip-10-0-138-149 kubenswrapper[5378]: W1030 20:38:29.634521 5378 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 20:38:29.653005 ip-10-0-138-149 kubenswrapper[5378]: W1030 20:38:29.634523 5378 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 20:38:29.653005 ip-10-0-138-149 kubenswrapper[5378]: W1030 20:38:29.634526 5378 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 20:38:29.653539 ip-10-0-138-149 kubenswrapper[5378]: W1030 20:38:29.634529 5378 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 20:38:29.653539 ip-10-0-138-149 kubenswrapper[5378]: W1030 20:38:29.634532 5378 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 20:38:29.653539 ip-10-0-138-149 kubenswrapper[5378]: W1030 20:38:29.634535 5378 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 20:38:29.653539 ip-10-0-138-149 kubenswrapper[5378]: W1030 20:38:29.634539 5378 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 20:38:29.653539 ip-10-0-138-149 kubenswrapper[5378]: W1030 20:38:29.634543 5378 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 20:38:29.653539 ip-10-0-138-149 kubenswrapper[5378]: W1030 20:38:29.634546 5378 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 20:38:29.653539 ip-10-0-138-149 kubenswrapper[5378]: W1030 20:38:29.634548 5378 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 20:38:29.653539 ip-10-0-138-149 kubenswrapper[5378]: W1030 20:38:29.634551 5378 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 20:38:29.653539 ip-10-0-138-149 kubenswrapper[5378]: W1030 20:38:29.634554 5378 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 20:38:29.653539 ip-10-0-138-149 kubenswrapper[5378]: W1030 20:38:29.634556 5378 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 20:38:29.653539 ip-10-0-138-149 kubenswrapper[5378]: W1030 20:38:29.634559 5378 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 20:38:29.653539 ip-10-0-138-149 kubenswrapper[5378]: W1030 20:38:29.634574 5378 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 20:38:29.653539 ip-10-0-138-149 kubenswrapper[5378]: W1030 20:38:29.634579 5378 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 20:38:29.653539 ip-10-0-138-149 kubenswrapper[5378]: W1030 20:38:29.634583 5378 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 20:38:29.653539 ip-10-0-138-149 kubenswrapper[5378]: W1030 20:38:29.634587 5378 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 20:38:29.653539 ip-10-0-138-149 kubenswrapper[5378]: W1030 20:38:29.634591 5378 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 20:38:29.653539 ip-10-0-138-149 kubenswrapper[5378]: W1030 20:38:29.634595 5378 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 20:38:29.653539 ip-10-0-138-149 kubenswrapper[5378]: W1030 20:38:29.634597 5378 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 20:38:29.653539 ip-10-0-138-149 kubenswrapper[5378]: W1030 20:38:29.634601 5378 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 20:38:29.654309 ip-10-0-138-149 kubenswrapper[5378]: W1030 20:38:29.634604 5378 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 20:38:29.654309 ip-10-0-138-149 kubenswrapper[5378]: W1030 20:38:29.634607 5378 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 20:38:29.654309 ip-10-0-138-149 kubenswrapper[5378]: W1030 20:38:29.634610 5378 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 20:38:29.654309 ip-10-0-138-149 kubenswrapper[5378]: W1030 20:38:29.634612 5378 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 20:38:29.654309 ip-10-0-138-149 kubenswrapper[5378]: W1030 20:38:29.634615 5378 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 20:38:29.654309 ip-10-0-138-149 kubenswrapper[5378]: W1030 20:38:29.634618 5378 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 20:38:29.654309 ip-10-0-138-149 kubenswrapper[5378]: W1030 20:38:29.634620 5378 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 20:38:29.654309 ip-10-0-138-149 kubenswrapper[5378]: W1030 20:38:29.634623 5378 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 20:38:29.654309 ip-10-0-138-149 kubenswrapper[5378]: W1030 20:38:29.634626 5378 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 20:38:29.654309 ip-10-0-138-149 kubenswrapper[5378]: W1030 20:38:29.634628 5378 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 20:38:29.654309 ip-10-0-138-149 kubenswrapper[5378]: W1030 20:38:29.634631 5378 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 20:38:29.654309 ip-10-0-138-149 kubenswrapper[5378]: W1030 20:38:29.634636 5378 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 20:38:29.654309 ip-10-0-138-149 kubenswrapper[5378]: W1030 20:38:29.634640 5378 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 20:38:29.654309 ip-10-0-138-149 kubenswrapper[5378]: W1030 20:38:29.634645 5378 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 20:38:29.654309 ip-10-0-138-149 kubenswrapper[5378]: W1030 20:38:29.634649 5378 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 20:38:29.654309 ip-10-0-138-149 kubenswrapper[5378]: W1030 20:38:29.634653 5378 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 20:38:29.654309 ip-10-0-138-149 kubenswrapper[5378]: W1030 20:38:29.634657 5378 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 20:38:29.654309 ip-10-0-138-149 kubenswrapper[5378]: W1030 20:38:29.634662 5378 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 20:38:29.654309 ip-10-0-138-149 kubenswrapper[5378]: W1030 20:38:29.634666 5378 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 20:38:29.654914 ip-10-0-138-149 kubenswrapper[5378]: W1030 20:38:29.634670 5378 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 20:38:29.654914 ip-10-0-138-149 kubenswrapper[5378]: W1030 20:38:29.634674 5378 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 20:38:29.654914 ip-10-0-138-149 kubenswrapper[5378]: W1030 20:38:29.634678 5378 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 20:38:29.654914 ip-10-0-138-149 kubenswrapper[5378]: W1030 20:38:29.634682 5378 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 20:38:29.654914 ip-10-0-138-149 kubenswrapper[5378]: W1030 20:38:29.634687 5378 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 20:38:29.654914 ip-10-0-138-149 kubenswrapper[5378]: W1030 20:38:29.634691 5378 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 20:38:29.654914 ip-10-0-138-149 kubenswrapper[5378]: W1030 20:38:29.634695 5378 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 20:38:29.654914 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:38:29.634738 5378 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Oct 30 20:38:29.654914 ip-10-0-138-149 kubenswrapper[5378]: W1030 20:38:29.634965 5378 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 20:38:29.654914 ip-10-0-138-149 kubenswrapper[5378]: W1030 20:38:29.634973 5378 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 20:38:29.654914 ip-10-0-138-149 kubenswrapper[5378]: W1030 20:38:29.634978 5378 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 20:38:29.654914 ip-10-0-138-149 kubenswrapper[5378]: W1030 20:38:29.634982 5378 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 20:38:29.654914 ip-10-0-138-149 kubenswrapper[5378]: W1030 20:38:29.634987 5378 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 20:38:29.654914 ip-10-0-138-149 kubenswrapper[5378]: W1030 20:38:29.634992 5378 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 20:38:29.654914 ip-10-0-138-149 kubenswrapper[5378]: W1030 20:38:29.634996 5378 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 20:38:29.654914 ip-10-0-138-149 kubenswrapper[5378]: W1030 20:38:29.635000 5378 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 20:38:29.655389 ip-10-0-138-149 kubenswrapper[5378]: W1030 20:38:29.635004 5378 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 20:38:29.655389 ip-10-0-138-149 kubenswrapper[5378]: W1030 20:38:29.635008 5378 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 20:38:29.655389 ip-10-0-138-149 kubenswrapper[5378]: W1030 20:38:29.635012 5378 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 20:38:29.655389 ip-10-0-138-149 kubenswrapper[5378]: W1030 20:38:29.635016 5378 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 20:38:29.655389 ip-10-0-138-149 kubenswrapper[5378]: W1030 20:38:29.635020 5378 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 20:38:29.655389 ip-10-0-138-149 kubenswrapper[5378]: W1030 20:38:29.635025 5378 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 20:38:29.655389 ip-10-0-138-149 kubenswrapper[5378]: W1030 20:38:29.635029 5378 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 20:38:29.655389 ip-10-0-138-149 kubenswrapper[5378]: W1030 20:38:29.635034 5378 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 20:38:29.655389 ip-10-0-138-149 kubenswrapper[5378]: W1030 20:38:29.635038 5378 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 20:38:29.655389 ip-10-0-138-149 kubenswrapper[5378]: W1030 20:38:29.635042 5378 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 20:38:29.655389 ip-10-0-138-149 kubenswrapper[5378]: W1030 20:38:29.635047 5378 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 20:38:29.655389 ip-10-0-138-149 kubenswrapper[5378]: W1030 20:38:29.635051 5378 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 20:38:29.655389 ip-10-0-138-149 kubenswrapper[5378]: W1030 20:38:29.635055 5378 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 20:38:29.655389 ip-10-0-138-149 kubenswrapper[5378]: W1030 20:38:29.635059 5378 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 20:38:29.655389 ip-10-0-138-149 kubenswrapper[5378]: W1030 20:38:29.635063 5378 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 20:38:29.655389 ip-10-0-138-149 kubenswrapper[5378]: W1030 20:38:29.635067 5378 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 20:38:29.655389 ip-10-0-138-149 kubenswrapper[5378]: W1030 20:38:29.635071 5378 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 20:38:29.655389 ip-10-0-138-149 kubenswrapper[5378]: W1030 20:38:29.635075 5378 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 20:38:29.655389 ip-10-0-138-149 kubenswrapper[5378]: W1030 20:38:29.635079 5378 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 20:38:29.655389 ip-10-0-138-149 kubenswrapper[5378]: W1030 20:38:29.635083 5378 feature_gate.go:386] unrecognized feature gate: Example Oct 30 20:38:29.656110 ip-10-0-138-149 kubenswrapper[5378]: W1030 20:38:29.635088 5378 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 20:38:29.656110 ip-10-0-138-149 kubenswrapper[5378]: W1030 20:38:29.635092 5378 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 20:38:29.656110 ip-10-0-138-149 kubenswrapper[5378]: W1030 20:38:29.635096 5378 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 20:38:29.656110 ip-10-0-138-149 kubenswrapper[5378]: W1030 20:38:29.635100 5378 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 20:38:29.656110 ip-10-0-138-149 kubenswrapper[5378]: W1030 20:38:29.635104 5378 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 20:38:29.656110 ip-10-0-138-149 kubenswrapper[5378]: W1030 20:38:29.635111 5378 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 20:38:29.656110 ip-10-0-138-149 kubenswrapper[5378]: W1030 20:38:29.635118 5378 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 20:38:29.656110 ip-10-0-138-149 kubenswrapper[5378]: W1030 20:38:29.635122 5378 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 20:38:29.656110 ip-10-0-138-149 kubenswrapper[5378]: W1030 20:38:29.635127 5378 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 20:38:29.656110 ip-10-0-138-149 kubenswrapper[5378]: W1030 20:38:29.635131 5378 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 20:38:29.656110 ip-10-0-138-149 kubenswrapper[5378]: W1030 20:38:29.635138 5378 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 20:38:29.656110 ip-10-0-138-149 kubenswrapper[5378]: W1030 20:38:29.635142 5378 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 20:38:29.656110 ip-10-0-138-149 kubenswrapper[5378]: W1030 20:38:29.635146 5378 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 20:38:29.656110 ip-10-0-138-149 kubenswrapper[5378]: W1030 20:38:29.635153 5378 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 20:38:29.656110 ip-10-0-138-149 kubenswrapper[5378]: W1030 20:38:29.635157 5378 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 20:38:29.656110 ip-10-0-138-149 kubenswrapper[5378]: W1030 20:38:29.635163 5378 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 20:38:29.656110 ip-10-0-138-149 kubenswrapper[5378]: W1030 20:38:29.635167 5378 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 20:38:29.656110 ip-10-0-138-149 kubenswrapper[5378]: W1030 20:38:29.635171 5378 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 20:38:29.656110 ip-10-0-138-149 kubenswrapper[5378]: W1030 20:38:29.635175 5378 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 20:38:29.656907 ip-10-0-138-149 kubenswrapper[5378]: W1030 20:38:29.635179 5378 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 20:38:29.656907 ip-10-0-138-149 kubenswrapper[5378]: W1030 20:38:29.635184 5378 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 20:38:29.656907 ip-10-0-138-149 kubenswrapper[5378]: W1030 20:38:29.635188 5378 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 20:38:29.656907 ip-10-0-138-149 kubenswrapper[5378]: W1030 20:38:29.635192 5378 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 20:38:29.656907 ip-10-0-138-149 kubenswrapper[5378]: W1030 20:38:29.635196 5378 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 20:38:29.656907 ip-10-0-138-149 kubenswrapper[5378]: W1030 20:38:29.635200 5378 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 20:38:29.656907 ip-10-0-138-149 kubenswrapper[5378]: W1030 20:38:29.635204 5378 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 20:38:29.656907 ip-10-0-138-149 kubenswrapper[5378]: W1030 20:38:29.635208 5378 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 20:38:29.656907 ip-10-0-138-149 kubenswrapper[5378]: W1030 20:38:29.635212 5378 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 20:38:29.656907 ip-10-0-138-149 kubenswrapper[5378]: W1030 20:38:29.635217 5378 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 20:38:29.656907 ip-10-0-138-149 kubenswrapper[5378]: W1030 20:38:29.635221 5378 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 20:38:29.656907 ip-10-0-138-149 kubenswrapper[5378]: W1030 20:38:29.635225 5378 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 20:38:29.656907 ip-10-0-138-149 kubenswrapper[5378]: W1030 20:38:29.635232 5378 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 20:38:29.656907 ip-10-0-138-149 kubenswrapper[5378]: W1030 20:38:29.635236 5378 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 20:38:29.656907 ip-10-0-138-149 kubenswrapper[5378]: W1030 20:38:29.635240 5378 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 20:38:29.656907 ip-10-0-138-149 kubenswrapper[5378]: W1030 20:38:29.635244 5378 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 20:38:29.656907 ip-10-0-138-149 kubenswrapper[5378]: W1030 20:38:29.635248 5378 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 20:38:29.656907 ip-10-0-138-149 kubenswrapper[5378]: W1030 20:38:29.635252 5378 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 20:38:29.656907 ip-10-0-138-149 kubenswrapper[5378]: W1030 20:38:29.635257 5378 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 20:38:29.656907 ip-10-0-138-149 kubenswrapper[5378]: W1030 20:38:29.635261 5378 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 20:38:29.657651 ip-10-0-138-149 kubenswrapper[5378]: W1030 20:38:29.635265 5378 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 20:38:29.657651 ip-10-0-138-149 kubenswrapper[5378]: W1030 20:38:29.635269 5378 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 20:38:29.657651 ip-10-0-138-149 kubenswrapper[5378]: W1030 20:38:29.635273 5378 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 20:38:29.657651 ip-10-0-138-149 kubenswrapper[5378]: W1030 20:38:29.635277 5378 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 20:38:29.657651 ip-10-0-138-149 kubenswrapper[5378]: W1030 20:38:29.635282 5378 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 20:38:29.657651 ip-10-0-138-149 kubenswrapper[5378]: W1030 20:38:29.635287 5378 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 20:38:29.657651 ip-10-0-138-149 kubenswrapper[5378]: W1030 20:38:29.635290 5378 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 20:38:29.657651 ip-10-0-138-149 kubenswrapper[5378]: W1030 20:38:29.635295 5378 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 20:38:29.657651 ip-10-0-138-149 kubenswrapper[5378]: W1030 20:38:29.635299 5378 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 20:38:29.657651 ip-10-0-138-149 kubenswrapper[5378]: W1030 20:38:29.635303 5378 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 20:38:29.657651 ip-10-0-138-149 kubenswrapper[5378]: W1030 20:38:29.635307 5378 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 20:38:29.657651 ip-10-0-138-149 kubenswrapper[5378]: W1030 20:38:29.635311 5378 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 20:38:29.657651 ip-10-0-138-149 kubenswrapper[5378]: W1030 20:38:29.635315 5378 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 20:38:29.657651 ip-10-0-138-149 kubenswrapper[5378]: W1030 20:38:29.635319 5378 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 20:38:29.657651 ip-10-0-138-149 kubenswrapper[5378]: W1030 20:38:29.635323 5378 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 20:38:29.657651 ip-10-0-138-149 kubenswrapper[5378]: W1030 20:38:29.635327 5378 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 20:38:29.657651 ip-10-0-138-149 kubenswrapper[5378]: W1030 20:38:29.635331 5378 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 20:38:29.657651 ip-10-0-138-149 kubenswrapper[5378]: W1030 20:38:29.635335 5378 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 20:38:29.657651 ip-10-0-138-149 kubenswrapper[5378]: W1030 20:38:29.635340 5378 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 20:38:29.658210 ip-10-0-138-149 kubenswrapper[5378]: W1030 20:38:29.635343 5378 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 20:38:29.658210 ip-10-0-138-149 kubenswrapper[5378]: W1030 20:38:29.635348 5378 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 20:38:29.658210 ip-10-0-138-149 kubenswrapper[5378]: W1030 20:38:29.635351 5378 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 20:38:29.658210 ip-10-0-138-149 kubenswrapper[5378]: W1030 20:38:29.635355 5378 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 20:38:29.658210 ip-10-0-138-149 kubenswrapper[5378]: W1030 20:38:29.635359 5378 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 20:38:29.658210 ip-10-0-138-149 kubenswrapper[5378]: W1030 20:38:29.635365 5378 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 20:38:29.658210 ip-10-0-138-149 kubenswrapper[5378]: W1030 20:38:29.635369 5378 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 20:38:29.658210 ip-10-0-138-149 kubenswrapper[5378]: W1030 20:38:29.635373 5378 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 20:38:29.658210 ip-10-0-138-149 kubenswrapper[5378]: W1030 20:38:29.635377 5378 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 20:38:29.658210 ip-10-0-138-149 kubenswrapper[5378]: W1030 20:38:29.635381 5378 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 20:38:29.658210 ip-10-0-138-149 kubenswrapper[5378]: W1030 20:38:29.635385 5378 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 20:38:29.658210 ip-10-0-138-149 kubenswrapper[5378]: W1030 20:38:29.635389 5378 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 20:38:29.658210 ip-10-0-138-149 kubenswrapper[5378]: W1030 20:38:29.635393 5378 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 20:38:29.658210 ip-10-0-138-149 kubenswrapper[5378]: W1030 20:38:29.635397 5378 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 20:38:29.658210 ip-10-0-138-149 kubenswrapper[5378]: W1030 20:38:29.635401 5378 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 20:38:29.658210 ip-10-0-138-149 kubenswrapper[5378]: W1030 20:38:29.635405 5378 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 20:38:29.658210 ip-10-0-138-149 kubenswrapper[5378]: W1030 20:38:29.635409 5378 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 20:38:29.658210 ip-10-0-138-149 kubenswrapper[5378]: W1030 20:38:29.635413 5378 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 20:38:29.658210 ip-10-0-138-149 kubenswrapper[5378]: W1030 20:38:29.635418 5378 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 20:38:29.658210 ip-10-0-138-149 kubenswrapper[5378]: W1030 20:38:29.635422 5378 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 20:38:29.658819 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:38:29.635464 5378 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Oct 30 20:38:29.658819 ip-10-0-138-149 kubenswrapper[5378]: W1030 20:38:29.635514 5378 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 20:38:29.658819 ip-10-0-138-149 kubenswrapper[5378]: W1030 20:38:29.635520 5378 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 20:38:29.658819 ip-10-0-138-149 kubenswrapper[5378]: W1030 20:38:29.635525 5378 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 20:38:29.658819 ip-10-0-138-149 kubenswrapper[5378]: W1030 20:38:29.635529 5378 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 20:38:29.658819 ip-10-0-138-149 kubenswrapper[5378]: W1030 20:38:29.635533 5378 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 20:38:29.658819 ip-10-0-138-149 kubenswrapper[5378]: W1030 20:38:29.635537 5378 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 20:38:29.658819 ip-10-0-138-149 kubenswrapper[5378]: W1030 20:38:29.635542 5378 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 20:38:29.658819 ip-10-0-138-149 kubenswrapper[5378]: W1030 20:38:29.635547 5378 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 20:38:29.658819 ip-10-0-138-149 kubenswrapper[5378]: W1030 20:38:29.635550 5378 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 20:38:29.658819 ip-10-0-138-149 kubenswrapper[5378]: W1030 20:38:29.635554 5378 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 20:38:29.658819 ip-10-0-138-149 kubenswrapper[5378]: W1030 20:38:29.635559 5378 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 20:38:29.658819 ip-10-0-138-149 kubenswrapper[5378]: W1030 20:38:29.635586 5378 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 20:38:29.658819 ip-10-0-138-149 kubenswrapper[5378]: W1030 20:38:29.635591 5378 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 20:38:29.658819 ip-10-0-138-149 kubenswrapper[5378]: W1030 20:38:29.635595 5378 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 20:38:29.658819 ip-10-0-138-149 kubenswrapper[5378]: W1030 20:38:29.635599 5378 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 20:38:29.658819 ip-10-0-138-149 kubenswrapper[5378]: W1030 20:38:29.635604 5378 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 20:38:29.659349 ip-10-0-138-149 kubenswrapper[5378]: W1030 20:38:29.635608 5378 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 20:38:29.659349 ip-10-0-138-149 kubenswrapper[5378]: W1030 20:38:29.635614 5378 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 20:38:29.659349 ip-10-0-138-149 kubenswrapper[5378]: W1030 20:38:29.635618 5378 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 20:38:29.659349 ip-10-0-138-149 kubenswrapper[5378]: W1030 20:38:29.635622 5378 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 20:38:29.659349 ip-10-0-138-149 kubenswrapper[5378]: W1030 20:38:29.635626 5378 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 20:38:29.659349 ip-10-0-138-149 kubenswrapper[5378]: W1030 20:38:29.635630 5378 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 20:38:29.659349 ip-10-0-138-149 kubenswrapper[5378]: W1030 20:38:29.635634 5378 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 20:38:29.659349 ip-10-0-138-149 kubenswrapper[5378]: W1030 20:38:29.635638 5378 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 20:38:29.659349 ip-10-0-138-149 kubenswrapper[5378]: W1030 20:38:29.635642 5378 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 20:38:29.659349 ip-10-0-138-149 kubenswrapper[5378]: W1030 20:38:29.635647 5378 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 20:38:29.659349 ip-10-0-138-149 kubenswrapper[5378]: W1030 20:38:29.635651 5378 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 20:38:29.659349 ip-10-0-138-149 kubenswrapper[5378]: W1030 20:38:29.635656 5378 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 20:38:29.659349 ip-10-0-138-149 kubenswrapper[5378]: W1030 20:38:29.635660 5378 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 20:38:29.659349 ip-10-0-138-149 kubenswrapper[5378]: W1030 20:38:29.635664 5378 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 20:38:29.659349 ip-10-0-138-149 kubenswrapper[5378]: W1030 20:38:29.635668 5378 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 20:38:29.659349 ip-10-0-138-149 kubenswrapper[5378]: W1030 20:38:29.635672 5378 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 20:38:29.659349 ip-10-0-138-149 kubenswrapper[5378]: W1030 20:38:29.635677 5378 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 20:38:29.659349 ip-10-0-138-149 kubenswrapper[5378]: W1030 20:38:29.635681 5378 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 20:38:29.659349 ip-10-0-138-149 kubenswrapper[5378]: W1030 20:38:29.635685 5378 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 20:38:29.659349 ip-10-0-138-149 kubenswrapper[5378]: W1030 20:38:29.635689 5378 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 20:38:29.660083 ip-10-0-138-149 kubenswrapper[5378]: W1030 20:38:29.635693 5378 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 20:38:29.660083 ip-10-0-138-149 kubenswrapper[5378]: W1030 20:38:29.635697 5378 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 20:38:29.660083 ip-10-0-138-149 kubenswrapper[5378]: W1030 20:38:29.635701 5378 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 20:38:29.660083 ip-10-0-138-149 kubenswrapper[5378]: W1030 20:38:29.635705 5378 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 20:38:29.660083 ip-10-0-138-149 kubenswrapper[5378]: W1030 20:38:29.635709 5378 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 20:38:29.660083 ip-10-0-138-149 kubenswrapper[5378]: W1030 20:38:29.635713 5378 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 20:38:29.660083 ip-10-0-138-149 kubenswrapper[5378]: W1030 20:38:29.635717 5378 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 20:38:29.660083 ip-10-0-138-149 kubenswrapper[5378]: W1030 20:38:29.635721 5378 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 20:38:29.660083 ip-10-0-138-149 kubenswrapper[5378]: W1030 20:38:29.635725 5378 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 20:38:29.660083 ip-10-0-138-149 kubenswrapper[5378]: W1030 20:38:29.635729 5378 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 20:38:29.660083 ip-10-0-138-149 kubenswrapper[5378]: W1030 20:38:29.635733 5378 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 20:38:29.660083 ip-10-0-138-149 kubenswrapper[5378]: W1030 20:38:29.635737 5378 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 20:38:29.660083 ip-10-0-138-149 kubenswrapper[5378]: W1030 20:38:29.635741 5378 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 20:38:29.660083 ip-10-0-138-149 kubenswrapper[5378]: W1030 20:38:29.635746 5378 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 20:38:29.660083 ip-10-0-138-149 kubenswrapper[5378]: W1030 20:38:29.635751 5378 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 20:38:29.660083 ip-10-0-138-149 kubenswrapper[5378]: W1030 20:38:29.635755 5378 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 20:38:29.660083 ip-10-0-138-149 kubenswrapper[5378]: W1030 20:38:29.635759 5378 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 20:38:29.660083 ip-10-0-138-149 kubenswrapper[5378]: W1030 20:38:29.635764 5378 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 20:38:29.660083 ip-10-0-138-149 kubenswrapper[5378]: W1030 20:38:29.635768 5378 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 20:38:29.660083 ip-10-0-138-149 kubenswrapper[5378]: W1030 20:38:29.635772 5378 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 20:38:29.660916 ip-10-0-138-149 kubenswrapper[5378]: W1030 20:38:29.635776 5378 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 20:38:29.660916 ip-10-0-138-149 kubenswrapper[5378]: W1030 20:38:29.635780 5378 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 20:38:29.660916 ip-10-0-138-149 kubenswrapper[5378]: W1030 20:38:29.635785 5378 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 20:38:29.660916 ip-10-0-138-149 kubenswrapper[5378]: W1030 20:38:29.635789 5378 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 20:38:29.660916 ip-10-0-138-149 kubenswrapper[5378]: W1030 20:38:29.635793 5378 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 20:38:29.660916 ip-10-0-138-149 kubenswrapper[5378]: W1030 20:38:29.635797 5378 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 20:38:29.660916 ip-10-0-138-149 kubenswrapper[5378]: W1030 20:38:29.635801 5378 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 20:38:29.660916 ip-10-0-138-149 kubenswrapper[5378]: W1030 20:38:29.635806 5378 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 20:38:29.660916 ip-10-0-138-149 kubenswrapper[5378]: W1030 20:38:29.635811 5378 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 20:38:29.660916 ip-10-0-138-149 kubenswrapper[5378]: W1030 20:38:29.635815 5378 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 20:38:29.660916 ip-10-0-138-149 kubenswrapper[5378]: W1030 20:38:29.635818 5378 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 20:38:29.660916 ip-10-0-138-149 kubenswrapper[5378]: W1030 20:38:29.635822 5378 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 20:38:29.660916 ip-10-0-138-149 kubenswrapper[5378]: W1030 20:38:29.635827 5378 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 20:38:29.660916 ip-10-0-138-149 kubenswrapper[5378]: W1030 20:38:29.635831 5378 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 20:38:29.660916 ip-10-0-138-149 kubenswrapper[5378]: W1030 20:38:29.635835 5378 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 20:38:29.660916 ip-10-0-138-149 kubenswrapper[5378]: W1030 20:38:29.635839 5378 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 20:38:29.660916 ip-10-0-138-149 kubenswrapper[5378]: W1030 20:38:29.635843 5378 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 20:38:29.660916 ip-10-0-138-149 kubenswrapper[5378]: W1030 20:38:29.635847 5378 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 20:38:29.660916 ip-10-0-138-149 kubenswrapper[5378]: W1030 20:38:29.635852 5378 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 20:38:29.661771 ip-10-0-138-149 kubenswrapper[5378]: W1030 20:38:29.635856 5378 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 20:38:29.661771 ip-10-0-138-149 kubenswrapper[5378]: W1030 20:38:29.635860 5378 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 20:38:29.661771 ip-10-0-138-149 kubenswrapper[5378]: W1030 20:38:29.635865 5378 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 20:38:29.661771 ip-10-0-138-149 kubenswrapper[5378]: W1030 20:38:29.635869 5378 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 20:38:29.661771 ip-10-0-138-149 kubenswrapper[5378]: W1030 20:38:29.635874 5378 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 20:38:29.661771 ip-10-0-138-149 kubenswrapper[5378]: W1030 20:38:29.635878 5378 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 20:38:29.661771 ip-10-0-138-149 kubenswrapper[5378]: W1030 20:38:29.635884 5378 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 20:38:29.661771 ip-10-0-138-149 kubenswrapper[5378]: W1030 20:38:29.635888 5378 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 20:38:29.661771 ip-10-0-138-149 kubenswrapper[5378]: W1030 20:38:29.635893 5378 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 20:38:29.661771 ip-10-0-138-149 kubenswrapper[5378]: W1030 20:38:29.635897 5378 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 20:38:29.661771 ip-10-0-138-149 kubenswrapper[5378]: W1030 20:38:29.635901 5378 feature_gate.go:386] unrecognized feature gate: Example Oct 30 20:38:29.661771 ip-10-0-138-149 kubenswrapper[5378]: W1030 20:38:29.635905 5378 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 20:38:29.661771 ip-10-0-138-149 kubenswrapper[5378]: W1030 20:38:29.635909 5378 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 20:38:29.661771 ip-10-0-138-149 kubenswrapper[5378]: W1030 20:38:29.635913 5378 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 20:38:29.661771 ip-10-0-138-149 kubenswrapper[5378]: W1030 20:38:29.635917 5378 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 20:38:29.661771 ip-10-0-138-149 kubenswrapper[5378]: W1030 20:38:29.635921 5378 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 20:38:29.661771 ip-10-0-138-149 kubenswrapper[5378]: W1030 20:38:29.635929 5378 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 20:38:29.661771 ip-10-0-138-149 kubenswrapper[5378]: W1030 20:38:29.635935 5378 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 20:38:29.661771 ip-10-0-138-149 kubenswrapper[5378]: W1030 20:38:29.635940 5378 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 20:38:29.662500 ip-10-0-138-149 kubenswrapper[5378]: W1030 20:38:29.635945 5378 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 20:38:29.662500 ip-10-0-138-149 kubenswrapper[5378]: W1030 20:38:29.635949 5378 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 20:38:29.662500 ip-10-0-138-149 kubenswrapper[5378]: W1030 20:38:29.635955 5378 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 20:38:29.662500 ip-10-0-138-149 kubenswrapper[5378]: W1030 20:38:29.635959 5378 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 20:38:29.662500 ip-10-0-138-149 kubenswrapper[5378]: W1030 20:38:29.635963 5378 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 20:38:29.662500 ip-10-0-138-149 kubenswrapper[5378]: W1030 20:38:29.635972 5378 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 20:38:29.662500 ip-10-0-138-149 kubenswrapper[5378]: W1030 20:38:29.635979 5378 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 20:38:29.662500 ip-10-0-138-149 kubenswrapper[5378]: W1030 20:38:29.635983 5378 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 20:38:29.662500 ip-10-0-138-149 kubenswrapper[5378]: W1030 20:38:29.635987 5378 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 20:38:29.662500 ip-10-0-138-149 kubenswrapper[5378]: W1030 20:38:29.635991 5378 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 20:38:29.662500 ip-10-0-138-149 kubenswrapper[5378]: W1030 20:38:29.635995 5378 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 20:38:29.662500 ip-10-0-138-149 kubenswrapper[5378]: W1030 20:38:29.635999 5378 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 20:38:29.662500 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:38:29.636084 5378 server.go:957] "Client rotation is on, will bootstrap in background" Oct 30 20:38:29.662500 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:38:29.637626 5378 bootstrap.go:85] "Current kubeconfig file contents are still valid, no bootstrap necessary" Oct 30 20:38:29.662500 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:38:29.637705 5378 certificate_store.go:147] "Loading cert/key pair from a file" filePath="/var/lib/kubelet/pki/kubelet-client-current.pem" Oct 30 20:38:29.662500 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:38:29.637968 5378 server.go:1014] "Starting client certificate rotation" Oct 30 20:38:29.662500 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:38:29.638167 5378 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Oct 30 20:38:29.663365 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:38:29.638226 5378 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-10-29 20:32:16 +0000 UTC" deadline="2028-08-14 20:26:30.240000794 +0000 UTC" Oct 30 20:38:29.663365 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:38:29.638301 5378 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="15695h48m0.601703246s" Oct 30 20:38:29.663365 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:38:29.638825 5378 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Oct 30 20:38:29.663365 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:38:29.640504 5378 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Oct 30 20:38:29.663365 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:38:29.642124 5378 log.go:25] "Validated CRI v1 runtime API" Oct 30 20:38:29.663365 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:38:29.644483 5378 log.go:25] "Validated CRI v1 image API" Oct 30 20:38:29.663365 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:38:29.645463 5378 server.go:1424] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Oct 30 20:38:29.663365 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:38:29.649205 5378 fs.go:135] Filesystem UUIDs: map[285ee138-af8f-418c-af93-c68f9ce708c2:/dev/nvme0n1p3 7B77-95E7:/dev/nvme0n1p2 8c3538b6-0559-401f-b1e6-c7c0dbc0ba6b:/dev/nvme0n1p4] Oct 30 20:38:29.663839 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:38:29.649226 5378 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:24 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:26 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/001f8ab63fe326590b9b7ea7f701d9a74e60cb5e83eaf3d6cace90bbf287965b/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/001f8ab63fe326590b9b7ea7f701d9a74e60cb5e83eaf3d6cace90bbf287965b/userdata/shm major:0 minor:295 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/10168eee9d7308abc6d1b01ecafd86974dd32d77377945c05b967dedad930adc/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/10168eee9d7308abc6d1b01ecafd86974dd32d77377945c05b967dedad930adc/userdata/shm major:0 minor:90 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/102ed47976de176fc09ed97a56fb62e603a1d5d576402ff178d3b2377f2145e4/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/102ed47976de176fc09ed97a56fb62e603a1d5d576402ff178d3b2377f2145e4/userdata/shm major:0 minor:271 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/19be35dbc9b22f4f2936dcf2085288bf8165912b6264f685ef62823add265a29/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/19be35dbc9b22f4f2936dcf2085288bf8165912b6264f685ef62823add265a29/userdata/shm major:0 minor:82 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/1afe02640a96809248834d4e5ef0671c215576bc3e9d994f39e65cbf46e658e8/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/1afe02640a96809248834d4e5ef0671c215576bc3e9d994f39e65cbf46e658e8/userdata/shm major:0 minor:94 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/220896563d049256ce2118869de7b04d501299d99a0a4a523e35d142fc735a63/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/220896563d049256ce2118869de7b04d501299d99a0a4a523e35d142fc735a63/userdata/shm major:0 minor:330 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/3ac9f12e7f98d1036fcc0c39f3aa3ab3c15fc903458208bef56f5c1969e7f9e0/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/3ac9f12e7f98d1036fcc0c39f3aa3ab3c15fc903458208bef56f5c1969e7f9e0/userdata/shm major:0 minor:48 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/3ad45e738307d25ff119e50ab25a4df89fe53b2944f14cdbaadd57c8a04859a7/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/3ad45e738307d25ff119e50ab25a4df89fe53b2944f14cdbaadd57c8a04859a7/userdata/shm major:0 minor:292 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/3e4aceef3fbbadd826485bb72e82b4999deefbb20d450e2ed6e48e9007593077/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/3e4aceef3fbbadd826485bb72e82b4999deefbb20d450e2ed6e48e9007593077/userdata/shm major:0 minor:98 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/45cc675d83bd08ae9d2107e90740daab9497b910e06df3435048aae121d8dcab/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/45cc675d83bd08ae9d2107e90740daab9497b910e06df3435048aae121d8dcab/userdata/shm major:0 minor:231 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/51320e7f03a5a3ba862c18c42484ea5b0fbe760a99611cf3cc11c780cd27bd38/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/51320e7f03a5a3ba862c18c42484ea5b0fbe760a99611cf3cc11c780cd27bd38/userdata/shm major:0 minor:86 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/5b61271db530e05551d571fbdb0978efcada0909ef41a847b1b2ee25ed94f87d/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/5b61271db530e05551d571fbdb0978efcada0909ef41a847b1b2ee25ed94f87d/userdata/shm major:0 minor:326 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/5cc6dd4df47e4cf12f0856338314d254fce6069ae6f0c51a2d22fa0cb408aad3/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/5cc6dd4df47e4cf12f0856338314d254fce6069ae6f0c51a2d22fa0cb408aad3/userdata/shm major:0 minor:328 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/88e0061ecb608b11dff851012f3eed16592fbdd5de53a886497f7b93664d46f8/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/88e0061ecb608b11dff851012f3eed16592fbdd5de53a886497f7b93664d46f8/userdata/shm major:0 minor:78 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/8e02897398b972d7732ae296827399ab18c15cdc39c3d7dddec36ba3ebb2b7dc/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/8e02897398b972d7732ae296827399ab18c15cdc39c3d7dddec36ba3ebb2b7dc/userdata/shm major:0 minor:205 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/9eae007fd35b17e1984bf02196ddc6a8933637392625a815cfb4747fde9e2909/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/9eae007fd35b17e1984bf02196ddc6a8933637392625a815cfb4747fde9e2909/userdata/shm major:0 minor:207 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/afaaaa72e718e44097a463278115e2c807138c3ff8eaddbe3af48fc8e2c92322/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/afaaaa72e718e44097a463278115e2c807138c3ff8eaddbe3af48fc8e2c92322/userdata/shm major:0 minor:68 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/b09ead73fb86a2392baeba9c87b84092951ed110487de47adab28bee7abb7826/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/b09ead73fb86a2392baeba9c87b84092951ed110487de47adab28bee7abb7826/userdata/shm major:0 minor:209 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/bc1c0047dc770cd5bcd474cbf88b4fde068c541eb9976869312899e1ea43f700/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/bc1c0047dc770cd5bcd474cbf88b4fde068c541eb9976869312899e1ea43f700/userdata/shm major:0 minor:303 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/c1c69985245a3e90dd11045025945aa9e230f6a0cddc66aa01c9131cf8ce0d3b/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/c1c69985245a3e90dd11045025945aa9e230f6a0cddc66aa01c9131cf8ce0d3b/userdata/shm major:0 minor:373 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/cc2ac291099ece3de0526ea3ae8a263d9310f204ebe3a96aa41bdfb6b10df725/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/cc2ac291099ece3de0526ea3ae8a263d9310f204ebe3a96aa41bdfb6b10df725/userdata/shm major:0 minor:72 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/da4df417dfa91620c245546301b2dbee88814652f2989ebdddb126fbdfc28a98/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/da4df417dfa91620c245546301b2dbee88814652f2989ebdddb126fbdfc28a98/userdata/shm major:0 minor:47 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:32 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/022c4cf0-9797-4428-8fa3-9f3112f2ccc5/volumes/kubernetes.io~projected/kube-api-access-qjn9k:{mountpoint:/var/lib/kubelet/pods/022c4cf0-9797-4428-8fa3-9f3112f2ccc5/volumes/kubernetes.io~projected/kube-api-access-qjn9k major:0 minor:59 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/0d6769f4-dc38-45ac-863b-c75fa672866a/volumes/kubernetes.io~secret/original-pull-secret:{mountpoint:/var/lib/kubelet/pods/0d6769f4-dc38-45ac-863b-c75fa672866a/volumes/kubernetes.io~secret/original-pull-secret major:0 minor:324 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/23527d33-4e73-4c36-ba42-7f0a5b7f62c6/volumes/kubernetes.io~empty-dir/etc-tuned:{mountpoint:/var/lib/kubelet/pods/23527d33-4e73-4c36-ba42-7f0a5b7f62c6/volumes/kubernetes.io~empty-dir/etc-tuned major:0 minor:55 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/23527d33-4e73-4c36-ba42-7f0a5b7f62c6/volumes/kubernetes.io~empty-dir/tmp:{mountpoint:/var/lib/kubelet/pods/23527d33-4e73-4c36-ba42-7f0a5b7f62c6/volumes/kubernetes.io~empty-dir/tmp major:0 minor:56 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/23527d33-4e73-4c36-ba42-7f0a5b7f62c6/volumes/kubernetes.io~projected/kube-api-access-xcs6c:{mountpoint:/var/lib/kubelet/pods/23527d33-4e73-4c36-ba42-7f0a5b7f62c6/volumes/kubernetes.io~projected/kube-api-access-xcs6c major:0 minor:61 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/3bcf59e7-befb-4623-9a0f-f01b70963b3e/volumes/kubernetes.io~projected/kube-api-access-r9qpr:{mountpoint:/var/lib/kubelet/pods/3bcf59e7-befb-4623-9a0f-f01b70963b3e/volumes/kubernetes.io~projected/kube-api-access-r9qpr major:0 minor:291 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/3bcf59e7-befb-4623-9a0f-f01b70963b3e/volumes/kubernetes.io~secret/secret-grpc-tls:{mountpoint:/var/lib/kubelet/pods/3bcf59e7-befb-4623-9a0f-f01b70963b3e/volumes/kubernetes.io~secret/secret-grpc-tls major:0 minor:290 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/3bcf59e7-befb-4623-9a0f-f01b70963b3e/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy:{mountpoint:/var/lib/kubelet/pods/3bcf59e7-befb-4623-9a0f-f01b70963b3e/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy major:0 minor:288 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/3bcf59e7-befb-4623-9a0f-f01b70963b3e/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy-metrics:{mountpoint:/var/lib/kubelet/pods/3bcf59e7-befb-4623-9a0f-f01b70963b3e/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy-metrics major:0 minor:287 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/3bcf59e7-befb-4623-9a0f-f01b70963b3e/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy-rules:{mountpoint:/var/lib/kubelet/pods/3bcf59e7-befb-4623-9a0f-f01b70963b3e/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy-rules major:0 minor:289 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/3bcf59e7-befb-4623-9a0f-f01b70963b3e/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy-web:{mountpoint:/var/lib/kubelet/pods/3bcf59e7-befb-4623-9a0f-f01b70963b3e/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy-web major:0 minor:282 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/3bcf59e7-befb-4623-9a0f-f01b70963b3e/volumes/kubernetes.io~secret/secret-thanos-querier-tls:{mountpoint:/var/lib/kubelet/pods/3bcf59e7-befb-4623-9a0f-f01b70963b3e/volumes/kubernetes.io~secret/secret-thanos-querier-tls major:0 minor:286 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/42765893-c3e3-41ab-811f-3135d4c74b07/volumes/kubernetes.io~projected/kube-api-access-wk62v:{mountpoint:/var/lib/kubelet/pods/42765893-c3e3-41ab-811f-3135d4c74b07/volumes/kubernetes.io~projected/kube-api-access-wk62v major:0 minor:302 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/42765893-c3e3-41ab-811f-3135d4c74b07/volumes/kubernetes.io~secret/monitoring-plugin-cert:{mountpoint:/var/lib/kubelet/pods/42765893-c3e3-41ab-811f-3135d4c74b07/volumes/kubernetes.io~secret/monitoring-plugin-cert major:0 minor:301 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/497c44c4-2ece-4009-b88c-6ecfa9920110/volumes/kubernetes.io~secret/agent-certs:{mountpoint:/var/lib/kubelet/pods/497c44c4-2ece-4009-b88c-6ecfa9920110/volumes/kubernetes.io~secret/agent-certs major:0 minor:63 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/5186676f-8c82-4667-a013-3efc22d2c4e7/volumes/kubernetes.io~projected/kube-api-access-9d2s8:{mountpoint:/var/lib/kubelet/pods/5186676f-8c82-4667-a013-3efc22d2c4e7/volumes/kubernetes.io~projected/kube-api-access-9d2s8 major:0 minor:230 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/5186676f-8c82-4667-a013-3efc22d2c4e7/volumes/kubernetes.io~secret/prometheus-operator-kube-rbac-proxy-config:{mountpoint:/var/lib/kubelet/pods/5186676f-8c82-4667-a013-3efc22d2c4e7/volumes/kubernetes.io~secret/prometheus-operator-kube-rbac-proxy-config major:0 minor:229 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/5186676f-8c82-4667-a013-3efc22d2c4e7/volumes/kubernetes.io~secret/prometheus-operator-tls:{mountpoint:/var/lib/kubelet/pods/5186676f-8c82-4667-a013-3efc22d2c4e7/volumes/kubernetes.io~secret/prometheus-operator-tls major:0 minor:225 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/567a72f6-035d-4c61-acaf-f46008029168/volumes/kubernetes.io~projected/kube-api-access-bc9kz:{mountpoint:/var/lib/kubelet/pods/567a72f6-035d-4c61-acaf-f46008029168/volumes/kubernetes.io~projected/kube-api-access-bc9kz major:0 minor:325 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/59046ab6-2f4c-4031-ba8c-21fb322a9d07/volumes/kubernetes.io~projected/kube-api-access-jmxp5:{mountpoint:/var/lib/kubelet/pods/59046ab6-2f4c-4031-ba8c-21fb322a9d07/volumes/kubernetes.io~projected/kube-api-access-jmxp5 major:0 minor:294 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/59406b2a-a784-4d6e-aedf-96e11ff99f68/volumes/kubernetes.io~projected/kube-api-access-fwkdk:{mountpoint:/var/lib/kubelet/pods/59406b2a-a784-4d6e-aedf-96e11ff99f68/volumes/kubernetes.io~projected/kube-api-access-fwkdk major:0 minor:204 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/59406b2a-a784-4d6e-aedf-96e11ff99f68/volumes/kubernetes.io~secret/insights-runtime-extractor-tls:{mountpoint:/var/lib/kubelet/pods/59406b2a-a784-4d6e-aedf-96e11ff99f68/volumes/kubernetes.io~secret/insights-runtime-extractor-tls major:0 minor:201 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/662e676d-c357-4ac8-a813-d34096051aff/volumes/kubernetes.io~projected/kube-api-access-k82pb:{mountpoint:/var/lib/kubelet/pods/662e676d-c357-4ac8-a813-d34096051aff/volumes/kubernetes.io~projected/kube-api-access-k82pb major:0 minor:66 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/6646b935-fe10-415a-be4b-a62a64ea45ec/volumes/kubernetes.io~projected/kube-api-access-fqlw7:{mountpoint:/var/lib/kubelet/pods/6646b935-fe10-415a-be4b-a62a64ea45ec/volumes/kubernetes.io~projected/kube-api-access-fqlw7 major:0 minor:372 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/6646b935-fe10-415a-be4b-a62a64ea45ec/volumes/kubernetes.io~secret/console-oauth-config:{mountpoint:/var/lib/kubelet/pods/6646b935-fe10-415a-be4b-a62a64ea45ec/volumes/kubernetes.io~secret/console-oauth-config major:0 minor:367 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/6646b935-fe10-415a-be4b-a62a64ea45ec/volumes/kubernetes.io~secret/console-serving-cert:{mountpoint:/var/lib/kubelet/pods/6646b935-fe10-415a-be4b-a62a64ea45ec/volumes/kubernetes.io~secret/console-serving-cert major:0 minor:371 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/666b6e91-4a60-45d9-9e3e-877867ded7fc/volumes/kubernetes.io~projected/kube-api-access-2w5c8:{mountpoint:/var/lib/kubelet/pods/666b6e91-4a60-45d9-9e3e-877867ded7fc/volumes/kubernetes.io~projected/kube-api-access-2w5c8 major:0 minor:67 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/6a710ade-5a54-4595-89a8-636653c02fc5/volumes/kubernetes.io~projected/kube-api-access-sg8pj:{mountpoint:/var/lib/kubelet/pods/6a710ade-5a54-4595-89a8-636653c02fc5/volumes/kubernetes.io~projected/kube-api-access-sg8pj major:0 minor:197 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/6a710ade-5a54-4595-89a8-636653c02fc5/volumes/kubernetes.io~secret/cert:{mountpoint:/var/lib/kubelet/pods/6a710ade-5a54-4595-89a8-636653c02fc5/volumes/kubernetes.io~secret/cert major:0 minor:45 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/743b90bd-3330-4751-a853-80b97fee0890/volumes/kubernetes.io~projected/kube-api-access-5x2gc:{mountpoint:/var/lib/kubelet/pods/743b90bd-3330-4751-a853-80b97fee0890/volumes/kubernetes.io~projected/kube-api-access-5x2gc major:0 minor:203 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/743b90bd-3330-4751-a853-80b97fee0890/volumes/kubernetes.io~secret/metrics-tls:{mountpoint:/var/lib/kubelet/pods/743b90bd-3330-4751-a853-80b97fee0890/volumes/kubernetes.io~secret/metrics-tls major:0 minor:202 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/7df66b3c-abe2-4d20-abda-45320cd4af1b/volumes/kubernetes.io~projected/kube-api-access-xxc6m:{mountpoint:/var/lib/kubelet/pods/7df66b3c-abe2-4d20-abda-45320cd4af1b/volumes/kubernetes.io~projected/kube-api-access-xxc6m major:0 minor:270 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/7df66b3c-abe2-4d20-abda-45320cd4af1b/volumes/kubernetes.io~secret/node-exporter-kube-rbac-proxy-config:{mountpoint:/var/lib/kubelet/pods/7df66b3c-abe2-4d20-abda-45320cd4af1b/volumes/kubernetes.io~secret/node-exporter-kube-rbac-proxy-config major:0 minor:265 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/7df66b3c-abe2-4d20-abda-45320cd4af1b/volumes/kubernetes.io~secret/node-exporter-tls:{mountpoint:/var/lib/kubelet/pods/7df66b3c-abe2-4d20-abda-45320cd4af1b/volumes/kubernetes.io~secret/node-exporter-tls major:0 minor:269 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/910c0d9f-c79b-4839-9a1b-f2d278d11fdb/volumes/kubernetes.io~projected/kube-api-access-qs9cm:{mountpoint:/var/lib/kubelet/pods/910c0d9f-c79b-4839-9a1b-f2d278d11fdb/volumes/kubernetes.io~projected/kube-api-access-qs9cm major:0 minor:64 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/affdfb88-e208-4ca1-843d-efc8384c27a1/volumes/kubernetes.io~projected/kube-api-access-t5psr:{mountpoint:/var/lib/kubelet/pods/affdfb88-e208-4ca1-843d-efc8384c27a1/volumes/kubernetes.io~projected/kube-api-access-t5psr major:0 minor:65 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/b7fa45b4-283a-4439-82b7-778abf2e9deb/volumes/kubernetes.io~projected/kube-api-access-crg8q:{mountpoint:/var/lib/kubelet/pods/b7fa45b4-283a-4439-82b7-778abf2e9deb/volumes/kubernetes.io~projected/kube-api-access-crg8q major:0 minor:58 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/c2ff359c-cf92-4a5b-8610-f6eb41acbe73/volume-subpaths/run-systemd/ovnkube-controller/6:{mountpoint:/var/lib/kubelet/pods/c2ff359c-cf92-4a5b-8610-f6eb41acbe73/volume-subpaths/run-systemd/ovnkube-controller/6 major:0 minor:26 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/c2ff359c-cf92-4a5b-8610-f6eb41acbe73/volumes/kubernetes.io~projected/kube-api-access-24qwx:{mountpoint:/var/lib/kubelet/pods/c2ff359c-cf92-4a5b-8610-f6eb41acbe73/volumes/kubernetes.io~projected/kube-api-access-24qwx major:0 minor:62 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/c2ff359c-cf92-4a5b-8610-f6eb41acbe73/volumes/kubernetes.io~secret/ovn-node-metrics-cert:{mountpoint:/var/lib/kubelet/pods/c2ff359c-cf92-4a5b-8610-f6eb41acbe73/volumes/kubernetes.io~secret/ovn-node-metrics-cert major:0 minor:57 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/e886897a-c6a3-414f-875f-aa112541b608/volumes/kubernetes.io~projected/kube-api-access-28zhr:{mountpoint:/var/lib/kubelet/pods/e886897a-c6a3-414f-875f-aa112541b608/volumes/kubernetes.io~projected/kube-api-access-28zhr major:0 minor:60 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/e886897a-c6a3-414f-875f-aa112541b608/volumes/kubernetes.io~secret/metrics-certs:{mountpoint:/var/lib/kubelet/pods/e886897a-c6a3-414f-875f-aa112541b608/volumes/kubernetes.io~secret/metrics-certs major:0 minor:320 fsType:tmpfs blockSize:0} composefs_0-34:{mountpoint:/ major:0 minor:34 fsType:overlay blockSize:0} overlay_0-100:{mountpoint:/var/lib/containers/storage/overlay/7c5af70b3559a684d5ad97e88c405181ae11feda7bca27bc94917bd1e2b97943/merged major:0 minor:100 fsType:overlay blockSize:0} overlay_0-102:{mountpoint:/var/lib/containers/storage/overlay/6909488d7ad34c1846138ec9523d79ecf997eb0f581a01c9a491f55845382045/merged major:0 minor:102 fsType:overlay blockSize:0} overlay_0-104:{mountpoint:/var/lib/containers/storage/overlay/c17f6aafaa7b14b1166e14f824932d1f1cbed86eb655811d4baabd16381d3637/merged major:0 minor:104 fsType:overlay blockSize:0} overlay_0-112:{mountpoint:/var/lib/containers/storage/overlay/c5fc93cb42245af9eb4f970aca42878aa9efcd441103dd4ff8ab63b529bf3ab3/merged major:0 minor:112 fsType:overlay blockSize:0} overlay_0-113:{mountpoint:/var/lib/containers/storage/overlay/268ad519078f8fba6f352aafc40c9876efcff11aaade80df786bab42f2c07e81/merged major:0 minor:113 fsType:overlay blockSize:0} overlay_0-119:{mountpoint:/var/lib/containers/storage/overlay/28ede51cc91bc603aa031c8f8c2cb478ed6b38b92d9698a1ace9820aa4a3e428/merged major:0 minor:119 fsType:overlay blockSize:0} overlay_0-124:{mountpoint:/var/lib/containers/storage/overlay/2044c7ec96d4b8b162089f481894f538dcb8243b2df95a6b03f32cad55fae5d9/merged major:0 minor:124 fsType:overlay blockSize:0} overlay_0-127:{mountpoint:/var/lib/containers/storage/overlay/17cd8886c1b8d8b66e107452118f2416f20d3209217ed09db3c0fa03719e451a/merged major:0 minor:127 fsType:overlay blockSize:0} overlay_0-132:{mountpoint:/var/lib/containers/storage/overlay/60e1b707952647ba8367b7f4e677dba3594548b88a35f87e21e2c2c0c68ce97c/merged major:0 minor:132 fsType:overlay blockSize:0} overlay_0-133:{mountpoint:/var/lib/containers/storage/overlay/95ea87b965060a1bea39045c5d652200ae1bbcadc96656c9781a12d8bfc2be17/merged major:0 minor:133 fsType:overlay blockSize:0} overlay_0-144:{mountpoint:/var/lib/containers/storage/overlay/ec4c82024aceea711c3b75f64d1d1348ceac46e5bcd4bafbfdc88de0b3cd3e29/merged major:0 minor:144 fsType:overlay blockSize:0} overlay_0-146:{mountpoint:/var/lib/containers/storage/overlay/cd4db1ecf1c0eee368a719d0f6d21c766714f877956434b19c6f2902f6b86ab9/merged major:0 minor:146 fsType:overlay blockSize:0} overlay_0-148:{mountpoint:/var/lib/containers/storage/overlay/ebd225e37e0742b4423f9888d8fc821f4bac6c2579c7499fef659e13347eab3c/merged major:0 minor:148 fsType:overlay blockSize:0} overlay_0-151:{mountpoint:/var/lib/containers/storage/overlay/b91c9fc6fa2649ab448e73bcb9f4cc903daa0c4aa1b357a455e54688041c6421/merged major:0 minor:151 fsType:overlay blockSize:0} overlay_0-162:{mountpoint:/var/lib/containers/storage/overlay/071bb22f1132082beba19d5b4e5f9d9f655b68834bcd01ec9bca2b49391cd85d/merged major:0 minor:162 fsType:overlay blockSize:0} overlay_0-167:{mountpoint:/var/lib/containers/storage/overlay/f8b20c698570e31c1e9d5c4bca7dc03137bd6e5ad2ef126c97bccff87ba2af3d/merged major:0 minor:167 fsType:overlay blockSize:0} overlay_0-172:{mountpoint:/var/lib/containers/storage/overlay/3cdae44c181ff5a296f9cdc6193a700058ca1971f3a92d525d420e90ae85f37e/merged major:0 minor:172 fsType:overlay blockSize:0} overlay_0-177:{mountpoint:/var/lib/containers/storage/overlay/361840bda2dd7482961335a93525c6f28a1ebed71d66aa382025c2c8adae22e9/merged major:0 minor:177 fsType:overlay blockSize:0} overlay_0-182:{mountpoint:/var/lib/containers/storage/overlay/317d740247b62475918d5cb1cbe268aaee14bccb473fb58921282334379641ff/merged major:0 minor:182 fsType:overlay blockSize:0} overlay_0-187:{mountpoint:/var/lib/containers/storage/overlay/d05e929b713f660cca9b1e9cdbf94679f1aed00b61115f7b7f1fb9c664c01342/merged major:0 minor:187 fsType:overlay blockSize:0} overlay_0-192:{mountpoint:/var/lib/containers/storage/overlay/1efa2450e25345b31219fe0c3283e592e3fe4ba9c68650a5842423b3883c81fc/merged major:0 minor:192 fsType:overlay blockSize:0} overlay_0-211:{mountpoint:/var/lib/containers/storage/overlay/f72a5b4b17c95fa7ec92a627f0a8b8e4a8527549e7116271fdb6cbaa5c36979e/merged major:0 minor:211 fsType:overlay blockSize:0} overlay_0-213:{mountpoint:/var/lib/containers/storage/overlay/393ff552987a725e1c8af56ff93aae9733aa9980da66eacf5c05fa80bcc16bd0/merged major:0 minor:213 fsType:overlay blockSize:0} overlay_0-215:{mountpoint:/var/lib/containers/storage/overlay/fc2eab642a5394d3395e15503df65b9ad7e21058e303ed8c06d42c17cdcaa08e/merged major:0 minor:215 fsType:overlay blockSize:0} overlay_0-217:{mountpoint:/var/lib/containers/storage/overlay/4bca771d44cefec5ad56271702814893f633d9d825baf835d1f54a3dc4f3cb3c/merged major:0 minor:217 fsType:overlay blockSize:0} overlay_0-222:{mountpoint:/var/lib/containers/storage/overlay/9a3d2ceab257f5fbb84bb6226883bf9c9b7eb4468527923de6c86de5a35f808c/merged major:0 minor:222 fsType:overlay blockSize:0} overlay_0-233:{mountpoint:/var/lib/containers/storage/overlay/5c8e17c7fad96237500e9aefe07d655157b2f7ee86bee366bfb6617575c13e2e/merged major:0 minor:233 fsType:overlay blockSize:0} overlay_0-235:{mountpoint:/var/lib/containers/storage/overlay/a221e76b01faf73daa62c404c13bb5c7e0f6b94fc4a9d3d1b374a401df173e46/merged major:0 minor:235 fsType:overlay blockSize:0} overlay_0-237:{mountpoint:/var/lib/containers/storage/overlay/c3ea911f206f030164582b1ac9c08b652fc8d9b3a649ef68d30b76038cec74c2/merged major:0 minor:237 fsType:overlay blockSize:0} overlay_0-243:{mountpoint:/var/lib/containers/storage/overlay/ef741a54bdf7cd842220a390600ecd9c96606be9c37a3d5789d744ba629fbfd8/merged major:0 minor:243 fsType:overlay blockSize:0} overlay_0-248:{mountpoint:/var/lib/containers/storage/overlay/4d61cf62697a21540a7b192b31b66d7b3cb37217af79ad052ea738825e17ed05/merged major:0 minor:248 fsType:overlay blockSize:0} overlay_0-250:{mountpoint:/var/lib/containers/storage/overlay/1b089159d72e2028839b6959e3eb132023b6bb1e279a6b6b6eeac4dbec9c1092/merged major:0 minor:250 fsType:overlay blockSize:0} overlay_0-259:{mountpoint:/var/lib/containers/storage/overlay/04136a4449047b4886053c06b976580034d15248f54954974d8bc5d746f5883f/merged major:0 minor:259 fsType:overlay blockSize:0} overlay_0-273:{mountpoint:/var/lib/containers/storage/overlay/5a4d77df58226859c2fded24ae923ac7c2b83450e263fbec150e230e9482170b/merged major:0 minor:273 fsType:overlay blockSize:0} overlay_0-275:{mountpoint:/var/lib/containers/storage/overlay/e81e11788f797e4d67ec6f08182d402174331fe64c9eb208f9a9325629dace49/merged major:0 minor:275 fsType:overlay blockSize:0} overlay_0-277:{mountpoint:/var/lib/containers/storage/overlay/0129adf0919ab3acb4924c6845b2a0cd29c6026fa9dbc856786ed7c8ae69b5b2/merged major:0 minor:277 fsType:overlay blockSize:0} overlay_0-297:{mountpoint:/var/lib/containers/storage/overlay/420fa54e5614d151b6649174158341e372b338d0cf266b311bbc453c32bd576e/merged major:0 minor:297 fsType:overlay blockSize:0} overlay_0-299:{mountpoint:/var/lib/containers/storage/overlay/3eeb7094bdf8fe03a298863db137470d803a342d28dabae0ae1cd40a502b5f6b/merged major:0 minor:299 fsType:overlay blockSize:0} overlay_0-305:{mountpoint:/var/lib/containers/storage/overlay/e58f6377451752daa31afaa35888023eb2884ccc3162509a6a6c6388667e02c0/merged major:0 minor:305 fsType:overlay blockSize:0} overlay_0-307:{mountpoint:/var/lib/containers/storage/overlay/898fce387e9485ec7c0b99ddac71caa28547039e7e9d364aea0e33b0eb328bbb/merged major:0 minor:307 fsType:overlay blockSize:0} overlay_0-309:{mountpoint:/var/lib/containers/storage/overlay/224200eb69e1f290a5d830e4b100590f3791d8d046f3141e79ed058954f2a18c/merged major:0 minor:309 fsType:overlay blockSize:0} overlay_0-315:{mountpoint:/var/lib/containers/storage/overlay/a37cc7f2a7f22634d3e93a6aa50f8b2d75ffe3098bb2b15c665a6099942f14e1/merged major:0 minor:315 fsType:overlay blockSize:0} overlay_0-332:{mountpoint:/var/lib/containers/storage/overlay/5f3b49f3f3001d4b7c81bdd2ad5f7bd6b52edc7b8cbe152b6c955aa1b42f2671/merged major:0 minor:332 fsType:overlay blockSize:0} overlay_0-334:{mountpoint:/var/lib/containers/storage/overlay/e0fd9e33afa93f8a94f2af26c0d3f3f1a6f86a361bbd23a06c9df62282bf155c/merged major:0 minor:334 fsType:overlay blockSize:0} overlay_0-336:{mountpoint:/var/lib/containers/storage/overlay/1fb04b3a712b3633cb1517df545b6b6b2fa935efcaa7aec89c3f4b160eea8a28/merged major:0 minor:336 fsType:overlay blockSize:0} overlay_0-341:{mountpoint:/var/lib/containers/storage/overlay/1fd73d00c9013f5771624dfda6cede0c28b79ebf9ce7965b9d56def21a1c4149/merged major:0 minor:341 fsType:overlay blockSize:0} overlay_0-343:{mountpoint:/var/lib/containers/storage/overlay/6af60998c7ae48acb22ac6d213ca48305c30c34d2e70d370d422bb905f988fb0/merged major:0 minor:343 fsType:overlay blockSize:0} overlay_0-344:{mountpoint:/var/lib/containers/storage/overlay/d66f78a5fc9349fc2bd3a5364e5f6270241d0af36631dde585990e64e8e75241/merged major:0 minor:344 fsType:overlay blockSize:0} overlay_0-357:{mountpoint:/var/lib/containers/storage/overlay/4d5c476c57aa1557e6ee78b5b5ce0f7f9d3553540566592c2eeb133a2ba79d3a/merged major:0 minor:357 fsType:overlay blockSize:0} overlay_0-359:{mountpoint:/var/lib/containers/storage/overlay/a42d2f670c0f10a7bd098238a2a5899d44da48d9b64298f6e953df05dc8be231/merged major:0 minor:359 fsType:overlay blockSize:0} overlay_0-361:{mountpoint:/var/lib/containers/storage/overlay/06fce77c6efa8dad214df6bcc6b11ecd0955a7d8b76a719d8942f66a0fdb569e/merged major:0 minor:361 fsType:overlay blockSize:0} overlay_0-375:{mountpoint:/var/lib/containers/storage/overlay/59e2c04c1b5a969494dd01536d49b737804c6ba44c5475608c450de9f430dc73/merged major:0 minor:375 fsType:overlay blockSize:0} overlay_0-377:{mountpoint:/var/lib/containers/storage/overlay/98acd4c3185e71285f988800383a40e29fa76c2689f4bb980365fd252e2ca119/merged major:0 minor:377 fsType:overlay blockSize:0} overlay_0-383:{mountpoint:/var/lib/containers/storage/overlay/9f820a2278c2e5b60d1f6b690dbd98b0ac17891eca73dd9d97da856c06f17f26/merged major:0 minor:383 fsType:overlay blockSize:0} overlay_0-51:{mountpoint:/var/lib/containers/storage/overlay/743770c11468f3bc2ceb4e389685e4d61ace4abe7ec2f2f27f5d71d682c9b0e2/merged major:0 minor:51 fsType:overlay blockSize:0} overlay_0-53:{mountpoint:/var/lib/containers/storage/overlay/0efe011a051bc980f17106c60d609cd7273e39aa8b5d7e69881eb5ad7c2ffbf2/merged major:0 minor:53 fsType:overlay blockSize:0} overlay_0-70:{mountpoint:/var/lib/containers/storage/overlay/f833bf1de9d252547c6e16e3bbf00a73c0fba5a99e8aa95009ce1fa75395424e/merged major:0 minor:70 fsType:overlay blockSize:0} overlay_0-74:{mountpoint:/var/lib/containers/storage/overlay/bbcd92bd50b33f87e2bc5fdda7fe64a5d22af92cfdf19788581dbe9982c3e92b/merged major:0 minor:74 fsType:overlay blockSize:0} overlay_0-76:{mountpoint:/var/lib/containers/storage/overlay/33768082fdbe409bb73c9c735be5ca2649f3e883aa05a18daf6be5af933c6260/merged major:0 minor:76 fsType:overlay blockSize:0} overlay_0-80:{mountpoint:/var/lib/containers/storage/overlay/bcb238347c430022bc01a3ca20cd6923a6f1e4403763534c4e28798838724392/merged major:0 minor:80 fsType:overlay blockSize:0} overlay_0-84:{mountpoint:/var/lib/containers/storage/overlay/e2c6385ec1d28439371d8d875cd08f414f9997f6563af8d086593d1a866e357b/merged major:0 minor:84 fsType:overlay blockSize:0} overlay_0-88:{mountpoint:/var/lib/containers/storage/overlay/7e72377b98e3f6168c601aff78caca72761cad02dc111f26e1433d4454d8e691/merged major:0 minor:88 fsType:overlay blockSize:0} overlay_0-92:{mountpoint:/var/lib/containers/storage/overlay/cb91812f023eb252eabee943f016a623d0a780e231e91561f97f731e29869392/merged major:0 minor:92 fsType:overlay blockSize:0} overlay_0-96:{mountpoint:/var/lib/containers/storage/overlay/c44beb76c9ba0ceb1956cb8af3c27e9a4f93eb6ca45575a74474b3900df879a3/merged major:0 minor:96 fsType:overlay blockSize:0}] Oct 30 20:38:29.664197 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:38:29.658902 5378 manager.go:217] Machine: {Timestamp:2026-10-30 20:38:29.658013637 +0000 UTC m=+0.086127550 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3099947 MemoryCapacity:33164341248 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2c7df77bcd86752b2f17ccecc031b6 SystemUUID:ec2c7df7-7bcd-8675-2b2f-17ccecc031b6 BootID:e1c502c8-2a55-4c24-b41a-b750355263d4 Filesystems:[{Device:/var/lib/kubelet/pods/b7fa45b4-283a-4439-82b7-778abf2e9deb/volumes/kubernetes.io~projected/kube-api-access-crg8q DeviceMajor:0 DeviceMinor:58 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/88e0061ecb608b11dff851012f3eed16592fbdd5de53a886497f7b93664d46f8/userdata/shm DeviceMajor:0 DeviceMinor:78 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/3bcf59e7-befb-4623-9a0f-f01b70963b3e/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy-web DeviceMajor:0 DeviceMinor:282 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/3ad45e738307d25ff119e50ab25a4df89fe53b2944f14cdbaadd57c8a04859a7/userdata/shm DeviceMajor:0 DeviceMinor:292 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-305 DeviceMajor:0 DeviceMinor:305 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/e886897a-c6a3-414f-875f-aa112541b608/volumes/kubernetes.io~secret/metrics-certs DeviceMajor:0 DeviceMinor:320 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-375 DeviceMajor:0 DeviceMinor:375 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-377 DeviceMajor:0 DeviceMinor:377 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-112 DeviceMajor:0 DeviceMinor:112 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/42765893-c3e3-41ab-811f-3135d4c74b07/volumes/kubernetes.io~secret/monitoring-plugin-cert DeviceMajor:0 DeviceMinor:301 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-361 DeviceMajor:0 DeviceMinor:361 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/6646b935-fe10-415a-be4b-a62a64ea45ec/volumes/kubernetes.io~projected/kube-api-access-fqlw7 DeviceMajor:0 DeviceMinor:372 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:32 Capacity:16582172672 Type:vfs Inodes:1048576 HasInodes:true} {Device:overlay_0-51 DeviceMajor:0 DeviceMinor:51 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/affdfb88-e208-4ca1-843d-efc8384c27a1/volumes/kubernetes.io~projected/kube-api-access-t5psr DeviceMajor:0 DeviceMinor:65 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-127 DeviceMajor:0 DeviceMinor:127 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-76 DeviceMajor:0 DeviceMinor:76 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/6a710ade-5a54-4595-89a8-636653c02fc5/volumes/kubernetes.io~secret/cert DeviceMajor:0 DeviceMinor:45 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-235 DeviceMajor:0 DeviceMinor:235 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-275 DeviceMajor:0 DeviceMinor:275 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/3bcf59e7-befb-4623-9a0f-f01b70963b3e/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy-rules DeviceMajor:0 DeviceMinor:289 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-113 DeviceMajor:0 DeviceMinor:113 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/3bcf59e7-befb-4623-9a0f-f01b70963b3e/volumes/kubernetes.io~secret/secret-grpc-tls DeviceMajor:0 DeviceMinor:290 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-315 DeviceMajor:0 DeviceMinor:315 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/220896563d049256ce2118869de7b04d501299d99a0a4a523e35d142fc735a63/userdata/shm DeviceMajor:0 DeviceMinor:330 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-80 DeviceMajor:0 DeviceMinor:80 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/3bcf59e7-befb-4623-9a0f-f01b70963b3e/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy DeviceMajor:0 DeviceMinor:288 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/da4df417dfa91620c245546301b2dbee88814652f2989ebdddb126fbdfc28a98/userdata/shm DeviceMajor:0 DeviceMinor:47 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-132 DeviceMajor:0 DeviceMinor:132 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-162 DeviceMajor:0 DeviceMinor:162 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/5b61271db530e05551d571fbdb0978efcada0909ef41a847b1b2ee25ed94f87d/userdata/shm DeviceMajor:0 DeviceMinor:326 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/6646b935-fe10-415a-be4b-a62a64ea45ec/volumes/kubernetes.io~secret/console-oauth-config DeviceMajor:0 DeviceMinor:367 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:26 Capacity:6632869888 Type:vfs Inodes:819200 HasInodes:true} {Device:/var/lib/kubelet/pods/23527d33-4e73-4c36-ba42-7f0a5b7f62c6/volumes/kubernetes.io~empty-dir/tmp DeviceMajor:0 DeviceMinor:56 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-192 DeviceMajor:0 DeviceMinor:192 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/b09ead73fb86a2392baeba9c87b84092951ed110487de47adab28bee7abb7826/userdata/shm DeviceMajor:0 DeviceMinor:209 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-211 DeviceMajor:0 DeviceMinor:211 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/7df66b3c-abe2-4d20-abda-45320cd4af1b/volumes/kubernetes.io~projected/kube-api-access-xxc6m DeviceMajor:0 DeviceMinor:270 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/567a72f6-035d-4c61-acaf-f46008029168/volumes/kubernetes.io~projected/kube-api-access-bc9kz DeviceMajor:0 DeviceMinor:325 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-341 DeviceMajor:0 DeviceMinor:341 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-144 DeviceMajor:0 DeviceMinor:144 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-151 DeviceMajor:0 DeviceMinor:151 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-177 DeviceMajor:0 DeviceMinor:177 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-215 DeviceMajor:0 DeviceMinor:215 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-74 DeviceMajor:0 DeviceMinor:74 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/run/containers/storage/overlay-containers/10168eee9d7308abc6d1b01ecafd86974dd32d77377945c05b967dedad930adc/userdata/shm DeviceMajor:0 DeviceMinor:90 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-119 DeviceMajor:0 DeviceMinor:119 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-182 DeviceMajor:0 DeviceMinor:182 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/8e02897398b972d7732ae296827399ab18c15cdc39c3d7dddec36ba3ebb2b7dc/userdata/shm DeviceMajor:0 DeviceMinor:205 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-237 DeviceMajor:0 DeviceMinor:237 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/3bcf59e7-befb-4623-9a0f-f01b70963b3e/volumes/kubernetes.io~projected/kube-api-access-r9qpr DeviceMajor:0 DeviceMinor:291 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-259 DeviceMajor:0 DeviceMinor:259 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:24 Capacity:16582168576 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/afaaaa72e718e44097a463278115e2c807138c3ff8eaddbe3af48fc8e2c92322/userdata/shm DeviceMajor:0 DeviceMinor:68 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-243 DeviceMajor:0 DeviceMinor:243 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/bc1c0047dc770cd5bcd474cbf88b4fde068c541eb9976869312899e1ea43f700/userdata/shm DeviceMajor:0 DeviceMinor:303 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-309 DeviceMajor:0 DeviceMinor:309 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-124 DeviceMajor:0 DeviceMinor:124 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-332 DeviceMajor:0 DeviceMinor:332 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-53 DeviceMajor:0 DeviceMinor:53 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/497c44c4-2ece-4009-b88c-6ecfa9920110/volumes/kubernetes.io~secret/agent-certs DeviceMajor:0 DeviceMinor:63 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/3bcf59e7-befb-4623-9a0f-f01b70963b3e/volumes/kubernetes.io~secret/secret-thanos-querier-tls DeviceMajor:0 DeviceMinor:286 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/59046ab6-2f4c-4031-ba8c-21fb322a9d07/volumes/kubernetes.io~projected/kube-api-access-jmxp5 DeviceMajor:0 DeviceMinor:294 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-70 DeviceMajor:0 DeviceMinor:70 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-307 DeviceMajor:0 DeviceMinor:307 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/666b6e91-4a60-45d9-9e3e-877867ded7fc/volumes/kubernetes.io~projected/kube-api-access-2w5c8 DeviceMajor:0 DeviceMinor:67 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-102 DeviceMajor:0 DeviceMinor:102 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/59406b2a-a784-4d6e-aedf-96e11ff99f68/volumes/kubernetes.io~secret/insights-runtime-extractor-tls DeviceMajor:0 DeviceMinor:201 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-217 DeviceMajor:0 DeviceMinor:217 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-250 DeviceMajor:0 DeviceMinor:250 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/0d6769f4-dc38-45ac-863b-c75fa672866a/volumes/kubernetes.io~secret/original-pull-secret DeviceMajor:0 DeviceMinor:324 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/6646b935-fe10-415a-be4b-a62a64ea45ec/volumes/kubernetes.io~secret/console-serving-cert DeviceMajor:0 DeviceMinor:371 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-84 DeviceMajor:0 DeviceMinor:84 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-187 DeviceMajor:0 DeviceMinor:187 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/7df66b3c-abe2-4d20-abda-45320cd4af1b/volumes/kubernetes.io~secret/node-exporter-tls DeviceMajor:0 DeviceMinor:269 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/42765893-c3e3-41ab-811f-3135d4c74b07/volumes/kubernetes.io~projected/kube-api-access-wk62v DeviceMajor:0 DeviceMinor:302 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/e886897a-c6a3-414f-875f-aa112541b608/volumes/kubernetes.io~projected/kube-api-access-28zhr DeviceMajor:0 DeviceMinor:60 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/5cc6dd4df47e4cf12f0856338314d254fce6069ae6f0c51a2d22fa0cb408aad3/userdata/shm DeviceMajor:0 DeviceMinor:328 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-383 DeviceMajor:0 DeviceMinor:383 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/743b90bd-3330-4751-a853-80b97fee0890/volumes/kubernetes.io~projected/kube-api-access-5x2gc DeviceMajor:0 DeviceMinor:203 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/5186676f-8c82-4667-a013-3efc22d2c4e7/volumes/kubernetes.io~secret/prometheus-operator-kube-rbac-proxy-config DeviceMajor:0 DeviceMinor:229 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/45cc675d83bd08ae9d2107e90740daab9497b910e06df3435048aae121d8dcab/userdata/shm DeviceMajor:0 DeviceMinor:231 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-344 DeviceMajor:0 DeviceMinor:344 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-357 DeviceMajor:0 DeviceMinor:357 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-92 DeviceMajor:0 DeviceMinor:92 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/59406b2a-a784-4d6e-aedf-96e11ff99f68/volumes/kubernetes.io~projected/kube-api-access-fwkdk DeviceMajor:0 DeviceMinor:204 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-222 DeviceMajor:0 DeviceMinor:222 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-233 DeviceMajor:0 DeviceMinor:233 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/102ed47976de176fc09ed97a56fb62e603a1d5d576402ff178d3b2377f2145e4/userdata/shm DeviceMajor:0 DeviceMinor:271 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-273 DeviceMajor:0 DeviceMinor:273 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/c2ff359c-cf92-4a5b-8610-f6eb41acbe73/volume-subpaths/run-systemd/ovnkube-controller/6 DeviceMajor:0 DeviceMinor:26 Capacity:6632869888 Type:vfs Inodes:819200 HasInodes:true} {Device:overlay_0-343 DeviceMajor:0 DeviceMinor:343 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/23527d33-4e73-4c36-ba42-7f0a5b7f62c6/volumes/kubernetes.io~empty-dir/etc-tuned DeviceMajor:0 DeviceMinor:55 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/910c0d9f-c79b-4839-9a1b-f2d278d11fdb/volumes/kubernetes.io~projected/kube-api-access-qs9cm DeviceMajor:0 DeviceMinor:64 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/19be35dbc9b22f4f2936dcf2085288bf8165912b6264f685ef62823add265a29/userdata/shm DeviceMajor:0 DeviceMinor:82 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/51320e7f03a5a3ba862c18c42484ea5b0fbe760a99611cf3cc11c780cd27bd38/userdata/shm DeviceMajor:0 DeviceMinor:86 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-88 DeviceMajor:0 DeviceMinor:88 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/7df66b3c-abe2-4d20-abda-45320cd4af1b/volumes/kubernetes.io~secret/node-exporter-kube-rbac-proxy-config DeviceMajor:0 DeviceMinor:265 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-359 DeviceMajor:0 DeviceMinor:359 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-167 DeviceMajor:0 DeviceMinor:167 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/c2ff359c-cf92-4a5b-8610-f6eb41acbe73/volumes/kubernetes.io~secret/ovn-node-metrics-cert DeviceMajor:0 DeviceMinor:57 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/022c4cf0-9797-4428-8fa3-9f3112f2ccc5/volumes/kubernetes.io~projected/kube-api-access-qjn9k DeviceMajor:0 DeviceMinor:59 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/1afe02640a96809248834d4e5ef0671c215576bc3e9d994f39e65cbf46e658e8/userdata/shm DeviceMajor:0 DeviceMinor:94 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/3e4aceef3fbbadd826485bb72e82b4999deefbb20d450e2ed6e48e9007593077/userdata/shm DeviceMajor:0 DeviceMinor:98 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-336 DeviceMajor:0 DeviceMinor:336 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-34 DeviceMajor:0 DeviceMinor:34 Capacity:6422528 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/var/lib/kubelet/pods/c2ff359c-cf92-4a5b-8610-f6eb41acbe73/volumes/kubernetes.io~projected/kube-api-access-24qwx DeviceMajor:0 DeviceMinor:62 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/662e676d-c357-4ac8-a813-d34096051aff/volumes/kubernetes.io~projected/kube-api-access-k82pb DeviceMajor:0 DeviceMinor:66 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-104 DeviceMajor:0 DeviceMinor:104 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-146 DeviceMajor:0 DeviceMinor:146 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/001f8ab63fe326590b9b7ea7f701d9a74e60cb5e83eaf3d6cace90bbf287965b/userdata/shm DeviceMajor:0 DeviceMinor:295 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/6a710ade-5a54-4595-89a8-636653c02fc5/volumes/kubernetes.io~projected/kube-api-access-sg8pj DeviceMajor:0 DeviceMinor:197 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/743b90bd-3330-4751-a853-80b97fee0890/volumes/kubernetes.io~secret/metrics-tls DeviceMajor:0 DeviceMinor:202 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/5186676f-8c82-4667-a013-3efc22d2c4e7/volumes/kubernetes.io~secret/prometheus-operator-tls DeviceMajor:0 DeviceMinor:225 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/3bcf59e7-befb-4623-9a0f-f01b70963b3e/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy-metrics DeviceMajor:0 DeviceMinor:287 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/cc2ac291099ece3de0526ea3ae8a263d9310f204ebe3a96aa41bdfb6b10df725/userdata/shm DeviceMajor:0 DeviceMinor:72 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-297 DeviceMajor:0 DeviceMinor:297 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-334 DeviceMajor:0 DeviceMinor:334 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/23527d33-4e73-4c36-ba42-7f0a5b7f62c6/volumes/kubernetes.io~projected/kube-api-access-xcs6c DeviceMajor:0 DeviceMinor:61 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-100 DeviceMajor:0 DeviceMinor:100 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-96 DeviceMajor:0 DeviceMinor:96 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-213 DeviceMajor:0 DeviceMinor:213 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/5186676f-8c82-4667-a013-3efc22d2c4e7/volumes/kubernetes.io~projected/kube-api-access-9d2s8 DeviceMajor:0 DeviceMinor:230 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-248 DeviceMajor:0 DeviceMinor:248 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-277 DeviceMajor:0 DeviceMinor:277 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-299 DeviceMajor:0 DeviceMinor:299 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-148 DeviceMajor:0 DeviceMinor:148 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/3ac9f12e7f98d1036fcc0c39f3aa3ab3c15fc903458208bef56f5c1969e7f9e0/userdata/shm DeviceMajor:0 DeviceMinor:48 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-133 DeviceMajor:0 DeviceMinor:133 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/c1c69985245a3e90dd11045025945aa9e230f6a0cddc66aa01c9131cf8ce0d3b/userdata/shm DeviceMajor:0 DeviceMinor:373 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-172 DeviceMajor:0 DeviceMinor:172 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/9eae007fd35b17e1984bf02196ddc6a8933637392625a815cfb4747fde9e2909/userdata/shm DeviceMajor:0 DeviceMinor:207 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:001f8ab63fe3265 MacAddress:a6:8d:89:9f:12:bf Speed:10000 Mtu:8901} {Name:220896563d04925 MacAddress:62:41:4c:d3:98:16 Speed:10000 Mtu:8901} {Name:3ad45e738307d25 MacAddress:56:12:5e:ad:dc:f8 Speed:10000 Mtu:8901} {Name:45cc675d83bd08a MacAddress:42:c6:16:59:8d:be Speed:10000 Mtu:8901} {Name:5b61271db530e05 MacAddress:e6:ba:93:ac:bf:5a Speed:10000 Mtu:8901} {Name:5cc6dd4df47e4cf MacAddress:72:d7:ca:af:ff:68 Speed:10000 Mtu:8901} {Name:8e02897398b972d MacAddress:a6:b6:20:70:57:8e Speed:10000 Mtu:8901} {Name:9eae007fd35b17e MacAddress:32:7b:39:54:cd:7c Speed:10000 Mtu:8901} {Name:b09ead73fb86a23 MacAddress:a2:12:ff:22:1c:25 Speed:10000 Mtu:8901} {Name:bc1c0047dc770cd MacAddress:46:48:97:f3:32:b0 Speed:10000 Mtu:8901} {Name:br-ex MacAddress:02:3f:92:bd:4e:f1 Speed:0 Mtu:9001} {Name:br-int MacAddress:ea:94:39:04:99:18 Speed:0 Mtu:8901} {Name:c1c69985245a3e9 MacAddress:d2:b1:90:1b:b1:83 Speed:10000 Mtu:8901} {Name:ens5 MacAddress:02:3f:92:bd:4e:f1 Speed:0 Mtu:9001} {Name:genev_sys_6081 MacAddress:36:03:cc:13:33:f7 Speed:0 Mtu:65000} {Name:ovn-k8s-mp0 MacAddress:0a:58:0a:85:00:02 Speed:0 Mtu:8901} {Name:ovs-system MacAddress:da:25:e1:58:1d:73 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164341248 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Oct 30 20:38:29.664197 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:38:29.659471 5378 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Oct 30 20:38:29.664197 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:38:29.659531 5378 manager.go:233] Version: {KernelVersion:5.14.0-687.60.1.el9_8.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.8.20261023-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Oct 30 20:38:29.664197 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:38:29.659926 5378 container_manager_linux.go:275] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Oct 30 20:38:29.664197 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:38:29.659963 5378 container_manager_linux.go:280] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-138-149.us-west-1.compute.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"/system.slice","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{},"system-reserved-compressible":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Oct 30 20:38:29.664197 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:38:29.660150 5378 topology_manager.go:143] "Creating topology manager with none policy" Oct 30 20:38:29.664197 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:38:29.660164 5378 container_manager_linux.go:311] "Creating device plugin manager" Oct 30 20:38:29.664197 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:38:29.660178 5378 manager.go:144] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Oct 30 20:38:29.664197 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:38:29.660206 5378 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Oct 30 20:38:29.664197 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:38:29.660314 5378 container_manager_linux.go:320] "Creating Dynamic Resource Allocation (DRA) manager" Oct 30 20:38:29.664197 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:38:29.660525 5378 state_mem.go:41] "Initialized" logger="CPUManager state memory" Oct 30 20:38:29.664197 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:38:29.660677 5378 server.go:1235] "Using root directory" path="/var/lib/kubelet" Oct 30 20:38:29.664197 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:38:29.660723 5378 kubelet.go:493] "Attempting to sync node with API server" Oct 30 20:38:29.664197 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:38:29.660737 5378 kubelet.go:394] "Adding static pod path" path="/etc/kubernetes/manifests" Oct 30 20:38:29.664197 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:38:29.660752 5378 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Oct 30 20:38:29.664197 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:38:29.660765 5378 kubelet.go:405] "Adding apiserver pod source" Oct 30 20:38:29.664197 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:38:29.660776 5378 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Oct 30 20:38:29.664197 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:38:29.661662 5378 state_checkpoint.go:81] "State checkpoint: restored pod resource state from checkpoint" Oct 30 20:38:29.664197 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:38:29.661675 5378 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Oct 30 20:38:29.664197 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:38:29.662212 5378 kuberuntime_manager.go:304] "Container runtime initialized" containerRuntime="cri-o" version="1.35.10-3.rhaos4.22.git9383502.el9" apiVersion="v1" Oct 30 20:38:29.675761 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:38:29.675740 5378 state_checkpoint.go:81] "State checkpoint: restored pod resource state from checkpoint" Oct 30 20:38:29.675761 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:38:29.675757 5378 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Oct 30 20:38:29.675903 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:38:29.675879 5378 certificate_store.go:147] "Loading cert/key pair from a file" filePath="/var/lib/kubelet/pki/kubelet-server-current.pem" Oct 30 20:38:29.676111 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:38:29.676095 5378 kubelet.go:963] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Oct 30 20:38:29.676177 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:38:29.676135 5378 kubelet.go:990] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Oct 30 20:38:29.676262 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:38:29.676251 5378 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Oct 30 20:38:29.676324 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:38:29.676271 5378 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Oct 30 20:38:29.676324 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:38:29.676282 5378 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Oct 30 20:38:29.676324 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:38:29.676290 5378 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Oct 30 20:38:29.676324 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:38:29.676301 5378 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Oct 30 20:38:29.676324 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:38:29.676310 5378 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Oct 30 20:38:29.676324 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:38:29.676321 5378 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Oct 30 20:38:29.676603 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:38:29.676330 5378 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Oct 30 20:38:29.676603 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:38:29.676341 5378 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Oct 30 20:38:29.676603 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:38:29.676350 5378 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Oct 30 20:38:29.676603 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:38:29.676364 5378 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Oct 30 20:38:29.676603 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:38:29.676378 5378 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Oct 30 20:38:29.676603 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:38:29.676418 5378 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Oct 30 20:38:29.676603 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:38:29.676428 5378 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Oct 30 20:38:29.679990 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:38:29.679973 5378 server.go:1263] "Started kubelet" Oct 30 20:38:29.680414 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:38:29.680383 5378 server.go:182] "Starting to listen" address="0.0.0.0" port=10250 Oct 30 20:38:29.680441 ip-10-0-138-149 systemd[1]: Started Kubernetes Kubelet. Oct 30 20:38:29.681467 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:38:29.681365 5378 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Oct 30 20:38:29.681467 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:38:29.681430 5378 server_v1.go:49] "podresources" method="list" useActivePods=true Oct 30 20:38:29.684823 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:38:29.682356 5378 server.go:254] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Oct 30 20:38:29.689675 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:38:29.689477 5378 fs_resource_analyzer.go:69] "Starting FS ResourceAnalyzer" Oct 30 20:38:29.689749 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:38:29.689719 5378 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Oct 30 20:38:29.689843 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:38:29.689802 5378 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-10-29 20:32:18 +0000 UTC" deadline="2028-08-16 01:17:06.758839792 +0000 UTC" Oct 30 20:38:29.689918 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:38:29.689843 5378 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="15724h38m37.06900003s" Oct 30 20:38:29.691596 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:38:29.690001 5378 volume_manager.go:309] "The desired_state_of_world populator starts" Oct 30 20:38:29.691596 ip-10-0-138-149 kubenswrapper[5378]: E1030 20:38:29.690007 5378 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-138-149.us-west-1.compute.internal\" not found" Oct 30 20:38:29.691596 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:38:29.690014 5378 volume_manager.go:311] "Starting Kubelet Volume Manager" Oct 30 20:38:29.691596 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:38:29.690024 5378 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Oct 30 20:38:29.691596 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:38:29.690534 5378 server.go:317] "Adding debug handlers to kubelet server" Oct 30 20:38:29.692411 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:38:29.692393 5378 factory.go:55] Registering systemd factory Oct 30 20:38:29.692494 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:38:29.692414 5378 factory.go:223] Registration of the systemd container factory successfully Oct 30 20:38:29.692954 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:38:29.692938 5378 factory.go:153] Registering CRI-O factory Oct 30 20:38:29.693037 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:38:29.692958 5378 factory.go:223] Registration of the crio container factory successfully Oct 30 20:38:29.693037 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:38:29.693013 5378 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Oct 30 20:38:29.693037 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:38:29.693033 5378 factory.go:103] Registering Raw factory Oct 30 20:38:29.693179 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:38:29.693048 5378 manager.go:1199] Started watching for new ooms in manager Oct 30 20:38:29.693247 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:38:29.693192 5378 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3bcf59e7-befb-4623-9a0f-f01b70963b3e" volumeName="kubernetes.io/configmap/3bcf59e7-befb-4623-9a0f-f01b70963b3e-metrics-client-ca" seLinuxMountContext="" Oct 30 20:38:29.693307 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:38:29.693246 5378 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="6646b935-fe10-415a-be4b-a62a64ea45ec" volumeName="kubernetes.io/configmap/6646b935-fe10-415a-be4b-a62a64ea45ec-service-ca" seLinuxMountContext="" Oct 30 20:38:29.693307 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:38:29.693260 5378 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7df66b3c-abe2-4d20-abda-45320cd4af1b" volumeName="kubernetes.io/secret/7df66b3c-abe2-4d20-abda-45320cd4af1b-node-exporter-tls" seLinuxMountContext="" Oct 30 20:38:29.693307 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:38:29.693273 5378 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="910c0d9f-c79b-4839-9a1b-f2d278d11fdb" volumeName="kubernetes.io/configmap/910c0d9f-c79b-4839-9a1b-f2d278d11fdb-whereabouts-flatfile-configmap" seLinuxMountContext="" Oct 30 20:38:29.693307 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:38:29.693285 5378 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="910c0d9f-c79b-4839-9a1b-f2d278d11fdb" volumeName="kubernetes.io/configmap/910c0d9f-c79b-4839-9a1b-f2d278d11fdb-cni-sysctl-allowlist" seLinuxMountContext="" Oct 30 20:38:29.693307 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:38:29.693296 5378 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3bcf59e7-befb-4623-9a0f-f01b70963b3e" volumeName="kubernetes.io/secret/3bcf59e7-befb-4623-9a0f-f01b70963b3e-secret-thanos-querier-kube-rbac-proxy-metrics" seLinuxMountContext="" Oct 30 20:38:29.693307 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:38:29.693307 5378 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="59046ab6-2f4c-4031-ba8c-21fb322a9d07" volumeName="kubernetes.io/empty-dir/59046ab6-2f4c-4031-ba8c-21fb322a9d07-tmp" seLinuxMountContext="" Oct 30 20:38:29.693611 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:38:29.693319 5378 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="59046ab6-2f4c-4031-ba8c-21fb322a9d07" volumeName="kubernetes.io/projected/59046ab6-2f4c-4031-ba8c-21fb322a9d07-kube-api-access-jmxp5" seLinuxMountContext="" Oct 30 20:38:29.693611 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:38:29.693333 5378 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="6646b935-fe10-415a-be4b-a62a64ea45ec" volumeName="kubernetes.io/configmap/6646b935-fe10-415a-be4b-a62a64ea45ec-console-config" seLinuxMountContext="" Oct 30 20:38:29.693611 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:38:29.693391 5378 manager.go:319] Starting recovery of all containers Oct 30 20:38:29.695368 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:38:29.693939 5378 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="743b90bd-3330-4751-a853-80b97fee0890" volumeName="kubernetes.io/secret/743b90bd-3330-4751-a853-80b97fee0890-metrics-tls" seLinuxMountContext="" Oct 30 20:38:29.695368 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:38:29.693970 5378 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3bcf59e7-befb-4623-9a0f-f01b70963b3e" volumeName="kubernetes.io/secret/3bcf59e7-befb-4623-9a0f-f01b70963b3e-secret-thanos-querier-kube-rbac-proxy" seLinuxMountContext="" Oct 30 20:38:29.695368 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:38:29.693984 5378 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="23527d33-4e73-4c36-ba42-7f0a5b7f62c6" volumeName="kubernetes.io/projected/23527d33-4e73-4c36-ba42-7f0a5b7f62c6-kube-api-access-xcs6c" seLinuxMountContext="" Oct 30 20:38:29.695368 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:38:29.694020 5378 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="59406b2a-a784-4d6e-aedf-96e11ff99f68" volumeName="kubernetes.io/empty-dir/59406b2a-a784-4d6e-aedf-96e11ff99f68-data-volume" seLinuxMountContext="" Oct 30 20:38:29.695368 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:38:29.694046 5378 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="59406b2a-a784-4d6e-aedf-96e11ff99f68" volumeName="kubernetes.io/projected/59406b2a-a784-4d6e-aedf-96e11ff99f68-kube-api-access-fwkdk" seLinuxMountContext="" Oct 30 20:38:29.695368 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:38:29.694065 5378 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="6646b935-fe10-415a-be4b-a62a64ea45ec" volumeName="kubernetes.io/configmap/6646b935-fe10-415a-be4b-a62a64ea45ec-oauth-serving-cert" seLinuxMountContext="" Oct 30 20:38:29.695368 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:38:29.694079 5378 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c2ff359c-cf92-4a5b-8610-f6eb41acbe73" volumeName="kubernetes.io/configmap/c2ff359c-cf92-4a5b-8610-f6eb41acbe73-ovnkube-config" seLinuxMountContext="" Oct 30 20:38:29.695368 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:38:29.694108 5378 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="497c44c4-2ece-4009-b88c-6ecfa9920110" volumeName="kubernetes.io/secret/497c44c4-2ece-4009-b88c-6ecfa9920110-agent-certs" seLinuxMountContext="" Oct 30 20:38:29.695368 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:38:29.694119 5378 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3bcf59e7-befb-4623-9a0f-f01b70963b3e" volumeName="kubernetes.io/secret/3bcf59e7-befb-4623-9a0f-f01b70963b3e-secret-thanos-querier-kube-rbac-proxy-rules" seLinuxMountContext="" Oct 30 20:38:29.695368 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:38:29.694130 5378 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5186676f-8c82-4667-a013-3efc22d2c4e7" volumeName="kubernetes.io/secret/5186676f-8c82-4667-a013-3efc22d2c4e7-prometheus-operator-kube-rbac-proxy-config" seLinuxMountContext="" Oct 30 20:38:29.695368 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:38:29.694159 5378 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="6646b935-fe10-415a-be4b-a62a64ea45ec" volumeName="kubernetes.io/empty-dir/6646b935-fe10-415a-be4b-a62a64ea45ec-tmp" seLinuxMountContext="" Oct 30 20:38:29.695368 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:38:29.694173 5378 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c2ff359c-cf92-4a5b-8610-f6eb41acbe73" volumeName="kubernetes.io/projected/c2ff359c-cf92-4a5b-8610-f6eb41acbe73-kube-api-access-24qwx" seLinuxMountContext="" Oct 30 20:38:29.695368 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:38:29.694186 5378 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3bcf59e7-befb-4623-9a0f-f01b70963b3e" volumeName="kubernetes.io/empty-dir/3bcf59e7-befb-4623-9a0f-f01b70963b3e-tmp" seLinuxMountContext="" Oct 30 20:38:29.695368 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:38:29.694197 5378 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="910c0d9f-c79b-4839-9a1b-f2d278d11fdb" volumeName="kubernetes.io/configmap/910c0d9f-c79b-4839-9a1b-f2d278d11fdb-cni-binary-copy" seLinuxMountContext="" Oct 30 20:38:29.695368 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:38:29.694210 5378 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="567a72f6-035d-4c61-acaf-f46008029168" volumeName="kubernetes.io/projected/567a72f6-035d-4c61-acaf-f46008029168-kube-api-access-bc9kz" seLinuxMountContext="" Oct 30 20:38:29.695368 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:38:29.694221 5378 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="59406b2a-a784-4d6e-aedf-96e11ff99f68" volumeName="kubernetes.io/configmap/59406b2a-a784-4d6e-aedf-96e11ff99f68-kube-rbac-proxy-cm" seLinuxMountContext="" Oct 30 20:38:29.695368 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:38:29.694250 5378 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0d6769f4-dc38-45ac-863b-c75fa672866a" volumeName="kubernetes.io/secret/0d6769f4-dc38-45ac-863b-c75fa672866a-original-pull-secret" seLinuxMountContext="" Oct 30 20:38:29.695368 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:38:29.694265 5378 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="59406b2a-a784-4d6e-aedf-96e11ff99f68" volumeName="kubernetes.io/secret/59406b2a-a784-4d6e-aedf-96e11ff99f68-insights-runtime-extractor-tls" seLinuxMountContext="" Oct 30 20:38:29.695368 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:38:29.694277 5378 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="6646b935-fe10-415a-be4b-a62a64ea45ec" volumeName="kubernetes.io/secret/6646b935-fe10-415a-be4b-a62a64ea45ec-console-oauth-config" seLinuxMountContext="" Oct 30 20:38:29.695368 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:38:29.694294 5378 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="666b6e91-4a60-45d9-9e3e-877867ded7fc" volumeName="kubernetes.io/projected/666b6e91-4a60-45d9-9e3e-877867ded7fc-kube-api-access-2w5c8" seLinuxMountContext="" Oct 30 20:38:29.695368 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:38:29.694305 5378 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e886897a-c6a3-414f-875f-aa112541b608" volumeName="kubernetes.io/projected/e886897a-c6a3-414f-875f-aa112541b608-kube-api-access-28zhr" seLinuxMountContext="" Oct 30 20:38:29.695368 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:38:29.694338 5378 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="23527d33-4e73-4c36-ba42-7f0a5b7f62c6" volumeName="kubernetes.io/empty-dir/23527d33-4e73-4c36-ba42-7f0a5b7f62c6-tmp" seLinuxMountContext="" Oct 30 20:38:29.695368 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:38:29.694382 5378 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="6a710ade-5a54-4595-89a8-636653c02fc5" volumeName="kubernetes.io/projected/6a710ade-5a54-4595-89a8-636653c02fc5-kube-api-access-sg8pj" seLinuxMountContext="" Oct 30 20:38:29.695368 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:38:29.694397 5378 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b7fa45b4-283a-4439-82b7-778abf2e9deb" volumeName="kubernetes.io/projected/b7fa45b4-283a-4439-82b7-778abf2e9deb-kube-api-access-crg8q" seLinuxMountContext="" Oct 30 20:38:29.695368 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:38:29.694409 5378 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="022c4cf0-9797-4428-8fa3-9f3112f2ccc5" volumeName="kubernetes.io/configmap/022c4cf0-9797-4428-8fa3-9f3112f2ccc5-cni-binary-copy" seLinuxMountContext="" Oct 30 20:38:29.695368 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:38:29.694431 5378 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="23527d33-4e73-4c36-ba42-7f0a5b7f62c6" volumeName="kubernetes.io/empty-dir/23527d33-4e73-4c36-ba42-7f0a5b7f62c6-etc-tuned" seLinuxMountContext="" Oct 30 20:38:29.695368 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:38:29.694442 5378 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3bcf59e7-befb-4623-9a0f-f01b70963b3e" volumeName="kubernetes.io/projected/3bcf59e7-befb-4623-9a0f-f01b70963b3e-kube-api-access-r9qpr" seLinuxMountContext="" Oct 30 20:38:29.695368 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:38:29.694454 5378 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="6a710ade-5a54-4595-89a8-636653c02fc5" volumeName="kubernetes.io/secret/6a710ade-5a54-4595-89a8-636653c02fc5-cert" seLinuxMountContext="" Oct 30 20:38:29.695368 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:38:29.694477 5378 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="743b90bd-3330-4751-a853-80b97fee0890" volumeName="kubernetes.io/projected/743b90bd-3330-4751-a853-80b97fee0890-kube-api-access-5x2gc" seLinuxMountContext="" Oct 30 20:38:29.695368 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:38:29.694489 5378 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="910c0d9f-c79b-4839-9a1b-f2d278d11fdb" volumeName="kubernetes.io/projected/910c0d9f-c79b-4839-9a1b-f2d278d11fdb-kube-api-access-qs9cm" seLinuxMountContext="" Oct 30 20:38:29.695368 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:38:29.694500 5378 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7df66b3c-abe2-4d20-abda-45320cd4af1b" volumeName="kubernetes.io/projected/7df66b3c-abe2-4d20-abda-45320cd4af1b-kube-api-access-xxc6m" seLinuxMountContext="" Oct 30 20:38:29.695368 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:38:29.694520 5378 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7df66b3c-abe2-4d20-abda-45320cd4af1b" volumeName="kubernetes.io/configmap/7df66b3c-abe2-4d20-abda-45320cd4af1b-node-exporter-accelerators-collector-config" seLinuxMountContext="" Oct 30 20:38:29.695368 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:38:29.694533 5378 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="022c4cf0-9797-4428-8fa3-9f3112f2ccc5" volumeName="kubernetes.io/projected/022c4cf0-9797-4428-8fa3-9f3112f2ccc5-kube-api-access-qjn9k" seLinuxMountContext="" Oct 30 20:38:29.695368 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:38:29.694546 5378 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="42765893-c3e3-41ab-811f-3135d4c74b07" volumeName="kubernetes.io/secret/42765893-c3e3-41ab-811f-3135d4c74b07-monitoring-plugin-cert" seLinuxMountContext="" Oct 30 20:38:29.695368 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:38:29.694582 5378 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="6646b935-fe10-415a-be4b-a62a64ea45ec" volumeName="kubernetes.io/secret/6646b935-fe10-415a-be4b-a62a64ea45ec-console-serving-cert" seLinuxMountContext="" Oct 30 20:38:29.695368 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:38:29.694594 5378 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e886897a-c6a3-414f-875f-aa112541b608" volumeName="kubernetes.io/secret/e886897a-c6a3-414f-875f-aa112541b608-metrics-certs" seLinuxMountContext="" Oct 30 20:38:29.695368 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:38:29.694608 5378 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3bcf59e7-befb-4623-9a0f-f01b70963b3e" volumeName="kubernetes.io/secret/3bcf59e7-befb-4623-9a0f-f01b70963b3e-secret-grpc-tls" seLinuxMountContext="" Oct 30 20:38:29.695368 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:38:29.694632 5378 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="42765893-c3e3-41ab-811f-3135d4c74b07" volumeName="kubernetes.io/projected/42765893-c3e3-41ab-811f-3135d4c74b07-kube-api-access-wk62v" seLinuxMountContext="" Oct 30 20:38:29.695368 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:38:29.694645 5378 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="6646b935-fe10-415a-be4b-a62a64ea45ec" volumeName="kubernetes.io/projected/6646b935-fe10-415a-be4b-a62a64ea45ec-kube-api-access-fqlw7" seLinuxMountContext="" Oct 30 20:38:29.695368 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:38:29.694656 5378 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="affdfb88-e208-4ca1-843d-efc8384c27a1" volumeName="kubernetes.io/configmap/affdfb88-e208-4ca1-843d-efc8384c27a1-iptables-alerter-script" seLinuxMountContext="" Oct 30 20:38:29.695368 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:38:29.694669 5378 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b7fa45b4-283a-4439-82b7-778abf2e9deb" volumeName="kubernetes.io/configmap/b7fa45b4-283a-4439-82b7-778abf2e9deb-serviceca" seLinuxMountContext="" Oct 30 20:38:29.695368 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:38:29.694681 5378 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="666b6e91-4a60-45d9-9e3e-877867ded7fc" volumeName="kubernetes.io/empty-dir/666b6e91-4a60-45d9-9e3e-877867ded7fc-tmp-dir" seLinuxMountContext="" Oct 30 20:38:29.695368 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:38:29.694695 5378 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="022c4cf0-9797-4428-8fa3-9f3112f2ccc5" volumeName="kubernetes.io/configmap/022c4cf0-9797-4428-8fa3-9f3112f2ccc5-multus-daemon-config" seLinuxMountContext="" Oct 30 20:38:29.695368 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:38:29.694707 5378 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3bcf59e7-befb-4623-9a0f-f01b70963b3e" volumeName="kubernetes.io/secret/3bcf59e7-befb-4623-9a0f-f01b70963b3e-secret-thanos-querier-kube-rbac-proxy-web" seLinuxMountContext="" Oct 30 20:38:29.695368 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:38:29.694719 5378 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="497c44c4-2ece-4009-b88c-6ecfa9920110" volumeName="kubernetes.io/configmap/497c44c4-2ece-4009-b88c-6ecfa9920110-konnectivity-ca" seLinuxMountContext="" Oct 30 20:38:29.695368 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:38:29.694734 5378 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5186676f-8c82-4667-a013-3efc22d2c4e7" volumeName="kubernetes.io/projected/5186676f-8c82-4667-a013-3efc22d2c4e7-kube-api-access-9d2s8" seLinuxMountContext="" Oct 30 20:38:29.695368 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:38:29.694747 5378 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="662e676d-c357-4ac8-a813-d34096051aff" volumeName="kubernetes.io/projected/662e676d-c357-4ac8-a813-d34096051aff-kube-api-access-k82pb" seLinuxMountContext="" Oct 30 20:38:29.695368 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:38:29.694769 5378 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="743b90bd-3330-4751-a853-80b97fee0890" volumeName="kubernetes.io/empty-dir/743b90bd-3330-4751-a853-80b97fee0890-tmp-dir" seLinuxMountContext="" Oct 30 20:38:29.695368 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:38:29.694781 5378 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="affdfb88-e208-4ca1-843d-efc8384c27a1" volumeName="kubernetes.io/projected/affdfb88-e208-4ca1-843d-efc8384c27a1-kube-api-access-t5psr" seLinuxMountContext="" Oct 30 20:38:29.695368 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:38:29.694793 5378 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c2ff359c-cf92-4a5b-8610-f6eb41acbe73" volumeName="kubernetes.io/secret/c2ff359c-cf92-4a5b-8610-f6eb41acbe73-ovn-node-metrics-cert" seLinuxMountContext="" Oct 30 20:38:29.695368 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:38:29.694804 5378 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7df66b3c-abe2-4d20-abda-45320cd4af1b" volumeName="kubernetes.io/empty-dir/7df66b3c-abe2-4d20-abda-45320cd4af1b-node-exporter-textfile" seLinuxMountContext="" Oct 30 20:38:29.695368 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:38:29.694814 5378 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c2ff359c-cf92-4a5b-8610-f6eb41acbe73" volumeName="kubernetes.io/configmap/c2ff359c-cf92-4a5b-8610-f6eb41acbe73-env-overrides" seLinuxMountContext="" Oct 30 20:38:29.695368 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:38:29.694827 5378 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c2ff359c-cf92-4a5b-8610-f6eb41acbe73" volumeName="kubernetes.io/configmap/c2ff359c-cf92-4a5b-8610-f6eb41acbe73-ovnkube-script-lib" seLinuxMountContext="" Oct 30 20:38:29.695368 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:38:29.694849 5378 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3bcf59e7-befb-4623-9a0f-f01b70963b3e" volumeName="kubernetes.io/secret/3bcf59e7-befb-4623-9a0f-f01b70963b3e-secret-thanos-querier-tls" seLinuxMountContext="" Oct 30 20:38:29.695368 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:38:29.694861 5378 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5186676f-8c82-4667-a013-3efc22d2c4e7" volumeName="kubernetes.io/configmap/5186676f-8c82-4667-a013-3efc22d2c4e7-metrics-client-ca" seLinuxMountContext="" Oct 30 20:38:29.695368 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:38:29.694872 5378 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5186676f-8c82-4667-a013-3efc22d2c4e7" volumeName="kubernetes.io/secret/5186676f-8c82-4667-a013-3efc22d2c4e7-prometheus-operator-tls" seLinuxMountContext="" Oct 30 20:38:29.695368 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:38:29.694884 5378 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="743b90bd-3330-4751-a853-80b97fee0890" volumeName="kubernetes.io/configmap/743b90bd-3330-4751-a853-80b97fee0890-config-volume" seLinuxMountContext="" Oct 30 20:38:29.695368 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:38:29.694895 5378 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7df66b3c-abe2-4d20-abda-45320cd4af1b" volumeName="kubernetes.io/configmap/7df66b3c-abe2-4d20-abda-45320cd4af1b-metrics-client-ca" seLinuxMountContext="" Oct 30 20:38:29.695368 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:38:29.694907 5378 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7df66b3c-abe2-4d20-abda-45320cd4af1b" volumeName="kubernetes.io/secret/7df66b3c-abe2-4d20-abda-45320cd4af1b-node-exporter-kube-rbac-proxy-config" seLinuxMountContext="" Oct 30 20:38:29.695368 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:38:29.694921 5378 reconstruct.go:97] "Volume reconstruction finished" Oct 30 20:38:29.695368 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:38:29.694930 5378 reconciler.go:29] "Reconciler: start to sync state" Oct 30 20:38:29.721580 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:38:29.721530 5378 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-7x9jg_c2ff359c-cf92-4a5b-8610-f6eb41acbe73/ovn-acl-logging/0.log" Oct 30 20:38:29.730723 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:38:29.730694 5378 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Oct 30 20:38:29.731380 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:38:29.731358 5378 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-7x9jg_c2ff359c-cf92-4a5b-8610-f6eb41acbe73/ovn-acl-logging/0.log" Oct 30 20:38:29.733293 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:38:29.733237 5378 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Oct 30 20:38:29.733293 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:38:29.733261 5378 status_manager.go:249] "Starting to sync pod status with apiserver" Oct 30 20:38:29.733293 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:38:29.733283 5378 kubelet.go:2521] "Starting kubelet main sync loop" Oct 30 20:38:29.733499 ip-10-0-138-149 kubenswrapper[5378]: E1030 20:38:29.733321 5378 kubelet.go:2545] "Skipping pod synchronization" err="[container runtime status check may not have completed yet, PLEG is not healthy: pleg has yet to be successful]" Oct 30 20:38:29.750798 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:38:29.750751 5378 generic.go:356] "Generic (PLEG): container finished" podID="910c0d9f-c79b-4839-9a1b-f2d278d11fdb" containerID="518c10ef1a5ae0d2822952207f1f5bcff92efbef74e13186da746539ac48d3e3" exitCode=0 Oct 30 20:38:29.750798 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:38:29.750794 5378 generic.go:356] "Generic (PLEG): container finished" podID="910c0d9f-c79b-4839-9a1b-f2d278d11fdb" containerID="3997d28afa3738cf1530b6d2b2ed3e034bc9873d5b8c50ea05c7c3570e2c1ea2" exitCode=0 Oct 30 20:38:29.750798 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:38:29.750801 5378 generic.go:356] "Generic (PLEG): container finished" podID="910c0d9f-c79b-4839-9a1b-f2d278d11fdb" containerID="49857697620af61532dbf1d52834ddac9bad36f61ec5936b3424359ce1c69970" exitCode=0 Oct 30 20:38:29.750972 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:38:29.750807 5378 generic.go:356] "Generic (PLEG): container finished" podID="910c0d9f-c79b-4839-9a1b-f2d278d11fdb" containerID="c27d3efddeec26f74a3705d8eb76ea39320cd98af64fabce37e8bae458c8c87a" exitCode=0 Oct 30 20:38:29.750972 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:38:29.750813 5378 generic.go:356] "Generic (PLEG): container finished" podID="910c0d9f-c79b-4839-9a1b-f2d278d11fdb" containerID="5814699cadaf62a2bf4a4282326b8dcba613ce6e0559c80f56610ca8fa8bf2b3" exitCode=0 Oct 30 20:38:29.750972 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:38:29.750818 5378 generic.go:356] "Generic (PLEG): container finished" podID="910c0d9f-c79b-4839-9a1b-f2d278d11fdb" containerID="51ae0082eac1fe9c20081136f6a5f46288ab3cb06de3e424497f41191fc2b3af" exitCode=0 Oct 30 20:38:29.779315 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:38:29.779288 5378 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-7x9jg_c2ff359c-cf92-4a5b-8610-f6eb41acbe73/ovn-acl-logging/0.log" Oct 30 20:38:29.779991 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:38:29.779960 5378 generic.go:356] "Generic (PLEG): container finished" podID="c2ff359c-cf92-4a5b-8610-f6eb41acbe73" containerID="29de0471a6b007a7c1375d5961078e11c1ca7d03092695918b4db8e245ea19b3" exitCode=1 Oct 30 20:38:29.786546 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:38:29.786514 5378 generic.go:356] "Generic (PLEG): container finished" podID="60d05e7d5fab914587e076383d73f1df" containerID="f87aaaf5fc68a52fc39a2ffb73ec2f494c38bd0f2efbd90e2a2ba5a311044017" exitCode=0 Oct 30 20:38:29.788818 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:38:29.788794 5378 generic.go:356] "Generic (PLEG): container finished" podID="7df66b3c-abe2-4d20-abda-45320cd4af1b" containerID="47daf950889e16ba586bd60eca28cf7c6fd6f0789681ccd3d78f919841c63bae" exitCode=0 Oct 30 20:38:29.790098 ip-10-0-138-149 kubenswrapper[5378]: E1030 20:38:29.790071 5378 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-138-149.us-west-1.compute.internal\" not found" Oct 30 20:38:29.803495 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:38:29.803475 5378 manager.go:324] Recovery completed Oct 30 20:38:29.827650 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:38:29.827627 5378 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 20:38:29.829385 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:38:29.829368 5378 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-138-149.us-west-1.compute.internal" event="NodeHasSufficientMemory" Oct 30 20:38:29.829473 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:38:29.829402 5378 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-138-149.us-west-1.compute.internal" event="NodeHasNoDiskPressure" Oct 30 20:38:29.829473 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:38:29.829414 5378 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-138-149.us-west-1.compute.internal" event="NodeHasSufficientPID" Oct 30 20:38:29.833584 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:38:29.832387 5378 cpu_manager.go:226] "Starting" policy="none" Oct 30 20:38:29.833584 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:38:29.832404 5378 cpu_manager.go:227] "Reconciling" reconcilePeriod="10s" Oct 30 20:38:29.833584 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:38:29.832423 5378 state_mem.go:41] "Initialized" logger="CPUManager state checkpoint.CPUManager state memory" Oct 30 20:38:29.833584 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:38:29.832464 5378 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 20:38:29.833584 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:38:29.832559 5378 state_mem.go:94] "Updated default CPUSet" logger="CPUManager state checkpoint.CPUManager state memory" cpuSet="" Oct 30 20:38:29.833584 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:38:29.832587 5378 state_mem.go:102] "Updated CPUSet assignments" logger="CPUManager state checkpoint.CPUManager state memory" assignments={} Oct 30 20:38:29.833584 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:38:29.832615 5378 state_checkpoint.go:142] "restored state from checkpoint" logger="CPUManager state checkpoint" Oct 30 20:38:29.833584 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:38:29.832623 5378 state_checkpoint.go:143] "defaultCPUSet" logger="CPUManager state checkpoint" defaultCpuSet="" Oct 30 20:38:29.833584 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:38:29.832631 5378 policy_none.go:50] "Start" Oct 30 20:38:29.833584 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:38:29.832655 5378 memory_manager.go:187] "Starting memorymanager" policy="None" Oct 30 20:38:29.833584 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:38:29.832671 5378 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Oct 30 20:38:29.833584 ip-10-0-138-149 kubenswrapper[5378]: E1030 20:38:29.833408 5378 kubelet.go:2545] "Skipping pod synchronization" err="container runtime status check may not have completed yet" Oct 30 20:38:29.834151 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:38:29.833692 5378 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-138-149.us-west-1.compute.internal" event="NodeHasSufficientMemory" Oct 30 20:38:29.834151 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:38:29.833725 5378 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-138-149.us-west-1.compute.internal" event="NodeHasNoDiskPressure" Oct 30 20:38:29.834151 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:38:29.833742 5378 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-138-149.us-west-1.compute.internal" event="NodeHasSufficientPID" Oct 30 20:38:29.834151 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:38:29.833781 5378 state_mem.go:77] "Updated machine memory state" logger="Memory Manager state checkpoint" Oct 30 20:38:29.834151 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:38:29.833801 5378 state_checkpoint.go:85] "State checkpoint: restored state from checkpoint" logger="Memory Manager state checkpoint" Oct 30 20:38:29.834151 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:38:29.833810 5378 policy_none.go:44] "Start" Oct 30 20:38:29.840804 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:38:29.840777 5378 node_container_manager_linux.go:130] "Enforcing system reserved compressible on cgroup" cgroupName="/system.slice" limits={"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"} Oct 30 20:38:29.843812 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:38:29.843799 5378 manager.go:341] "Starting Device Plugin manager" Oct 30 20:38:29.843912 ip-10-0-138-149 kubenswrapper[5378]: E1030 20:38:29.843823 5378 manager.go:525] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Oct 30 20:38:29.843912 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:38:29.843832 5378 server.go:87] "Starting device plugin registration server" Oct 30 20:38:29.844179 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:38:29.844164 5378 eviction_manager.go:194] "Eviction manager: starting control loop" Oct 30 20:38:29.844253 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:38:29.844181 5378 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Oct 30 20:38:29.846751 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:38:29.844354 5378 plugin_watcher.go:53] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Oct 30 20:38:29.846751 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:38:29.844476 5378 plugin_watcher.go:201] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Oct 30 20:38:29.846751 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:38:29.844519 5378 plugin_manager.go:119] "The desired_state_of_world populator (plugin watcher) starts" Oct 30 20:38:29.846751 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:38:29.844526 5378 plugin_manager.go:121] "Starting Kubelet Plugin Manager" Oct 30 20:38:29.846751 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:38:29.844598 5378 reconciler.go:167] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-10-30T20:38:29.844507854Z","UUID":"509900bc-4fa1-43aa-b163-2c761ea2dfcf","Handler":null,"Name":"","Endpoint":""} Oct 30 20:38:29.847276 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:38:29.847156 5378 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Oct 30 20:38:29.847276 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:38:29.847181 5378 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Oct 30 20:38:29.853845 ip-10-0-138-149 kubenswrapper[5378]: E1030 20:38:29.853822 5378 eviction_manager.go:272] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Oct 30 20:38:29.853948 ip-10-0-138-149 kubenswrapper[5378]: E1030 20:38:29.853877 5378 eviction_manager.go:297] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-138-149.us-west-1.compute.internal\" not found" Oct 30 20:38:29.875757 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:38:29.875739 5378 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 20:38:29.875867 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:38:29.875847 5378 reconstruct.go:205] "DevicePaths of reconstructed volumes updated" Oct 30 20:38:29.876319 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:38:29.876302 5378 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 20:38:29.876684 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:38:29.876666 5378 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 20:38:29.944652 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:38:29.944623 5378 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 20:38:29.945902 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:38:29.945884 5378 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-138-149.us-west-1.compute.internal" event="NodeHasSufficientMemory" Oct 30 20:38:29.946024 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:38:29.945914 5378 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-138-149.us-west-1.compute.internal" event="NodeHasNoDiskPressure" Oct 30 20:38:29.946024 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:38:29.945928 5378 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-138-149.us-west-1.compute.internal" event="NodeHasSufficientPID" Oct 30 20:38:29.946024 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:38:29.945967 5378 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-138-149.us-west-1.compute.internal" Oct 30 20:38:29.976358 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:38:29.976324 5378 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 20:38:30.033781 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:38:30.033684 5378 kubelet.go:2608] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-149.us-west-1.compute.internal","kube-system/kube-apiserver-proxy-ip-10-0-138-149.us-west-1.compute.internal"] Oct 30 20:38:30.034417 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:38:30.034369 5378 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-138-149.us-west-1.compute.internal" event={"ID":"0d2e0d42ed810f19933d4cd655966ed4","Type":"ContainerStarted","Data":"189448b3eda32f8fbaa0c723099f6d88ab5b649ccb1077284ac1384b1e6b6fcc"} Oct 30 20:38:30.034547 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:38:30.034425 5378 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-138-149.us-west-1.compute.internal" event={"ID":"0d2e0d42ed810f19933d4cd655966ed4","Type":"ContainerStarted","Data":"da4df417dfa91620c245546301b2dbee88814652f2989ebdddb126fbdfc28a98"} Oct 30 20:38:30.034547 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:38:30.034461 5378 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-149.us-west-1.compute.internal" Oct 30 20:38:30.034547 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:38:30.034533 5378 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-149.us-west-1.compute.internal" event={"ID":"60d05e7d5fab914587e076383d73f1df","Type":"ContainerStarted","Data":"56e0009de6d65db637c2c16b3018480161a58ad176e5f6395d81ece7ba6b2d3d"} Oct 30 20:38:30.034660 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:38:30.034553 5378 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-149.us-west-1.compute.internal" event={"ID":"60d05e7d5fab914587e076383d73f1df","Type":"ContainerDied","Data":"f87aaaf5fc68a52fc39a2ffb73ec2f494c38bd0f2efbd90e2a2ba5a311044017"} Oct 30 20:38:30.034660 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:38:30.034595 5378 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-149.us-west-1.compute.internal" event={"ID":"60d05e7d5fab914587e076383d73f1df","Type":"ContainerStarted","Data":"3ac9f12e7f98d1036fcc0c39f3aa3ab3c15fc903458208bef56f5c1969e7f9e0"} Oct 30 20:38:30.034768 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:38:30.034750 5378 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-138-149.us-west-1.compute.internal" Oct 30 20:38:30.085024 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:38:30.084995 5378 kubelet_node_status.go:126] "Node was previously registered" node="ip-10-0-138-149.us-west-1.compute.internal" Oct 30 20:38:30.085171 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:38:30.085081 5378 kubelet_node_status.go:80] "Successfully registered node" node="ip-10-0-138-149.us-west-1.compute.internal" Oct 30 20:38:30.093165 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:38:30.093137 5378 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Oct 30 20:38:30.093300 ip-10-0-138-149 kubenswrapper[5378]: E1030 20:38:30.093264 5378 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-apiserver-proxy-ip-10-0-138-149.us-west-1.compute.internal\" already exists" pod="kube-system/kube-apiserver-proxy-ip-10-0-138-149.us-west-1.compute.internal" Oct 30 20:38:30.096680 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:38:30.096656 5378 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Oct 30 20:38:30.096798 ip-10-0-138-149 kubenswrapper[5378]: E1030 20:38:30.096714 5378 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-rbac-proxy-crio-ip-10-0-138-149.us-west-1.compute.internal\" already exists" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-149.us-west-1.compute.internal" Oct 30 20:38:30.176992 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:38:30.176960 5378 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/60d05e7d5fab914587e076383d73f1df-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-138-149.us-west-1.compute.internal\" (UID: \"60d05e7d5fab914587e076383d73f1df\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-149.us-west-1.compute.internal" Oct 30 20:38:30.177155 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:38:30.177008 5378 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/0d2e0d42ed810f19933d4cd655966ed4-config\") pod \"kube-apiserver-proxy-ip-10-0-138-149.us-west-1.compute.internal\" (UID: \"0d2e0d42ed810f19933d4cd655966ed4\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-138-149.us-west-1.compute.internal" Oct 30 20:38:30.177155 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:38:30.177027 5378 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/60d05e7d5fab914587e076383d73f1df-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-138-149.us-west-1.compute.internal\" (UID: \"60d05e7d5fab914587e076383d73f1df\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-149.us-west-1.compute.internal" Oct 30 20:38:30.277292 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:38:30.277260 5378 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/60d05e7d5fab914587e076383d73f1df-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-138-149.us-west-1.compute.internal\" (UID: \"60d05e7d5fab914587e076383d73f1df\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-149.us-west-1.compute.internal" Oct 30 20:38:30.277292 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:38:30.277295 5378 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/60d05e7d5fab914587e076383d73f1df-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-138-149.us-west-1.compute.internal\" (UID: \"60d05e7d5fab914587e076383d73f1df\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-149.us-west-1.compute.internal" Oct 30 20:38:30.277512 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:38:30.277326 5378 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/0d2e0d42ed810f19933d4cd655966ed4-config\") pod \"kube-apiserver-proxy-ip-10-0-138-149.us-west-1.compute.internal\" (UID: \"0d2e0d42ed810f19933d4cd655966ed4\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-138-149.us-west-1.compute.internal" Oct 30 20:38:30.277512 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:38:30.277376 5378 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/0d2e0d42ed810f19933d4cd655966ed4-config\") pod \"kube-apiserver-proxy-ip-10-0-138-149.us-west-1.compute.internal\" (UID: \"0d2e0d42ed810f19933d4cd655966ed4\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-138-149.us-west-1.compute.internal" Oct 30 20:38:30.277512 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:38:30.277376 5378 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/60d05e7d5fab914587e076383d73f1df-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-138-149.us-west-1.compute.internal\" (UID: \"60d05e7d5fab914587e076383d73f1df\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-149.us-west-1.compute.internal" Oct 30 20:38:30.277512 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:38:30.277382 5378 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/60d05e7d5fab914587e076383d73f1df-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-138-149.us-west-1.compute.internal\" (UID: \"60d05e7d5fab914587e076383d73f1df\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-149.us-west-1.compute.internal" Oct 30 20:38:30.661595 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:38:30.661554 5378 apiserver.go:52] "Watching apiserver" Oct 30 20:38:30.806619 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:38:30.806591 5378 reflector.go:446] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Oct 30 20:38:30.807203 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:38:30.807178 5378 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-brbhg","openshift-dns/dns-default-8wwgv","openshift-monitoring/monitoring-plugin-77f4f76594-64rbk","kube-system/global-pull-secret-syncer-bcmwc","openshift-image-registry/node-ca-tlch5","openshift-monitoring/node-exporter-jzxqq","openshift-multus/network-metrics-daemon-nhs9k","openshift-network-operator/iptables-alerter-kvq2l","kube-system/konnectivity-agent-cjrkj","openshift-monitoring/prometheus-operator-9cd89b74c-2xw4j","openshift-console/downloads-6dd789c8b7-5pslw","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-svf4h","openshift-console/console-6bfdc6f474-hj4hq","openshift-monitoring/thanos-querier-7d8d77f945-zfsgx","openshift-ovn-kubernetes/ovnkube-node-7x9jg","openshift-multus/multus-additional-cni-plugins-blccx","openshift-insights/insights-runtime-extractor-q6mx7","openshift-multus/multus-dzbhx","openshift-network-diagnostics/network-check-target-ws655","kube-system/kube-apiserver-proxy-ip-10-0-138-149.us-west-1.compute.internal","openshift-cluster-node-tuning-operator/tuned-w9npq","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-149.us-west-1.compute.internal","openshift-dns/node-resolver-nv2rk"] Oct 30 20:38:30.809645 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:38:30.809625 5378 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-149.us-west-1.compute.internal" Oct 30 20:38:30.862185 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:38:30.862136 5378 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Oct 30 20:38:30.862185 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:38:30.862151 5378 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Oct 30 20:38:30.862185 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:38:30.862180 5378 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-hjmcj\"" Oct 30 20:38:30.862381 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:38:30.862217 5378 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Oct 30 20:38:30.862381 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:38:30.862257 5378 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Oct 30 20:38:30.862381 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:38:30.862256 5378 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-86vxf\"" Oct 30 20:38:30.862610 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:38:30.862597 5378 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Oct 30 20:38:30.862887 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:38:30.862869 5378 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Oct 30 20:38:30.863025 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:38:30.862898 5378 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Oct 30 20:38:30.863025 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:38:30.862902 5378 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Oct 30 20:38:30.863104 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:38:30.863064 5378 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Oct 30 20:38:30.863150 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:38:30.863119 5378 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Oct 30 20:38:30.863266 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:38:30.863148 5378 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Oct 30 20:38:30.863266 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:38:30.863162 5378 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Oct 30 20:38:30.863476 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:38:30.863453 5378 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Oct 30 20:38:30.863476 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:38:30.863460 5378 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Oct 30 20:38:30.863608 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:38:30.863490 5378 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Oct 30 20:38:30.863608 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:38:30.863518 5378 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Oct 30 20:38:30.863608 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:38:30.863547 5378 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-node-identity-dockercfg-rv75s\"" Oct 30 20:38:30.863608 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:38:30.863546 5378 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-4lpn7\"" Oct 30 20:38:30.863608 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:38:30.863575 5378 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-g2pwr\"" Oct 30 20:38:30.863608 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:38:30.863583 5378 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-sr85n\"" Oct 30 20:38:30.863608 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:38:30.863602 5378 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Oct 30 20:38:30.863608 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:38:30.863609 5378 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Oct 30 20:38:30.863863 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:38:30.863609 5378 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Oct 30 20:38:30.863863 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:38:30.863705 5378 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Oct 30 20:38:30.863863 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:38:30.863751 5378 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-jwfqc\"" Oct 30 20:38:30.868932 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:38:30.868918 5378 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Oct 30 20:38:30.869251 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:38:30.869230 5378 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Oct 30 20:38:30.869344 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:38:30.869266 5378 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Oct 30 20:38:30.869344 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:38:30.869304 5378 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Oct 30 20:38:30.869452 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:38:30.869419 5378 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Oct 30 20:38:30.869743 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:38:30.869723 5378 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Oct 30 20:38:30.869863 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:38:30.869752 5378 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Oct 30 20:38:30.869863 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:38:30.869763 5378 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Oct 30 20:38:30.869863 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:38:30.869779 5378 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Oct 30 20:38:30.869863 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:38:30.869791 5378 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Oct 30 20:38:30.869863 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:38:30.869838 5378 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-bmpds\"" Oct 30 20:38:30.869863 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:38:30.869850 5378 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Oct 30 20:38:30.870130 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:38:30.869898 5378 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Oct 30 20:38:30.870130 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:38:30.869901 5378 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Oct 30 20:38:30.870130 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:38:30.869926 5378 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Oct 30 20:38:30.870130 ip-10-0-138-149 kubenswrapper[5378]: E1030 20:38:30.869976 5378 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-rbac-proxy-crio-ip-10-0-138-149.us-west-1.compute.internal\" already exists" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-149.us-west-1.compute.internal" Oct 30 20:38:30.870492 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:38:30.870364 5378 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-nvzwj\"" Oct 30 20:38:30.870492 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:38:30.870423 5378 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent-dockercfg-c5chh\"" Oct 30 20:38:30.870492 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:38:30.870469 5378 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-94s75\"" Oct 30 20:38:30.870693 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:38:30.870504 5378 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-49f2v\"" Oct 30 20:38:30.892016 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:38:30.891994 5378 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Oct 30 20:38:30.901095 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:38:30.901073 5378 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-kube-rbac-proxy-config\"" Oct 30 20:38:30.901095 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:38:30.901077 5378 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Oct 30 20:38:30.901326 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:38:30.901305 5378 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Oct 30 20:38:30.901382 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:38:30.901310 5378 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Oct 30 20:38:30.901382 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:38:30.901314 5378 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Oct 30 20:38:30.901618 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:38:30.901603 5378 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Oct 30 20:38:30.901738 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:38:30.901623 5378 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Oct 30 20:38:30.901738 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:38:30.901661 5378 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-2srwc\"" Oct 30 20:38:30.901738 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:38:30.901672 5378 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-tls\"" Oct 30 20:38:30.901939 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:38:30.901922 5378 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-dockercfg-dttss\"" Oct 30 20:38:30.902002 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:38:30.901946 5378 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Oct 30 20:38:30.902059 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:38:30.902004 5378 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"ingress-canary-dockercfg-h8n42\"" Oct 30 20:38:30.902109 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:38:30.902098 5378 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" Oct 30 20:38:30.908264 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:38:30.908249 5378 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Oct 30 20:38:30.908590 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:38:30.908559 5378 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Oct 30 20:38:30.908691 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:38:30.908599 5378 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-tls\"" Oct 30 20:38:30.908691 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:38:30.908636 5378 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\"" Oct 30 20:38:30.908782 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:38:30.908768 5378 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-rd5vq\"" Oct 30 20:38:30.908835 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:38:30.908821 5378 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\"" Oct 30 20:38:30.908884 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:38:30.908840 5378 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\"" Oct 30 20:38:30.908884 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:38:30.908841 5378 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-cert\"" Oct 30 20:38:30.908994 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:38:30.908893 5378 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-metrics\"" Oct 30 20:38:30.908994 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:38:30.908926 5378 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\"" Oct 30 20:38:30.909321 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:38:30.909306 5378 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\"" Oct 30 20:38:30.909372 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:38:30.909350 5378 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" Oct 30 20:38:30.911912 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:38:30.911865 5378 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-web\"" Oct 30 20:38:30.911978 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:38:30.911922 5378 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-dockercfg-dfkzz\"" Oct 30 20:38:30.911978 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:38:30.911928 5378 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" Oct 30 20:38:30.911978 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:38:30.911966 5378 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy\"" Oct 30 20:38:30.912122 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:38:30.911979 5378 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-wsjv8\"" Oct 30 20:38:30.912122 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:38:30.911997 5378 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-rules\"" Oct 30 20:38:30.912122 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:38:30.911998 5378 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\"" Oct 30 20:38:30.912122 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:38:30.912020 5378 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\"" Oct 30 20:38:30.912836 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:38:30.912822 5378 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-dockercfg-ngvh2\"" Oct 30 20:38:30.912880 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:38:30.912847 5378 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-dmr5l\"" Oct 30 20:38:30.912910 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:38:30.912901 5378 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-grpc-tls-dvvj6jlp1e008\"" Oct 30 20:38:30.912973 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:38:30.912961 5378 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" Oct 30 20:38:30.982761 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:38:30.982741 5378 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/6646b935-fe10-415a-be4b-a62a64ea45ec-console-serving-cert\") pod \"console-6bfdc6f474-hj4hq\" (UID: \"6646b935-fe10-415a-be4b-a62a64ea45ec\") " pod="openshift-console/console-6bfdc6f474-hj4hq" Oct 30 20:38:30.982838 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:38:30.982770 5378 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/23527d33-4e73-4c36-ba42-7f0a5b7f62c6-etc-sysctl-d\") pod \"tuned-w9npq\" (UID: \"23527d33-4e73-4c36-ba42-7f0a5b7f62c6\") " pod="openshift-cluster-node-tuning-operator/tuned-w9npq" Oct 30 20:38:30.982838 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:38:30.982789 5378 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/910c0d9f-c79b-4839-9a1b-f2d278d11fdb-system-cni-dir\") pod \"multus-additional-cni-plugins-blccx\" (UID: \"910c0d9f-c79b-4839-9a1b-f2d278d11fdb\") " pod="openshift-multus/multus-additional-cni-plugins-blccx" Oct 30 20:38:30.982838 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:38:30.982806 5378 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/42765893-c3e3-41ab-811f-3135d4c74b07-monitoring-plugin-cert\") pod \"monitoring-plugin-77f4f76594-64rbk\" (UID: \"42765893-c3e3-41ab-811f-3135d4c74b07\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-64rbk" Oct 30 20:38:30.982935 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:38:30.982883 5378 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/022c4cf0-9797-4428-8fa3-9f3112f2ccc5-multus-daemon-config\") pod \"multus-dzbhx\" (UID: \"022c4cf0-9797-4428-8fa3-9f3112f2ccc5\") " pod="openshift-multus/multus-dzbhx" Oct 30 20:38:30.982965 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:38:30.982956 5378 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t5psr\" (UniqueName: \"kubernetes.io/projected/affdfb88-e208-4ca1-843d-efc8384c27a1-kube-api-access-t5psr\") pod \"iptables-alerter-kvq2l\" (UID: \"affdfb88-e208-4ca1-843d-efc8384c27a1\") " pod="openshift-network-operator/iptables-alerter-kvq2l" Oct 30 20:38:30.983050 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:38:30.983034 5378 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/59406b2a-a784-4d6e-aedf-96e11ff99f68-data-volume\") pod \"insights-runtime-extractor-q6mx7\" (UID: \"59406b2a-a784-4d6e-aedf-96e11ff99f68\") " pod="openshift-insights/insights-runtime-extractor-q6mx7" Oct 30 20:38:30.983123 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:38:30.983108 5378 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/59406b2a-a784-4d6e-aedf-96e11ff99f68-data-volume\") pod \"insights-runtime-extractor-q6mx7\" (UID: \"59406b2a-a784-4d6e-aedf-96e11ff99f68\") " pod="openshift-insights/insights-runtime-extractor-q6mx7" Oct 30 20:38:30.983175 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:38:30.983111 5378 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/6646b935-fe10-415a-be4b-a62a64ea45ec-console-serving-cert\") pod \"console-6bfdc6f474-hj4hq\" (UID: \"6646b935-fe10-415a-be4b-a62a64ea45ec\") " pod="openshift-console/console-6bfdc6f474-hj4hq" Oct 30 20:38:30.983175 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:38:30.983123 5378 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/022c4cf0-9797-4428-8fa3-9f3112f2ccc5-multus-daemon-config\") pod \"multus-dzbhx\" (UID: \"022c4cf0-9797-4428-8fa3-9f3112f2ccc5\") " pod="openshift-multus/multus-dzbhx" Oct 30 20:38:30.983175 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:38:30.983129 5378 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jmxp5\" (UniqueName: \"kubernetes.io/projected/59046ab6-2f4c-4031-ba8c-21fb322a9d07-kube-api-access-jmxp5\") pod \"downloads-6dd789c8b7-5pslw\" (UID: \"59046ab6-2f4c-4031-ba8c-21fb322a9d07\") " pod="openshift-console/downloads-6dd789c8b7-5pslw" Oct 30 20:38:30.983175 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:38:30.983111 5378 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/42765893-c3e3-41ab-811f-3135d4c74b07-monitoring-plugin-cert\") pod \"monitoring-plugin-77f4f76594-64rbk\" (UID: \"42765893-c3e3-41ab-811f-3135d4c74b07\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-64rbk" Oct 30 20:38:30.983175 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:38:30.983167 5378 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/0d6769f4-dc38-45ac-863b-c75fa672866a-original-pull-secret\") pod \"global-pull-secret-syncer-bcmwc\" (UID: \"0d6769f4-dc38-45ac-863b-c75fa672866a\") " pod="kube-system/global-pull-secret-syncer-bcmwc" Oct 30 20:38:30.983388 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:38:30.983280 5378 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/c2ff359c-cf92-4a5b-8610-f6eb41acbe73-host-cni-netd\") pod \"ovnkube-node-7x9jg\" (UID: \"c2ff359c-cf92-4a5b-8610-f6eb41acbe73\") " pod="openshift-ovn-kubernetes/ovnkube-node-7x9jg" Oct 30 20:38:30.983388 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:38:30.983314 5378 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k82pb\" (UniqueName: \"kubernetes.io/projected/662e676d-c357-4ac8-a813-d34096051aff-kube-api-access-k82pb\") pod \"aws-ebs-csi-driver-node-svf4h\" (UID: \"662e676d-c357-4ac8-a813-d34096051aff\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-svf4h" Oct 30 20:38:30.983388 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:38:30.983348 5378 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/5186676f-8c82-4667-a013-3efc22d2c4e7-metrics-client-ca\") pod \"prometheus-operator-9cd89b74c-2xw4j\" (UID: \"5186676f-8c82-4667-a013-3efc22d2c4e7\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-2xw4j" Oct 30 20:38:30.983388 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:38:30.983373 5378 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/022c4cf0-9797-4428-8fa3-9f3112f2ccc5-cnibin\") pod \"multus-dzbhx\" (UID: \"022c4cf0-9797-4428-8fa3-9f3112f2ccc5\") " pod="openshift-multus/multus-dzbhx" Oct 30 20:38:30.983588 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:38:30.983399 5378 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/662e676d-c357-4ac8-a813-d34096051aff-registration-dir\") pod \"aws-ebs-csi-driver-node-svf4h\" (UID: \"662e676d-c357-4ac8-a813-d34096051aff\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-svf4h" Oct 30 20:38:30.983588 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:38:30.983427 5378 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/3bcf59e7-befb-4623-9a0f-f01b70963b3e-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-7d8d77f945-zfsgx\" (UID: \"3bcf59e7-befb-4623-9a0f-f01b70963b3e\") " pod="openshift-monitoring/thanos-querier-7d8d77f945-zfsgx" Oct 30 20:38:30.983588 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:38:30.983452 5378 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c2ff359c-cf92-4a5b-8610-f6eb41acbe73-host-run-ovn-kubernetes\") pod \"ovnkube-node-7x9jg\" (UID: \"c2ff359c-cf92-4a5b-8610-f6eb41acbe73\") " pod="openshift-ovn-kubernetes/ovnkube-node-7x9jg" Oct 30 20:38:30.983588 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:38:30.983494 5378 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fwkdk\" (UniqueName: \"kubernetes.io/projected/59406b2a-a784-4d6e-aedf-96e11ff99f68-kube-api-access-fwkdk\") pod \"insights-runtime-extractor-q6mx7\" (UID: \"59406b2a-a784-4d6e-aedf-96e11ff99f68\") " pod="openshift-insights/insights-runtime-extractor-q6mx7" Oct 30 20:38:30.983588 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:38:30.983518 5378 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/7df66b3c-abe2-4d20-abda-45320cd4af1b-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-jzxqq\" (UID: \"7df66b3c-abe2-4d20-abda-45320cd4af1b\") " pod="openshift-monitoring/node-exporter-jzxqq" Oct 30 20:38:30.983588 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:38:30.983544 5378 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/0d6769f4-dc38-45ac-863b-c75fa672866a-original-pull-secret\") pod \"global-pull-secret-syncer-bcmwc\" (UID: \"0d6769f4-dc38-45ac-863b-c75fa672866a\") " pod="kube-system/global-pull-secret-syncer-bcmwc" Oct 30 20:38:30.983588 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:38:30.983583 5378 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/743b90bd-3330-4751-a853-80b97fee0890-metrics-tls\") pod \"dns-default-8wwgv\" (UID: \"743b90bd-3330-4751-a853-80b97fee0890\") " pod="openshift-dns/dns-default-8wwgv" Oct 30 20:38:30.983887 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:38:30.983544 5378 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/5186676f-8c82-4667-a013-3efc22d2c4e7-metrics-client-ca\") pod \"prometheus-operator-9cd89b74c-2xw4j\" (UID: \"5186676f-8c82-4667-a013-3efc22d2c4e7\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-2xw4j" Oct 30 20:38:30.983887 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:38:30.983612 5378 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/022c4cf0-9797-4428-8fa3-9f3112f2ccc5-host-run-multus-certs\") pod \"multus-dzbhx\" (UID: \"022c4cf0-9797-4428-8fa3-9f3112f2ccc5\") " pod="openshift-multus/multus-dzbhx" Oct 30 20:38:30.983887 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:38:30.983653 5378 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/6646b935-fe10-415a-be4b-a62a64ea45ec-tmp\") pod \"console-6bfdc6f474-hj4hq\" (UID: \"6646b935-fe10-415a-be4b-a62a64ea45ec\") " pod="openshift-console/console-6bfdc6f474-hj4hq" Oct 30 20:38:30.983887 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:38:30.983681 5378 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/7df66b3c-abe2-4d20-abda-45320cd4af1b-node-exporter-wtmp\") pod \"node-exporter-jzxqq\" (UID: \"7df66b3c-abe2-4d20-abda-45320cd4af1b\") " pod="openshift-monitoring/node-exporter-jzxqq" Oct 30 20:38:30.983887 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:38:30.983705 5378 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/6646b935-fe10-415a-be4b-a62a64ea45ec-console-oauth-config\") pod \"console-6bfdc6f474-hj4hq\" (UID: \"6646b935-fe10-415a-be4b-a62a64ea45ec\") " pod="openshift-console/console-6bfdc6f474-hj4hq" Oct 30 20:38:30.983887 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:38:30.983731 5378 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/910c0d9f-c79b-4839-9a1b-f2d278d11fdb-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-blccx\" (UID: \"910c0d9f-c79b-4839-9a1b-f2d278d11fdb\") " pod="openshift-multus/multus-additional-cni-plugins-blccx" Oct 30 20:38:30.983887 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:38:30.983731 5378 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/3bcf59e7-befb-4623-9a0f-f01b70963b3e-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-7d8d77f945-zfsgx\" (UID: \"3bcf59e7-befb-4623-9a0f-f01b70963b3e\") " pod="openshift-monitoring/thanos-querier-7d8d77f945-zfsgx" Oct 30 20:38:30.983887 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:38:30.983736 5378 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/7df66b3c-abe2-4d20-abda-45320cd4af1b-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-jzxqq\" (UID: \"7df66b3c-abe2-4d20-abda-45320cd4af1b\") " pod="openshift-monitoring/node-exporter-jzxqq" Oct 30 20:38:30.983887 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:38:30.983758 5378 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/6646b935-fe10-415a-be4b-a62a64ea45ec-tmp\") pod \"console-6bfdc6f474-hj4hq\" (UID: \"6646b935-fe10-415a-be4b-a62a64ea45ec\") " pod="openshift-console/console-6bfdc6f474-hj4hq" Oct 30 20:38:30.983887 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:38:30.983764 5378 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/022c4cf0-9797-4428-8fa3-9f3112f2ccc5-host-run-netns\") pod \"multus-dzbhx\" (UID: \"022c4cf0-9797-4428-8fa3-9f3112f2ccc5\") " pod="openshift-multus/multus-dzbhx" Oct 30 20:38:30.983887 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:38:30.983772 5378 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/743b90bd-3330-4751-a853-80b97fee0890-metrics-tls\") pod \"dns-default-8wwgv\" (UID: \"743b90bd-3330-4751-a853-80b97fee0890\") " pod="openshift-dns/dns-default-8wwgv" Oct 30 20:38:30.983887 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:38:30.983802 5378 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/7df66b3c-abe2-4d20-abda-45320cd4af1b-sys\") pod \"node-exporter-jzxqq\" (UID: \"7df66b3c-abe2-4d20-abda-45320cd4af1b\") " pod="openshift-monitoring/node-exporter-jzxqq" Oct 30 20:38:30.983887 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:38:30.983845 5378 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xcs6c\" (UniqueName: \"kubernetes.io/projected/23527d33-4e73-4c36-ba42-7f0a5b7f62c6-kube-api-access-xcs6c\") pod \"tuned-w9npq\" (UID: \"23527d33-4e73-4c36-ba42-7f0a5b7f62c6\") " pod="openshift-cluster-node-tuning-operator/tuned-w9npq" Oct 30 20:38:30.983887 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:38:30.983873 5378 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-crg8q\" (UniqueName: \"kubernetes.io/projected/b7fa45b4-283a-4439-82b7-778abf2e9deb-kube-api-access-crg8q\") pod \"node-ca-tlch5\" (UID: \"b7fa45b4-283a-4439-82b7-778abf2e9deb\") " pod="openshift-image-registry/node-ca-tlch5" Oct 30 20:38:30.984303 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:38:30.983906 5378 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/6646b935-fe10-415a-be4b-a62a64ea45ec-console-oauth-config\") pod \"console-6bfdc6f474-hj4hq\" (UID: \"6646b935-fe10-415a-be4b-a62a64ea45ec\") " pod="openshift-console/console-6bfdc6f474-hj4hq" Oct 30 20:38:30.984303 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:38:30.983929 5378 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/910c0d9f-c79b-4839-9a1b-f2d278d11fdb-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-blccx\" (UID: \"910c0d9f-c79b-4839-9a1b-f2d278d11fdb\") " pod="openshift-multus/multus-additional-cni-plugins-blccx" Oct 30 20:38:30.984303 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:38:30.983942 5378 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/662e676d-c357-4ac8-a813-d34096051aff-device-dir\") pod \"aws-ebs-csi-driver-node-svf4h\" (UID: \"662e676d-c357-4ac8-a813-d34096051aff\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-svf4h" Oct 30 20:38:30.984303 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:38:30.983972 5378 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/c2ff359c-cf92-4a5b-8610-f6eb41acbe73-ovnkube-script-lib\") pod \"ovnkube-node-7x9jg\" (UID: \"c2ff359c-cf92-4a5b-8610-f6eb41acbe73\") " pod="openshift-ovn-kubernetes/ovnkube-node-7x9jg" Oct 30 20:38:30.984303 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:38:30.983999 5378 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/022c4cf0-9797-4428-8fa3-9f3112f2ccc5-multus-cni-dir\") pod \"multus-dzbhx\" (UID: \"022c4cf0-9797-4428-8fa3-9f3112f2ccc5\") " pod="openshift-multus/multus-dzbhx" Oct 30 20:38:30.984303 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:38:30.984025 5378 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bc9kz\" (UniqueName: \"kubernetes.io/projected/567a72f6-035d-4c61-acaf-f46008029168-kube-api-access-bc9kz\") pod \"network-check-target-ws655\" (UID: \"567a72f6-035d-4c61-acaf-f46008029168\") " pod="openshift-network-diagnostics/network-check-target-ws655" Oct 30 20:38:30.984303 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:38:30.984042 5378 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/0d6769f4-dc38-45ac-863b-c75fa672866a-kubelet-config\") pod \"global-pull-secret-syncer-bcmwc\" (UID: \"0d6769f4-dc38-45ac-863b-c75fa672866a\") " pod="kube-system/global-pull-secret-syncer-bcmwc" Oct 30 20:38:30.984303 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:38:30.984055 5378 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/c2ff359c-cf92-4a5b-8610-f6eb41acbe73-node-log\") pod \"ovnkube-node-7x9jg\" (UID: \"c2ff359c-cf92-4a5b-8610-f6eb41acbe73\") " pod="openshift-ovn-kubernetes/ovnkube-node-7x9jg" Oct 30 20:38:30.984303 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:38:30.984084 5378 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/022c4cf0-9797-4428-8fa3-9f3112f2ccc5-host-var-lib-kubelet\") pod \"multus-dzbhx\" (UID: \"022c4cf0-9797-4428-8fa3-9f3112f2ccc5\") " pod="openshift-multus/multus-dzbhx" Oct 30 20:38:30.984303 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:38:30.984164 5378 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/c2ff359c-cf92-4a5b-8610-f6eb41acbe73-ovnkube-script-lib\") pod \"ovnkube-node-7x9jg\" (UID: \"c2ff359c-cf92-4a5b-8610-f6eb41acbe73\") " pod="openshift-ovn-kubernetes/ovnkube-node-7x9jg" Oct 30 20:38:30.984303 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:38:30.984200 5378 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/59406b2a-a784-4d6e-aedf-96e11ff99f68-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-q6mx7\" (UID: \"59406b2a-a784-4d6e-aedf-96e11ff99f68\") " pod="openshift-insights/insights-runtime-extractor-q6mx7" Oct 30 20:38:30.984303 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:38:30.984231 5378 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/6a710ade-5a54-4595-89a8-636653c02fc5-cert\") pod \"ingress-canary-brbhg\" (UID: \"6a710ade-5a54-4595-89a8-636653c02fc5\") " pod="openshift-ingress-canary/ingress-canary-brbhg" Oct 30 20:38:30.984303 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:38:30.984270 5378 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/6646b935-fe10-415a-be4b-a62a64ea45ec-service-ca\") pod \"console-6bfdc6f474-hj4hq\" (UID: \"6646b935-fe10-415a-be4b-a62a64ea45ec\") " pod="openshift-console/console-6bfdc6f474-hj4hq" Oct 30 20:38:30.984303 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:38:30.984295 5378 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/6646b935-fe10-415a-be4b-a62a64ea45ec-oauth-serving-cert\") pod \"console-6bfdc6f474-hj4hq\" (UID: \"6646b935-fe10-415a-be4b-a62a64ea45ec\") " pod="openshift-console/console-6bfdc6f474-hj4hq" Oct 30 20:38:30.984827 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:38:30.984397 5378 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/497c44c4-2ece-4009-b88c-6ecfa9920110-agent-certs\") pod \"konnectivity-agent-cjrkj\" (UID: \"497c44c4-2ece-4009-b88c-6ecfa9920110\") " pod="kube-system/konnectivity-agent-cjrkj" Oct 30 20:38:30.984827 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:38:30.984416 5378 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/c2ff359c-cf92-4a5b-8610-f6eb41acbe73-run-openvswitch\") pod \"ovnkube-node-7x9jg\" (UID: \"c2ff359c-cf92-4a5b-8610-f6eb41acbe73\") " pod="openshift-ovn-kubernetes/ovnkube-node-7x9jg" Oct 30 20:38:30.984827 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:38:30.984413 5378 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/59406b2a-a784-4d6e-aedf-96e11ff99f68-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-q6mx7\" (UID: \"59406b2a-a784-4d6e-aedf-96e11ff99f68\") " pod="openshift-insights/insights-runtime-extractor-q6mx7" Oct 30 20:38:30.984827 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:38:30.984442 5378 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5x2gc\" (UniqueName: \"kubernetes.io/projected/743b90bd-3330-4751-a853-80b97fee0890-kube-api-access-5x2gc\") pod \"dns-default-8wwgv\" (UID: \"743b90bd-3330-4751-a853-80b97fee0890\") " pod="openshift-dns/dns-default-8wwgv" Oct 30 20:38:30.984827 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:38:30.984445 5378 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/6646b935-fe10-415a-be4b-a62a64ea45ec-service-ca\") pod \"console-6bfdc6f474-hj4hq\" (UID: \"6646b935-fe10-415a-be4b-a62a64ea45ec\") " pod="openshift-console/console-6bfdc6f474-hj4hq" Oct 30 20:38:30.984827 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:38:30.984473 5378 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/6a710ade-5a54-4595-89a8-636653c02fc5-cert\") pod \"ingress-canary-brbhg\" (UID: \"6a710ade-5a54-4595-89a8-636653c02fc5\") " pod="openshift-ingress-canary/ingress-canary-brbhg" Oct 30 20:38:30.984827 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:38:30.984612 5378 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/23527d33-4e73-4c36-ba42-7f0a5b7f62c6-run\") pod \"tuned-w9npq\" (UID: \"23527d33-4e73-4c36-ba42-7f0a5b7f62c6\") " pod="openshift-cluster-node-tuning-operator/tuned-w9npq" Oct 30 20:38:30.984827 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:38:30.984642 5378 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/743b90bd-3330-4751-a853-80b97fee0890-tmp-dir\") pod \"dns-default-8wwgv\" (UID: \"743b90bd-3330-4751-a853-80b97fee0890\") " pod="openshift-dns/dns-default-8wwgv" Oct 30 20:38:30.984827 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:38:30.984706 5378 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/6646b935-fe10-415a-be4b-a62a64ea45ec-oauth-serving-cert\") pod \"console-6bfdc6f474-hj4hq\" (UID: \"6646b935-fe10-415a-be4b-a62a64ea45ec\") " pod="openshift-console/console-6bfdc6f474-hj4hq" Oct 30 20:38:30.984827 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:38:30.984714 5378 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/910c0d9f-c79b-4839-9a1b-f2d278d11fdb-cnibin\") pod \"multus-additional-cni-plugins-blccx\" (UID: \"910c0d9f-c79b-4839-9a1b-f2d278d11fdb\") " pod="openshift-multus/multus-additional-cni-plugins-blccx" Oct 30 20:38:30.984827 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:38:30.984742 5378 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/743b90bd-3330-4751-a853-80b97fee0890-tmp-dir\") pod \"dns-default-8wwgv\" (UID: \"743b90bd-3330-4751-a853-80b97fee0890\") " pod="openshift-dns/dns-default-8wwgv" Oct 30 20:38:30.984827 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:38:30.984748 5378 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/b7fa45b4-283a-4439-82b7-778abf2e9deb-serviceca\") pod \"node-ca-tlch5\" (UID: \"b7fa45b4-283a-4439-82b7-778abf2e9deb\") " pod="openshift-image-registry/node-ca-tlch5" Oct 30 20:38:30.984827 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:38:30.984815 5378 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/022c4cf0-9797-4428-8fa3-9f3112f2ccc5-host-var-lib-cni-bin\") pod \"multus-dzbhx\" (UID: \"022c4cf0-9797-4428-8fa3-9f3112f2ccc5\") " pod="openshift-multus/multus-dzbhx" Oct 30 20:38:30.985267 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:38:30.984861 5378 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/7df66b3c-abe2-4d20-abda-45320cd4af1b-node-exporter-tls\") pod \"node-exporter-jzxqq\" (UID: \"7df66b3c-abe2-4d20-abda-45320cd4af1b\") " pod="openshift-monitoring/node-exporter-jzxqq" Oct 30 20:38:30.985267 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:38:30.984902 5378 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/666b6e91-4a60-45d9-9e3e-877867ded7fc-hosts-file\") pod \"node-resolver-nv2rk\" (UID: \"666b6e91-4a60-45d9-9e3e-877867ded7fc\") " pod="openshift-dns/node-resolver-nv2rk" Oct 30 20:38:30.985267 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:38:30.984927 5378 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/23527d33-4e73-4c36-ba42-7f0a5b7f62c6-etc-sysctl-conf\") pod \"tuned-w9npq\" (UID: \"23527d33-4e73-4c36-ba42-7f0a5b7f62c6\") " pod="openshift-cluster-node-tuning-operator/tuned-w9npq" Oct 30 20:38:30.985267 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:38:30.984954 5378 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/c2ff359c-cf92-4a5b-8610-f6eb41acbe73-host-run-netns\") pod \"ovnkube-node-7x9jg\" (UID: \"c2ff359c-cf92-4a5b-8610-f6eb41acbe73\") " pod="openshift-ovn-kubernetes/ovnkube-node-7x9jg" Oct 30 20:38:30.985267 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:38:30.985005 5378 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/022c4cf0-9797-4428-8fa3-9f3112f2ccc5-system-cni-dir\") pod \"multus-dzbhx\" (UID: \"022c4cf0-9797-4428-8fa3-9f3112f2ccc5\") " pod="openshift-multus/multus-dzbhx" Oct 30 20:38:30.985267 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:38:30.985039 5378 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e886897a-c6a3-414f-875f-aa112541b608-metrics-certs\") pod \"network-metrics-daemon-nhs9k\" (UID: \"e886897a-c6a3-414f-875f-aa112541b608\") " pod="openshift-multus/network-metrics-daemon-nhs9k" Oct 30 20:38:30.985267 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:38:30.985061 5378 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/497c44c4-2ece-4009-b88c-6ecfa9920110-agent-certs\") pod \"konnectivity-agent-cjrkj\" (UID: \"497c44c4-2ece-4009-b88c-6ecfa9920110\") " pod="kube-system/konnectivity-agent-cjrkj" Oct 30 20:38:30.985267 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:38:30.985068 5378 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/022c4cf0-9797-4428-8fa3-9f3112f2ccc5-host-run-k8s-cni-cncf-io\") pod \"multus-dzbhx\" (UID: \"022c4cf0-9797-4428-8fa3-9f3112f2ccc5\") " pod="openshift-multus/multus-dzbhx" Oct 30 20:38:30.985267 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:38:30.985098 5378 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/022c4cf0-9797-4428-8fa3-9f3112f2ccc5-multus-conf-dir\") pod \"multus-dzbhx\" (UID: \"022c4cf0-9797-4428-8fa3-9f3112f2ccc5\") " pod="openshift-multus/multus-dzbhx" Oct 30 20:38:30.985267 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:38:30.985109 5378 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/7df66b3c-abe2-4d20-abda-45320cd4af1b-node-exporter-tls\") pod \"node-exporter-jzxqq\" (UID: \"7df66b3c-abe2-4d20-abda-45320cd4af1b\") " pod="openshift-monitoring/node-exporter-jzxqq" Oct 30 20:38:30.985267 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:38:30.985140 5378 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/affdfb88-e208-4ca1-843d-efc8384c27a1-iptables-alerter-script\") pod \"iptables-alerter-kvq2l\" (UID: \"affdfb88-e208-4ca1-843d-efc8384c27a1\") " pod="openshift-network-operator/iptables-alerter-kvq2l" Oct 30 20:38:30.985267 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:38:30.985195 5378 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/3bcf59e7-befb-4623-9a0f-f01b70963b3e-metrics-client-ca\") pod \"thanos-querier-7d8d77f945-zfsgx\" (UID: \"3bcf59e7-befb-4623-9a0f-f01b70963b3e\") " pod="openshift-monitoring/thanos-querier-7d8d77f945-zfsgx" Oct 30 20:38:30.985267 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:38:30.985228 5378 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/b7fa45b4-283a-4439-82b7-778abf2e9deb-serviceca\") pod \"node-ca-tlch5\" (UID: \"b7fa45b4-283a-4439-82b7-778abf2e9deb\") " pod="openshift-image-registry/node-ca-tlch5" Oct 30 20:38:30.985267 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:38:30.985241 5378 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e886897a-c6a3-414f-875f-aa112541b608-metrics-certs\") pod \"network-metrics-daemon-nhs9k\" (UID: \"e886897a-c6a3-414f-875f-aa112541b608\") " pod="openshift-multus/network-metrics-daemon-nhs9k" Oct 30 20:38:30.985267 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:38:30.985268 5378 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/497c44c4-2ece-4009-b88c-6ecfa9920110-konnectivity-ca\") pod \"konnectivity-agent-cjrkj\" (UID: \"497c44c4-2ece-4009-b88c-6ecfa9920110\") " pod="kube-system/konnectivity-agent-cjrkj" Oct 30 20:38:30.985998 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:38:30.985287 5378 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/affdfb88-e208-4ca1-843d-efc8384c27a1-iptables-alerter-script\") pod \"iptables-alerter-kvq2l\" (UID: \"affdfb88-e208-4ca1-843d-efc8384c27a1\") " pod="openshift-network-operator/iptables-alerter-kvq2l" Oct 30 20:38:30.985998 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:38:30.985302 5378 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-24qwx\" (UniqueName: \"kubernetes.io/projected/c2ff359c-cf92-4a5b-8610-f6eb41acbe73-kube-api-access-24qwx\") pod \"ovnkube-node-7x9jg\" (UID: \"c2ff359c-cf92-4a5b-8610-f6eb41acbe73\") " pod="openshift-ovn-kubernetes/ovnkube-node-7x9jg" Oct 30 20:38:30.985998 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:38:30.985331 5378 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/743b90bd-3330-4751-a853-80b97fee0890-config-volume\") pod \"dns-default-8wwgv\" (UID: \"743b90bd-3330-4751-a853-80b97fee0890\") " pod="openshift-dns/dns-default-8wwgv" Oct 30 20:38:30.985998 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:38:30.985356 5378 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/910c0d9f-c79b-4839-9a1b-f2d278d11fdb-cni-binary-copy\") pod \"multus-additional-cni-plugins-blccx\" (UID: \"910c0d9f-c79b-4839-9a1b-f2d278d11fdb\") " pod="openshift-multus/multus-additional-cni-plugins-blccx" Oct 30 20:38:30.985998 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:38:30.985380 5378 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/022c4cf0-9797-4428-8fa3-9f3112f2ccc5-cni-binary-copy\") pod \"multus-dzbhx\" (UID: \"022c4cf0-9797-4428-8fa3-9f3112f2ccc5\") " pod="openshift-multus/multus-dzbhx" Oct 30 20:38:30.985998 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:38:30.985385 5378 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/3bcf59e7-befb-4623-9a0f-f01b70963b3e-metrics-client-ca\") pod \"thanos-querier-7d8d77f945-zfsgx\" (UID: \"3bcf59e7-befb-4623-9a0f-f01b70963b3e\") " pod="openshift-monitoring/thanos-querier-7d8d77f945-zfsgx" Oct 30 20:38:30.985998 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:38:30.985399 5378 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/497c44c4-2ece-4009-b88c-6ecfa9920110-konnectivity-ca\") pod \"konnectivity-agent-cjrkj\" (UID: \"497c44c4-2ece-4009-b88c-6ecfa9920110\") " pod="kube-system/konnectivity-agent-cjrkj" Oct 30 20:38:30.985998 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:38:30.985407 5378 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/c2ff359c-cf92-4a5b-8610-f6eb41acbe73-systemd-units\") pod \"ovnkube-node-7x9jg\" (UID: \"c2ff359c-cf92-4a5b-8610-f6eb41acbe73\") " pod="openshift-ovn-kubernetes/ovnkube-node-7x9jg" Oct 30 20:38:30.985998 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:38:30.985431 5378 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/c2ff359c-cf92-4a5b-8610-f6eb41acbe73-ovn-node-metrics-cert\") pod \"ovnkube-node-7x9jg\" (UID: \"c2ff359c-cf92-4a5b-8610-f6eb41acbe73\") " pod="openshift-ovn-kubernetes/ovnkube-node-7x9jg" Oct 30 20:38:30.985998 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:38:30.985460 5378 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qs9cm\" (UniqueName: \"kubernetes.io/projected/910c0d9f-c79b-4839-9a1b-f2d278d11fdb-kube-api-access-qs9cm\") pod \"multus-additional-cni-plugins-blccx\" (UID: \"910c0d9f-c79b-4839-9a1b-f2d278d11fdb\") " pod="openshift-multus/multus-additional-cni-plugins-blccx" Oct 30 20:38:30.985998 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:38:30.985498 5378 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/662e676d-c357-4ac8-a813-d34096051aff-sys-fs\") pod \"aws-ebs-csi-driver-node-svf4h\" (UID: \"662e676d-c357-4ac8-a813-d34096051aff\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-svf4h" Oct 30 20:38:30.985998 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:38:30.985528 5378 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/7df66b3c-abe2-4d20-abda-45320cd4af1b-metrics-client-ca\") pod \"node-exporter-jzxqq\" (UID: \"7df66b3c-abe2-4d20-abda-45320cd4af1b\") " pod="openshift-monitoring/node-exporter-jzxqq" Oct 30 20:38:30.985998 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:38:30.985544 5378 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/743b90bd-3330-4751-a853-80b97fee0890-config-volume\") pod \"dns-default-8wwgv\" (UID: \"743b90bd-3330-4751-a853-80b97fee0890\") " pod="openshift-dns/dns-default-8wwgv" Oct 30 20:38:30.985998 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:38:30.985556 5378 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/c2ff359c-cf92-4a5b-8610-f6eb41acbe73-log-socket\") pod \"ovnkube-node-7x9jg\" (UID: \"c2ff359c-cf92-4a5b-8610-f6eb41acbe73\") " pod="openshift-ovn-kubernetes/ovnkube-node-7x9jg" Oct 30 20:38:30.985998 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:38:30.985559 5378 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/910c0d9f-c79b-4839-9a1b-f2d278d11fdb-cni-binary-copy\") pod \"multus-additional-cni-plugins-blccx\" (UID: \"910c0d9f-c79b-4839-9a1b-f2d278d11fdb\") " pod="openshift-multus/multus-additional-cni-plugins-blccx" Oct 30 20:38:30.985998 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:38:30.985559 5378 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/022c4cf0-9797-4428-8fa3-9f3112f2ccc5-cni-binary-copy\") pod \"multus-dzbhx\" (UID: \"022c4cf0-9797-4428-8fa3-9f3112f2ccc5\") " pod="openshift-multus/multus-dzbhx" Oct 30 20:38:30.985998 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:38:30.985612 5378 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-28zhr\" (UniqueName: \"kubernetes.io/projected/e886897a-c6a3-414f-875f-aa112541b608-kube-api-access-28zhr\") pod \"network-metrics-daemon-nhs9k\" (UID: \"e886897a-c6a3-414f-875f-aa112541b608\") " pod="openshift-multus/network-metrics-daemon-nhs9k" Oct 30 20:38:30.985998 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:38:30.985632 5378 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/662e676d-c357-4ac8-a813-d34096051aff-socket-dir\") pod \"aws-ebs-csi-driver-node-svf4h\" (UID: \"662e676d-c357-4ac8-a813-d34096051aff\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-svf4h" Oct 30 20:38:30.985998 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:38:30.985655 5378 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r9qpr\" (UniqueName: \"kubernetes.io/projected/3bcf59e7-befb-4623-9a0f-f01b70963b3e-kube-api-access-r9qpr\") pod \"thanos-querier-7d8d77f945-zfsgx\" (UID: \"3bcf59e7-befb-4623-9a0f-f01b70963b3e\") " pod="openshift-monitoring/thanos-querier-7d8d77f945-zfsgx" Oct 30 20:38:30.985998 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:38:30.985685 5378 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/c2ff359c-cf92-4a5b-8610-f6eb41acbe73-ovn-node-metrics-cert\") pod \"ovnkube-node-7x9jg\" (UID: \"c2ff359c-cf92-4a5b-8610-f6eb41acbe73\") " pod="openshift-ovn-kubernetes/ovnkube-node-7x9jg" Oct 30 20:38:30.985998 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:38:30.985700 5378 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/6646b935-fe10-415a-be4b-a62a64ea45ec-console-config\") pod \"console-6bfdc6f474-hj4hq\" (UID: \"6646b935-fe10-415a-be4b-a62a64ea45ec\") " pod="openshift-console/console-6bfdc6f474-hj4hq" Oct 30 20:38:30.985998 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:38:30.985741 5378 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/7df66b3c-abe2-4d20-abda-45320cd4af1b-metrics-client-ca\") pod \"node-exporter-jzxqq\" (UID: \"7df66b3c-abe2-4d20-abda-45320cd4af1b\") " pod="openshift-monitoring/node-exporter-jzxqq" Oct 30 20:38:30.985998 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:38:30.985751 5378 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/23527d33-4e73-4c36-ba42-7f0a5b7f62c6-host\") pod \"tuned-w9npq\" (UID: \"23527d33-4e73-4c36-ba42-7f0a5b7f62c6\") " pod="openshift-cluster-node-tuning-operator/tuned-w9npq" Oct 30 20:38:30.985998 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:38:30.985810 5378 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9d2s8\" (UniqueName: \"kubernetes.io/projected/5186676f-8c82-4667-a013-3efc22d2c4e7-kube-api-access-9d2s8\") pod \"prometheus-operator-9cd89b74c-2xw4j\" (UID: \"5186676f-8c82-4667-a013-3efc22d2c4e7\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-2xw4j" Oct 30 20:38:30.985998 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:38:30.985847 5378 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wk62v\" (UniqueName: \"kubernetes.io/projected/42765893-c3e3-41ab-811f-3135d4c74b07-kube-api-access-wk62v\") pod \"monitoring-plugin-77f4f76594-64rbk\" (UID: \"42765893-c3e3-41ab-811f-3135d4c74b07\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-64rbk" Oct 30 20:38:30.985998 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:38:30.985875 5378 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/662e676d-c357-4ac8-a813-d34096051aff-etc-selinux\") pod \"aws-ebs-csi-driver-node-svf4h\" (UID: \"662e676d-c357-4ac8-a813-d34096051aff\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-svf4h" Oct 30 20:38:30.985998 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:38:30.985909 5378 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/23527d33-4e73-4c36-ba42-7f0a5b7f62c6-etc-kubernetes\") pod \"tuned-w9npq\" (UID: \"23527d33-4e73-4c36-ba42-7f0a5b7f62c6\") " pod="openshift-cluster-node-tuning-operator/tuned-w9npq" Oct 30 20:38:30.985998 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:38:30.985927 5378 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/6646b935-fe10-415a-be4b-a62a64ea45ec-console-config\") pod \"console-6bfdc6f474-hj4hq\" (UID: \"6646b935-fe10-415a-be4b-a62a64ea45ec\") " pod="openshift-console/console-6bfdc6f474-hj4hq" Oct 30 20:38:30.987025 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:38:30.986028 5378 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/3bcf59e7-befb-4623-9a0f-f01b70963b3e-secret-thanos-querier-tls\") pod \"thanos-querier-7d8d77f945-zfsgx\" (UID: \"3bcf59e7-befb-4623-9a0f-f01b70963b3e\") " pod="openshift-monitoring/thanos-querier-7d8d77f945-zfsgx" Oct 30 20:38:30.987025 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:38:30.986060 5378 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/3bcf59e7-befb-4623-9a0f-f01b70963b3e-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-7d8d77f945-zfsgx\" (UID: \"3bcf59e7-befb-4623-9a0f-f01b70963b3e\") " pod="openshift-monitoring/thanos-querier-7d8d77f945-zfsgx" Oct 30 20:38:30.987025 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:38:30.986147 5378 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/23527d33-4e73-4c36-ba42-7f0a5b7f62c6-var-lib-kubelet\") pod \"tuned-w9npq\" (UID: \"23527d33-4e73-4c36-ba42-7f0a5b7f62c6\") " pod="openshift-cluster-node-tuning-operator/tuned-w9npq" Oct 30 20:38:30.987025 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:38:30.986230 5378 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/5186676f-8c82-4667-a013-3efc22d2c4e7-prometheus-operator-tls\") pod \"prometheus-operator-9cd89b74c-2xw4j\" (UID: \"5186676f-8c82-4667-a013-3efc22d2c4e7\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-2xw4j" Oct 30 20:38:30.987025 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:38:30.986247 5378 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/3bcf59e7-befb-4623-9a0f-f01b70963b3e-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-7d8d77f945-zfsgx\" (UID: \"3bcf59e7-befb-4623-9a0f-f01b70963b3e\") " pod="openshift-monitoring/thanos-querier-7d8d77f945-zfsgx" Oct 30 20:38:30.987025 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:38:30.986260 5378 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/022c4cf0-9797-4428-8fa3-9f3112f2ccc5-etc-kubernetes\") pod \"multus-dzbhx\" (UID: \"022c4cf0-9797-4428-8fa3-9f3112f2ccc5\") " pod="openshift-multus/multus-dzbhx" Oct 30 20:38:30.987025 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:38:30.986275 5378 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/3bcf59e7-befb-4623-9a0f-f01b70963b3e-secret-thanos-querier-tls\") pod \"thanos-querier-7d8d77f945-zfsgx\" (UID: \"3bcf59e7-befb-4623-9a0f-f01b70963b3e\") " pod="openshift-monitoring/thanos-querier-7d8d77f945-zfsgx" Oct 30 20:38:30.987025 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:38:30.986286 5378 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/7df66b3c-abe2-4d20-abda-45320cd4af1b-node-exporter-accelerators-collector-config\") pod \"node-exporter-jzxqq\" (UID: \"7df66b3c-abe2-4d20-abda-45320cd4af1b\") " pod="openshift-monitoring/node-exporter-jzxqq" Oct 30 20:38:30.987025 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:38:30.986321 5378 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/022c4cf0-9797-4428-8fa3-9f3112f2ccc5-host-var-lib-cni-multus\") pod \"multus-dzbhx\" (UID: \"022c4cf0-9797-4428-8fa3-9f3112f2ccc5\") " pod="openshift-multus/multus-dzbhx" Oct 30 20:38:30.987025 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:38:30.986413 5378 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/5186676f-8c82-4667-a013-3efc22d2c4e7-prometheus-operator-tls\") pod \"prometheus-operator-9cd89b74c-2xw4j\" (UID: \"5186676f-8c82-4667-a013-3efc22d2c4e7\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-2xw4j" Oct 30 20:38:30.987025 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:38:30.986414 5378 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2w5c8\" (UniqueName: \"kubernetes.io/projected/666b6e91-4a60-45d9-9e3e-877867ded7fc-kube-api-access-2w5c8\") pod \"node-resolver-nv2rk\" (UID: \"666b6e91-4a60-45d9-9e3e-877867ded7fc\") " pod="openshift-dns/node-resolver-nv2rk" Oct 30 20:38:30.987025 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:38:30.986435 5378 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/7df66b3c-abe2-4d20-abda-45320cd4af1b-node-exporter-accelerators-collector-config\") pod \"node-exporter-jzxqq\" (UID: \"7df66b3c-abe2-4d20-abda-45320cd4af1b\") " pod="openshift-monitoring/node-exporter-jzxqq" Oct 30 20:38:30.987025 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:38:30.986442 5378 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/662e676d-c357-4ac8-a813-d34096051aff-kubelet-dir\") pod \"aws-ebs-csi-driver-node-svf4h\" (UID: \"662e676d-c357-4ac8-a813-d34096051aff\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-svf4h" Oct 30 20:38:30.987025 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:38:30.986591 5378 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/affdfb88-e208-4ca1-843d-efc8384c27a1-host-slash\") pod \"iptables-alerter-kvq2l\" (UID: \"affdfb88-e208-4ca1-843d-efc8384c27a1\") " pod="openshift-network-operator/iptables-alerter-kvq2l" Oct 30 20:38:30.987025 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:38:30.986609 5378 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/7df66b3c-abe2-4d20-abda-45320cd4af1b-root\") pod \"node-exporter-jzxqq\" (UID: \"7df66b3c-abe2-4d20-abda-45320cd4af1b\") " pod="openshift-monitoring/node-exporter-jzxqq" Oct 30 20:38:30.987025 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:38:30.986626 5378 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/3bcf59e7-befb-4623-9a0f-f01b70963b3e-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-7d8d77f945-zfsgx\" (UID: \"3bcf59e7-befb-4623-9a0f-f01b70963b3e\") " pod="openshift-monitoring/thanos-querier-7d8d77f945-zfsgx" Oct 30 20:38:30.987025 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:38:30.986639 5378 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/23527d33-4e73-4c36-ba42-7f0a5b7f62c6-etc-systemd\") pod \"tuned-w9npq\" (UID: \"23527d33-4e73-4c36-ba42-7f0a5b7f62c6\") " pod="openshift-cluster-node-tuning-operator/tuned-w9npq" Oct 30 20:38:30.987025 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:38:30.986655 5378 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/23527d33-4e73-4c36-ba42-7f0a5b7f62c6-sys\") pod \"tuned-w9npq\" (UID: \"23527d33-4e73-4c36-ba42-7f0a5b7f62c6\") " pod="openshift-cluster-node-tuning-operator/tuned-w9npq" Oct 30 20:38:30.987025 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:38:30.986678 5378 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/c2ff359c-cf92-4a5b-8610-f6eb41acbe73-run-systemd\") pod \"ovnkube-node-7x9jg\" (UID: \"c2ff359c-cf92-4a5b-8610-f6eb41acbe73\") " pod="openshift-ovn-kubernetes/ovnkube-node-7x9jg" Oct 30 20:38:30.987025 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:38:30.986707 5378 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/c2ff359c-cf92-4a5b-8610-f6eb41acbe73-host-cni-bin\") pod \"ovnkube-node-7x9jg\" (UID: \"c2ff359c-cf92-4a5b-8610-f6eb41acbe73\") " pod="openshift-ovn-kubernetes/ovnkube-node-7x9jg" Oct 30 20:38:30.987025 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:38:30.986729 5378 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/5186676f-8c82-4667-a013-3efc22d2c4e7-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-9cd89b74c-2xw4j\" (UID: \"5186676f-8c82-4667-a013-3efc22d2c4e7\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-2xw4j" Oct 30 20:38:30.987025 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:38:30.986762 5378 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qjn9k\" (UniqueName: \"kubernetes.io/projected/022c4cf0-9797-4428-8fa3-9f3112f2ccc5-kube-api-access-qjn9k\") pod \"multus-dzbhx\" (UID: \"022c4cf0-9797-4428-8fa3-9f3112f2ccc5\") " pod="openshift-multus/multus-dzbhx" Oct 30 20:38:30.987025 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:38:30.986828 5378 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/3bcf59e7-befb-4623-9a0f-f01b70963b3e-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-7d8d77f945-zfsgx\" (UID: \"3bcf59e7-befb-4623-9a0f-f01b70963b3e\") " pod="openshift-monitoring/thanos-querier-7d8d77f945-zfsgx" Oct 30 20:38:30.987025 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:38:30.986850 5378 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/23527d33-4e73-4c36-ba42-7f0a5b7f62c6-tmp\") pod \"tuned-w9npq\" (UID: \"23527d33-4e73-4c36-ba42-7f0a5b7f62c6\") " pod="openshift-cluster-node-tuning-operator/tuned-w9npq" Oct 30 20:38:30.987025 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:38:30.986884 5378 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/b7fa45b4-283a-4439-82b7-778abf2e9deb-host\") pod \"node-ca-tlch5\" (UID: \"b7fa45b4-283a-4439-82b7-778abf2e9deb\") " pod="openshift-image-registry/node-ca-tlch5" Oct 30 20:38:30.987025 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:38:30.986895 5378 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/23527d33-4e73-4c36-ba42-7f0a5b7f62c6-tmp\") pod \"tuned-w9npq\" (UID: \"23527d33-4e73-4c36-ba42-7f0a5b7f62c6\") " pod="openshift-cluster-node-tuning-operator/tuned-w9npq" Oct 30 20:38:30.987025 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:38:30.986904 5378 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/5186676f-8c82-4667-a013-3efc22d2c4e7-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-9cd89b74c-2xw4j\" (UID: \"5186676f-8c82-4667-a013-3efc22d2c4e7\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-2xw4j" Oct 30 20:38:30.987025 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:38:30.986910 5378 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/3bcf59e7-befb-4623-9a0f-f01b70963b3e-secret-grpc-tls\") pod \"thanos-querier-7d8d77f945-zfsgx\" (UID: \"3bcf59e7-befb-4623-9a0f-f01b70963b3e\") " pod="openshift-monitoring/thanos-querier-7d8d77f945-zfsgx" Oct 30 20:38:30.987025 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:38:30.986933 5378 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/23527d33-4e73-4c36-ba42-7f0a5b7f62c6-etc-modprobe-d\") pod \"tuned-w9npq\" (UID: \"23527d33-4e73-4c36-ba42-7f0a5b7f62c6\") " pod="openshift-cluster-node-tuning-operator/tuned-w9npq" Oct 30 20:38:30.987025 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:38:30.986950 5378 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/c2ff359c-cf92-4a5b-8610-f6eb41acbe73-host-kubelet\") pod \"ovnkube-node-7x9jg\" (UID: \"c2ff359c-cf92-4a5b-8610-f6eb41acbe73\") " pod="openshift-ovn-kubernetes/ovnkube-node-7x9jg" Oct 30 20:38:30.987025 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:38:30.986966 5378 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/910c0d9f-c79b-4839-9a1b-f2d278d11fdb-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-blccx\" (UID: \"910c0d9f-c79b-4839-9a1b-f2d278d11fdb\") " pod="openshift-multus/multus-additional-cni-plugins-blccx" Oct 30 20:38:30.987025 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:38:30.986991 5378 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/022c4cf0-9797-4428-8fa3-9f3112f2ccc5-multus-socket-dir-parent\") pod \"multus-dzbhx\" (UID: \"022c4cf0-9797-4428-8fa3-9f3112f2ccc5\") " pod="openshift-multus/multus-dzbhx" Oct 30 20:38:30.987025 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:38:30.987012 5378 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/3bcf59e7-befb-4623-9a0f-f01b70963b3e-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-7d8d77f945-zfsgx\" (UID: \"3bcf59e7-befb-4623-9a0f-f01b70963b3e\") " pod="openshift-monitoring/thanos-querier-7d8d77f945-zfsgx" Oct 30 20:38:30.987025 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:38:30.987028 5378 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/23527d33-4e73-4c36-ba42-7f0a5b7f62c6-etc-sysconfig\") pod \"tuned-w9npq\" (UID: \"23527d33-4e73-4c36-ba42-7f0a5b7f62c6\") " pod="openshift-cluster-node-tuning-operator/tuned-w9npq" Oct 30 20:38:30.987025 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:38:30.987043 5378 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/c2ff359c-cf92-4a5b-8610-f6eb41acbe73-ovnkube-config\") pod \"ovnkube-node-7x9jg\" (UID: \"c2ff359c-cf92-4a5b-8610-f6eb41acbe73\") " pod="openshift-ovn-kubernetes/ovnkube-node-7x9jg" Oct 30 20:38:30.987025 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:38:30.987054 5378 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/3bcf59e7-befb-4623-9a0f-f01b70963b3e-secret-grpc-tls\") pod \"thanos-querier-7d8d77f945-zfsgx\" (UID: \"3bcf59e7-befb-4623-9a0f-f01b70963b3e\") " pod="openshift-monitoring/thanos-querier-7d8d77f945-zfsgx" Oct 30 20:38:30.988267 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:38:30.987062 5378 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/c2ff359c-cf92-4a5b-8610-f6eb41acbe73-env-overrides\") pod \"ovnkube-node-7x9jg\" (UID: \"c2ff359c-cf92-4a5b-8610-f6eb41acbe73\") " pod="openshift-ovn-kubernetes/ovnkube-node-7x9jg" Oct 30 20:38:30.988267 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:38:30.987151 5378 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/910c0d9f-c79b-4839-9a1b-f2d278d11fdb-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-blccx\" (UID: \"910c0d9f-c79b-4839-9a1b-f2d278d11fdb\") " pod="openshift-multus/multus-additional-cni-plugins-blccx" Oct 30 20:38:30.988267 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:38:30.987170 5378 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/666b6e91-4a60-45d9-9e3e-877867ded7fc-tmp-dir\") pod \"node-resolver-nv2rk\" (UID: \"666b6e91-4a60-45d9-9e3e-877867ded7fc\") " pod="openshift-dns/node-resolver-nv2rk" Oct 30 20:38:30.988267 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:38:30.987188 5378 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/3bcf59e7-befb-4623-9a0f-f01b70963b3e-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-7d8d77f945-zfsgx\" (UID: \"3bcf59e7-befb-4623-9a0f-f01b70963b3e\") " pod="openshift-monitoring/thanos-querier-7d8d77f945-zfsgx" Oct 30 20:38:30.988267 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:38:30.987201 5378 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c2ff359c-cf92-4a5b-8610-f6eb41acbe73-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-7x9jg\" (UID: \"c2ff359c-cf92-4a5b-8610-f6eb41acbe73\") " pod="openshift-ovn-kubernetes/ovnkube-node-7x9jg" Oct 30 20:38:30.988267 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:38:30.987225 5378 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/c2ff359c-cf92-4a5b-8610-f6eb41acbe73-var-lib-openvswitch\") pod \"ovnkube-node-7x9jg\" (UID: \"c2ff359c-cf92-4a5b-8610-f6eb41acbe73\") " pod="openshift-ovn-kubernetes/ovnkube-node-7x9jg" Oct 30 20:38:30.988267 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:38:30.987226 5378 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/666b6e91-4a60-45d9-9e3e-877867ded7fc-tmp-dir\") pod \"node-resolver-nv2rk\" (UID: \"666b6e91-4a60-45d9-9e3e-877867ded7fc\") " pod="openshift-dns/node-resolver-nv2rk" Oct 30 20:38:30.988267 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:38:30.987240 5378 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/c2ff359c-cf92-4a5b-8610-f6eb41acbe73-etc-openvswitch\") pod \"ovnkube-node-7x9jg\" (UID: \"c2ff359c-cf92-4a5b-8610-f6eb41acbe73\") " pod="openshift-ovn-kubernetes/ovnkube-node-7x9jg" Oct 30 20:38:30.988267 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:38:30.987255 5378 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/c2ff359c-cf92-4a5b-8610-f6eb41acbe73-env-overrides\") pod \"ovnkube-node-7x9jg\" (UID: \"c2ff359c-cf92-4a5b-8610-f6eb41acbe73\") " pod="openshift-ovn-kubernetes/ovnkube-node-7x9jg" Oct 30 20:38:30.988267 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:38:30.987276 5378 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/c2ff359c-cf92-4a5b-8610-f6eb41acbe73-run-ovn\") pod \"ovnkube-node-7x9jg\" (UID: \"c2ff359c-cf92-4a5b-8610-f6eb41acbe73\") " pod="openshift-ovn-kubernetes/ovnkube-node-7x9jg" Oct 30 20:38:30.988267 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:38:30.987278 5378 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/c2ff359c-cf92-4a5b-8610-f6eb41acbe73-ovnkube-config\") pod \"ovnkube-node-7x9jg\" (UID: \"c2ff359c-cf92-4a5b-8610-f6eb41acbe73\") " pod="openshift-ovn-kubernetes/ovnkube-node-7x9jg" Oct 30 20:38:30.988267 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:38:30.987301 5378 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/910c0d9f-c79b-4839-9a1b-f2d278d11fdb-tuning-conf-dir\") pod \"multus-additional-cni-plugins-blccx\" (UID: \"910c0d9f-c79b-4839-9a1b-f2d278d11fdb\") " pod="openshift-multus/multus-additional-cni-plugins-blccx" Oct 30 20:38:30.988267 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:38:30.987322 5378 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/59046ab6-2f4c-4031-ba8c-21fb322a9d07-tmp\") pod \"downloads-6dd789c8b7-5pslw\" (UID: \"59046ab6-2f4c-4031-ba8c-21fb322a9d07\") " pod="openshift-console/downloads-6dd789c8b7-5pslw" Oct 30 20:38:30.988267 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:38:30.987337 5378 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/0d6769f4-dc38-45ac-863b-c75fa672866a-dbus\") pod \"global-pull-secret-syncer-bcmwc\" (UID: \"0d6769f4-dc38-45ac-863b-c75fa672866a\") " pod="kube-system/global-pull-secret-syncer-bcmwc" Oct 30 20:38:30.988267 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:38:30.987388 5378 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/23527d33-4e73-4c36-ba42-7f0a5b7f62c6-etc-tuned\") pod \"tuned-w9npq\" (UID: \"23527d33-4e73-4c36-ba42-7f0a5b7f62c6\") " pod="openshift-cluster-node-tuning-operator/tuned-w9npq" Oct 30 20:38:30.988267 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:38:30.987408 5378 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/59046ab6-2f4c-4031-ba8c-21fb322a9d07-tmp\") pod \"downloads-6dd789c8b7-5pslw\" (UID: \"59046ab6-2f4c-4031-ba8c-21fb322a9d07\") " pod="openshift-console/downloads-6dd789c8b7-5pslw" Oct 30 20:38:30.988267 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:38:30.987416 5378 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/022c4cf0-9797-4428-8fa3-9f3112f2ccc5-hostroot\") pod \"multus-dzbhx\" (UID: \"022c4cf0-9797-4428-8fa3-9f3112f2ccc5\") " pod="openshift-multus/multus-dzbhx" Oct 30 20:38:30.988267 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:38:30.987451 5378 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/23527d33-4e73-4c36-ba42-7f0a5b7f62c6-etc-tuned\") pod \"tuned-w9npq\" (UID: \"23527d33-4e73-4c36-ba42-7f0a5b7f62c6\") " pod="openshift-cluster-node-tuning-operator/tuned-w9npq" Oct 30 20:38:30.988267 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:38:30.987470 5378 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/7df66b3c-abe2-4d20-abda-45320cd4af1b-node-exporter-textfile\") pod \"node-exporter-jzxqq\" (UID: \"7df66b3c-abe2-4d20-abda-45320cd4af1b\") " pod="openshift-monitoring/node-exporter-jzxqq" Oct 30 20:38:30.988267 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:38:30.987492 5378 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xxc6m\" (UniqueName: \"kubernetes.io/projected/7df66b3c-abe2-4d20-abda-45320cd4af1b-kube-api-access-xxc6m\") pod \"node-exporter-jzxqq\" (UID: \"7df66b3c-abe2-4d20-abda-45320cd4af1b\") " pod="openshift-monitoring/node-exporter-jzxqq" Oct 30 20:38:30.988267 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:38:30.987516 5378 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sg8pj\" (UniqueName: \"kubernetes.io/projected/6a710ade-5a54-4595-89a8-636653c02fc5-kube-api-access-sg8pj\") pod \"ingress-canary-brbhg\" (UID: \"6a710ade-5a54-4595-89a8-636653c02fc5\") " pod="openshift-ingress-canary/ingress-canary-brbhg" Oct 30 20:38:30.988267 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:38:30.987533 5378 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/7df66b3c-abe2-4d20-abda-45320cd4af1b-node-exporter-textfile\") pod \"node-exporter-jzxqq\" (UID: \"7df66b3c-abe2-4d20-abda-45320cd4af1b\") " pod="openshift-monitoring/node-exporter-jzxqq" Oct 30 20:38:30.988267 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:38:30.987546 5378 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fqlw7\" (UniqueName: \"kubernetes.io/projected/6646b935-fe10-415a-be4b-a62a64ea45ec-kube-api-access-fqlw7\") pod \"console-6bfdc6f474-hj4hq\" (UID: \"6646b935-fe10-415a-be4b-a62a64ea45ec\") " pod="openshift-console/console-6bfdc6f474-hj4hq" Oct 30 20:38:30.988267 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:38:30.987589 5378 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/23527d33-4e73-4c36-ba42-7f0a5b7f62c6-lib-modules\") pod \"tuned-w9npq\" (UID: \"23527d33-4e73-4c36-ba42-7f0a5b7f62c6\") " pod="openshift-cluster-node-tuning-operator/tuned-w9npq" Oct 30 20:38:30.988267 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:38:30.987615 5378 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/c2ff359c-cf92-4a5b-8610-f6eb41acbe73-host-slash\") pod \"ovnkube-node-7x9jg\" (UID: \"c2ff359c-cf92-4a5b-8610-f6eb41acbe73\") " pod="openshift-ovn-kubernetes/ovnkube-node-7x9jg" Oct 30 20:38:30.988267 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:38:30.987637 5378 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/59406b2a-a784-4d6e-aedf-96e11ff99f68-crio-socket\") pod \"insights-runtime-extractor-q6mx7\" (UID: \"59406b2a-a784-4d6e-aedf-96e11ff99f68\") " pod="openshift-insights/insights-runtime-extractor-q6mx7" Oct 30 20:38:30.988267 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:38:30.987672 5378 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/59406b2a-a784-4d6e-aedf-96e11ff99f68-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-q6mx7\" (UID: \"59406b2a-a784-4d6e-aedf-96e11ff99f68\") " pod="openshift-insights/insights-runtime-extractor-q6mx7" Oct 30 20:38:30.988267 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:38:30.987698 5378 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/3bcf59e7-befb-4623-9a0f-f01b70963b3e-tmp\") pod \"thanos-querier-7d8d77f945-zfsgx\" (UID: \"3bcf59e7-befb-4623-9a0f-f01b70963b3e\") " pod="openshift-monitoring/thanos-querier-7d8d77f945-zfsgx" Oct 30 20:38:30.988267 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:38:30.987795 5378 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/3bcf59e7-befb-4623-9a0f-f01b70963b3e-tmp\") pod \"thanos-querier-7d8d77f945-zfsgx\" (UID: \"3bcf59e7-befb-4623-9a0f-f01b70963b3e\") " pod="openshift-monitoring/thanos-querier-7d8d77f945-zfsgx" Oct 30 20:38:30.988267 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:38:30.987861 5378 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/59406b2a-a784-4d6e-aedf-96e11ff99f68-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-q6mx7\" (UID: \"59406b2a-a784-4d6e-aedf-96e11ff99f68\") " pod="openshift-insights/insights-runtime-extractor-q6mx7" Oct 30 20:38:31.039099 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:38:31.039072 5378 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t5psr\" (UniqueName: \"kubernetes.io/projected/affdfb88-e208-4ca1-843d-efc8384c27a1-kube-api-access-t5psr\") pod \"iptables-alerter-kvq2l\" (UID: \"affdfb88-e208-4ca1-843d-efc8384c27a1\") " pod="openshift-network-operator/iptables-alerter-kvq2l" Oct 30 20:38:31.039498 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:38:31.039478 5378 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k82pb\" (UniqueName: \"kubernetes.io/projected/662e676d-c357-4ac8-a813-d34096051aff-kube-api-access-k82pb\") pod \"aws-ebs-csi-driver-node-svf4h\" (UID: \"662e676d-c357-4ac8-a813-d34096051aff\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-svf4h" Oct 30 20:38:31.039582 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:38:31.039548 5378 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fwkdk\" (UniqueName: \"kubernetes.io/projected/59406b2a-a784-4d6e-aedf-96e11ff99f68-kube-api-access-fwkdk\") pod \"insights-runtime-extractor-q6mx7\" (UID: \"59406b2a-a784-4d6e-aedf-96e11ff99f68\") " pod="openshift-insights/insights-runtime-extractor-q6mx7" Oct 30 20:38:31.039784 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:38:31.039768 5378 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jmxp5\" (UniqueName: \"kubernetes.io/projected/59046ab6-2f4c-4031-ba8c-21fb322a9d07-kube-api-access-jmxp5\") pod \"downloads-6dd789c8b7-5pslw\" (UID: \"59046ab6-2f4c-4031-ba8c-21fb322a9d07\") " pod="openshift-console/downloads-6dd789c8b7-5pslw" Oct 30 20:38:31.040332 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:38:31.040306 5378 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bc9kz\" (UniqueName: \"kubernetes.io/projected/567a72f6-035d-4c61-acaf-f46008029168-kube-api-access-bc9kz\") pod \"network-check-target-ws655\" (UID: \"567a72f6-035d-4c61-acaf-f46008029168\") " pod="openshift-network-diagnostics/network-check-target-ws655" Oct 30 20:38:31.040423 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:38:31.040403 5378 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xcs6c\" (UniqueName: \"kubernetes.io/projected/23527d33-4e73-4c36-ba42-7f0a5b7f62c6-kube-api-access-xcs6c\") pod \"tuned-w9npq\" (UID: \"23527d33-4e73-4c36-ba42-7f0a5b7f62c6\") " pod="openshift-cluster-node-tuning-operator/tuned-w9npq" Oct 30 20:38:31.040468 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:38:31.040455 5378 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5x2gc\" (UniqueName: \"kubernetes.io/projected/743b90bd-3330-4751-a853-80b97fee0890-kube-api-access-5x2gc\") pod \"dns-default-8wwgv\" (UID: \"743b90bd-3330-4751-a853-80b97fee0890\") " pod="openshift-dns/dns-default-8wwgv" Oct 30 20:38:31.040750 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:38:31.040734 5378 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-crg8q\" (UniqueName: \"kubernetes.io/projected/b7fa45b4-283a-4439-82b7-778abf2e9deb-kube-api-access-crg8q\") pod \"node-ca-tlch5\" (UID: \"b7fa45b4-283a-4439-82b7-778abf2e9deb\") " pod="openshift-image-registry/node-ca-tlch5" Oct 30 20:38:31.041191 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:38:31.041176 5378 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qs9cm\" (UniqueName: \"kubernetes.io/projected/910c0d9f-c79b-4839-9a1b-f2d278d11fdb-kube-api-access-qs9cm\") pod \"multus-additional-cni-plugins-blccx\" (UID: \"910c0d9f-c79b-4839-9a1b-f2d278d11fdb\") " pod="openshift-multus/multus-additional-cni-plugins-blccx" Oct 30 20:38:31.041803 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:38:31.041787 5378 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wk62v\" (UniqueName: \"kubernetes.io/projected/42765893-c3e3-41ab-811f-3135d4c74b07-kube-api-access-wk62v\") pod \"monitoring-plugin-77f4f76594-64rbk\" (UID: \"42765893-c3e3-41ab-811f-3135d4c74b07\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-64rbk" Oct 30 20:38:31.041996 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:38:31.041977 5378 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2w5c8\" (UniqueName: \"kubernetes.io/projected/666b6e91-4a60-45d9-9e3e-877867ded7fc-kube-api-access-2w5c8\") pod \"node-resolver-nv2rk\" (UID: \"666b6e91-4a60-45d9-9e3e-877867ded7fc\") " pod="openshift-dns/node-resolver-nv2rk" Oct 30 20:38:31.042081 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:38:31.042041 5378 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9d2s8\" (UniqueName: \"kubernetes.io/projected/5186676f-8c82-4667-a013-3efc22d2c4e7-kube-api-access-9d2s8\") pod \"prometheus-operator-9cd89b74c-2xw4j\" (UID: \"5186676f-8c82-4667-a013-3efc22d2c4e7\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-2xw4j" Oct 30 20:38:31.042755 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:38:31.042729 5378 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-28zhr\" (UniqueName: \"kubernetes.io/projected/e886897a-c6a3-414f-875f-aa112541b608-kube-api-access-28zhr\") pod \"network-metrics-daemon-nhs9k\" (UID: \"e886897a-c6a3-414f-875f-aa112541b608\") " pod="openshift-multus/network-metrics-daemon-nhs9k" Oct 30 20:38:31.042878 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:38:31.042751 5378 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r9qpr\" (UniqueName: \"kubernetes.io/projected/3bcf59e7-befb-4623-9a0f-f01b70963b3e-kube-api-access-r9qpr\") pod \"thanos-querier-7d8d77f945-zfsgx\" (UID: \"3bcf59e7-befb-4623-9a0f-f01b70963b3e\") " pod="openshift-monitoring/thanos-querier-7d8d77f945-zfsgx" Oct 30 20:38:31.043089 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:38:31.043068 5378 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-24qwx\" (UniqueName: \"kubernetes.io/projected/c2ff359c-cf92-4a5b-8610-f6eb41acbe73-kube-api-access-24qwx\") pod \"ovnkube-node-7x9jg\" (UID: \"c2ff359c-cf92-4a5b-8610-f6eb41acbe73\") " pod="openshift-ovn-kubernetes/ovnkube-node-7x9jg" Oct 30 20:38:31.046154 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:38:31.043369 5378 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fqlw7\" (UniqueName: \"kubernetes.io/projected/6646b935-fe10-415a-be4b-a62a64ea45ec-kube-api-access-fqlw7\") pod \"console-6bfdc6f474-hj4hq\" (UID: \"6646b935-fe10-415a-be4b-a62a64ea45ec\") " pod="openshift-console/console-6bfdc6f474-hj4hq" Oct 30 20:38:31.046154 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:38:31.043951 5378 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sg8pj\" (UniqueName: \"kubernetes.io/projected/6a710ade-5a54-4595-89a8-636653c02fc5-kube-api-access-sg8pj\") pod \"ingress-canary-brbhg\" (UID: \"6a710ade-5a54-4595-89a8-636653c02fc5\") " pod="openshift-ingress-canary/ingress-canary-brbhg" Oct 30 20:38:31.046154 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:38:31.045922 5378 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xxc6m\" (UniqueName: \"kubernetes.io/projected/7df66b3c-abe2-4d20-abda-45320cd4af1b-kube-api-access-xxc6m\") pod \"node-exporter-jzxqq\" (UID: \"7df66b3c-abe2-4d20-abda-45320cd4af1b\") " pod="openshift-monitoring/node-exporter-jzxqq" Oct 30 20:38:31.046899 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:38:31.046879 5378 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qjn9k\" (UniqueName: \"kubernetes.io/projected/022c4cf0-9797-4428-8fa3-9f3112f2ccc5-kube-api-access-qjn9k\") pod \"multus-dzbhx\" (UID: \"022c4cf0-9797-4428-8fa3-9f3112f2ccc5\") " pod="openshift-multus/multus-dzbhx" Oct 30 20:38:31.088966 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:38:31.088943 5378 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/662e676d-c357-4ac8-a813-d34096051aff-registration-dir\") pod \"aws-ebs-csi-driver-node-svf4h\" (UID: \"662e676d-c357-4ac8-a813-d34096051aff\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-svf4h" Oct 30 20:38:31.089036 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:38:31.088969 5378 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c2ff359c-cf92-4a5b-8610-f6eb41acbe73-host-run-ovn-kubernetes\") pod \"ovnkube-node-7x9jg\" (UID: \"c2ff359c-cf92-4a5b-8610-f6eb41acbe73\") " pod="openshift-ovn-kubernetes/ovnkube-node-7x9jg" Oct 30 20:38:31.089036 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:38:31.088986 5378 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/022c4cf0-9797-4428-8fa3-9f3112f2ccc5-host-run-multus-certs\") pod \"multus-dzbhx\" (UID: \"022c4cf0-9797-4428-8fa3-9f3112f2ccc5\") " pod="openshift-multus/multus-dzbhx" Oct 30 20:38:31.089036 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:38:31.089001 5378 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/7df66b3c-abe2-4d20-abda-45320cd4af1b-node-exporter-wtmp\") pod \"node-exporter-jzxqq\" (UID: \"7df66b3c-abe2-4d20-abda-45320cd4af1b\") " pod="openshift-monitoring/node-exporter-jzxqq" Oct 30 20:38:31.089036 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:38:31.089017 5378 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/022c4cf0-9797-4428-8fa3-9f3112f2ccc5-host-run-netns\") pod \"multus-dzbhx\" (UID: \"022c4cf0-9797-4428-8fa3-9f3112f2ccc5\") " pod="openshift-multus/multus-dzbhx" Oct 30 20:38:31.089036 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:38:31.089031 5378 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/7df66b3c-abe2-4d20-abda-45320cd4af1b-sys\") pod \"node-exporter-jzxqq\" (UID: \"7df66b3c-abe2-4d20-abda-45320cd4af1b\") " pod="openshift-monitoring/node-exporter-jzxqq" Oct 30 20:38:31.089262 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:38:31.089057 5378 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/662e676d-c357-4ac8-a813-d34096051aff-device-dir\") pod \"aws-ebs-csi-driver-node-svf4h\" (UID: \"662e676d-c357-4ac8-a813-d34096051aff\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-svf4h" Oct 30 20:38:31.089262 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:38:31.089062 5378 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/662e676d-c357-4ac8-a813-d34096051aff-registration-dir\") pod \"aws-ebs-csi-driver-node-svf4h\" (UID: \"662e676d-c357-4ac8-a813-d34096051aff\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-svf4h" Oct 30 20:38:31.089262 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:38:31.089078 5378 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/7df66b3c-abe2-4d20-abda-45320cd4af1b-node-exporter-wtmp\") pod \"node-exporter-jzxqq\" (UID: \"7df66b3c-abe2-4d20-abda-45320cd4af1b\") " pod="openshift-monitoring/node-exporter-jzxqq" Oct 30 20:38:31.089262 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:38:31.089090 5378 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/022c4cf0-9797-4428-8fa3-9f3112f2ccc5-host-run-multus-certs\") pod \"multus-dzbhx\" (UID: \"022c4cf0-9797-4428-8fa3-9f3112f2ccc5\") " pod="openshift-multus/multus-dzbhx" Oct 30 20:38:31.089262 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:38:31.089100 5378 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/022c4cf0-9797-4428-8fa3-9f3112f2ccc5-host-run-netns\") pod \"multus-dzbhx\" (UID: \"022c4cf0-9797-4428-8fa3-9f3112f2ccc5\") " pod="openshift-multus/multus-dzbhx" Oct 30 20:38:31.089262 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:38:31.089078 5378 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c2ff359c-cf92-4a5b-8610-f6eb41acbe73-host-run-ovn-kubernetes\") pod \"ovnkube-node-7x9jg\" (UID: \"c2ff359c-cf92-4a5b-8610-f6eb41acbe73\") " pod="openshift-ovn-kubernetes/ovnkube-node-7x9jg" Oct 30 20:38:31.089262 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:38:31.089120 5378 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/662e676d-c357-4ac8-a813-d34096051aff-device-dir\") pod \"aws-ebs-csi-driver-node-svf4h\" (UID: \"662e676d-c357-4ac8-a813-d34096051aff\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-svf4h" Oct 30 20:38:31.089262 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:38:31.089132 5378 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/022c4cf0-9797-4428-8fa3-9f3112f2ccc5-multus-cni-dir\") pod \"multus-dzbhx\" (UID: \"022c4cf0-9797-4428-8fa3-9f3112f2ccc5\") " pod="openshift-multus/multus-dzbhx" Oct 30 20:38:31.089262 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:38:31.089124 5378 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/7df66b3c-abe2-4d20-abda-45320cd4af1b-sys\") pod \"node-exporter-jzxqq\" (UID: \"7df66b3c-abe2-4d20-abda-45320cd4af1b\") " pod="openshift-monitoring/node-exporter-jzxqq" Oct 30 20:38:31.089262 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:38:31.089157 5378 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/0d6769f4-dc38-45ac-863b-c75fa672866a-kubelet-config\") pod \"global-pull-secret-syncer-bcmwc\" (UID: \"0d6769f4-dc38-45ac-863b-c75fa672866a\") " pod="kube-system/global-pull-secret-syncer-bcmwc" Oct 30 20:38:31.089262 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:38:31.089176 5378 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/c2ff359c-cf92-4a5b-8610-f6eb41acbe73-node-log\") pod \"ovnkube-node-7x9jg\" (UID: \"c2ff359c-cf92-4a5b-8610-f6eb41acbe73\") " pod="openshift-ovn-kubernetes/ovnkube-node-7x9jg" Oct 30 20:38:31.089262 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:38:31.089190 5378 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/022c4cf0-9797-4428-8fa3-9f3112f2ccc5-host-var-lib-kubelet\") pod \"multus-dzbhx\" (UID: \"022c4cf0-9797-4428-8fa3-9f3112f2ccc5\") " pod="openshift-multus/multus-dzbhx" Oct 30 20:38:31.089262 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:38:31.089207 5378 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/c2ff359c-cf92-4a5b-8610-f6eb41acbe73-run-openvswitch\") pod \"ovnkube-node-7x9jg\" (UID: \"c2ff359c-cf92-4a5b-8610-f6eb41acbe73\") " pod="openshift-ovn-kubernetes/ovnkube-node-7x9jg" Oct 30 20:38:31.089262 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:38:31.089215 5378 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/022c4cf0-9797-4428-8fa3-9f3112f2ccc5-multus-cni-dir\") pod \"multus-dzbhx\" (UID: \"022c4cf0-9797-4428-8fa3-9f3112f2ccc5\") " pod="openshift-multus/multus-dzbhx" Oct 30 20:38:31.089262 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:38:31.089245 5378 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/022c4cf0-9797-4428-8fa3-9f3112f2ccc5-host-var-lib-kubelet\") pod \"multus-dzbhx\" (UID: \"022c4cf0-9797-4428-8fa3-9f3112f2ccc5\") " pod="openshift-multus/multus-dzbhx" Oct 30 20:38:31.089262 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:38:31.089247 5378 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/c2ff359c-cf92-4a5b-8610-f6eb41acbe73-run-openvswitch\") pod \"ovnkube-node-7x9jg\" (UID: \"c2ff359c-cf92-4a5b-8610-f6eb41acbe73\") " pod="openshift-ovn-kubernetes/ovnkube-node-7x9jg" Oct 30 20:38:31.089262 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:38:31.089248 5378 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/c2ff359c-cf92-4a5b-8610-f6eb41acbe73-node-log\") pod \"ovnkube-node-7x9jg\" (UID: \"c2ff359c-cf92-4a5b-8610-f6eb41acbe73\") " pod="openshift-ovn-kubernetes/ovnkube-node-7x9jg" Oct 30 20:38:31.089262 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:38:31.089269 5378 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/23527d33-4e73-4c36-ba42-7f0a5b7f62c6-run\") pod \"tuned-w9npq\" (UID: \"23527d33-4e73-4c36-ba42-7f0a5b7f62c6\") " pod="openshift-cluster-node-tuning-operator/tuned-w9npq" Oct 30 20:38:31.090020 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:38:31.089292 5378 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/910c0d9f-c79b-4839-9a1b-f2d278d11fdb-cnibin\") pod \"multus-additional-cni-plugins-blccx\" (UID: \"910c0d9f-c79b-4839-9a1b-f2d278d11fdb\") " pod="openshift-multus/multus-additional-cni-plugins-blccx" Oct 30 20:38:31.090020 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:38:31.089262 5378 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/0d6769f4-dc38-45ac-863b-c75fa672866a-kubelet-config\") pod \"global-pull-secret-syncer-bcmwc\" (UID: \"0d6769f4-dc38-45ac-863b-c75fa672866a\") " pod="kube-system/global-pull-secret-syncer-bcmwc" Oct 30 20:38:31.090020 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:38:31.089308 5378 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/022c4cf0-9797-4428-8fa3-9f3112f2ccc5-host-var-lib-cni-bin\") pod \"multus-dzbhx\" (UID: \"022c4cf0-9797-4428-8fa3-9f3112f2ccc5\") " pod="openshift-multus/multus-dzbhx" Oct 30 20:38:31.090020 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:38:31.089331 5378 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/910c0d9f-c79b-4839-9a1b-f2d278d11fdb-cnibin\") pod \"multus-additional-cni-plugins-blccx\" (UID: \"910c0d9f-c79b-4839-9a1b-f2d278d11fdb\") " pod="openshift-multus/multus-additional-cni-plugins-blccx" Oct 30 20:38:31.090020 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:38:31.089335 5378 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/23527d33-4e73-4c36-ba42-7f0a5b7f62c6-run\") pod \"tuned-w9npq\" (UID: \"23527d33-4e73-4c36-ba42-7f0a5b7f62c6\") " pod="openshift-cluster-node-tuning-operator/tuned-w9npq" Oct 30 20:38:31.090020 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:38:31.089346 5378 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/666b6e91-4a60-45d9-9e3e-877867ded7fc-hosts-file\") pod \"node-resolver-nv2rk\" (UID: \"666b6e91-4a60-45d9-9e3e-877867ded7fc\") " pod="openshift-dns/node-resolver-nv2rk" Oct 30 20:38:31.090020 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:38:31.089362 5378 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/23527d33-4e73-4c36-ba42-7f0a5b7f62c6-etc-sysctl-conf\") pod \"tuned-w9npq\" (UID: \"23527d33-4e73-4c36-ba42-7f0a5b7f62c6\") " pod="openshift-cluster-node-tuning-operator/tuned-w9npq" Oct 30 20:38:31.090020 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:38:31.089377 5378 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/c2ff359c-cf92-4a5b-8610-f6eb41acbe73-host-run-netns\") pod \"ovnkube-node-7x9jg\" (UID: \"c2ff359c-cf92-4a5b-8610-f6eb41acbe73\") " pod="openshift-ovn-kubernetes/ovnkube-node-7x9jg" Oct 30 20:38:31.090020 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:38:31.089391 5378 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/022c4cf0-9797-4428-8fa3-9f3112f2ccc5-system-cni-dir\") pod \"multus-dzbhx\" (UID: \"022c4cf0-9797-4428-8fa3-9f3112f2ccc5\") " pod="openshift-multus/multus-dzbhx" Oct 30 20:38:31.090020 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:38:31.089406 5378 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/022c4cf0-9797-4428-8fa3-9f3112f2ccc5-host-run-k8s-cni-cncf-io\") pod \"multus-dzbhx\" (UID: \"022c4cf0-9797-4428-8fa3-9f3112f2ccc5\") " pod="openshift-multus/multus-dzbhx" Oct 30 20:38:31.090020 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:38:31.089420 5378 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/022c4cf0-9797-4428-8fa3-9f3112f2ccc5-multus-conf-dir\") pod \"multus-dzbhx\" (UID: \"022c4cf0-9797-4428-8fa3-9f3112f2ccc5\") " pod="openshift-multus/multus-dzbhx" Oct 30 20:38:31.090020 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:38:31.089338 5378 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/022c4cf0-9797-4428-8fa3-9f3112f2ccc5-host-var-lib-cni-bin\") pod \"multus-dzbhx\" (UID: \"022c4cf0-9797-4428-8fa3-9f3112f2ccc5\") " pod="openshift-multus/multus-dzbhx" Oct 30 20:38:31.090020 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:38:31.089437 5378 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/c2ff359c-cf92-4a5b-8610-f6eb41acbe73-host-run-netns\") pod \"ovnkube-node-7x9jg\" (UID: \"c2ff359c-cf92-4a5b-8610-f6eb41acbe73\") " pod="openshift-ovn-kubernetes/ovnkube-node-7x9jg" Oct 30 20:38:31.090020 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:38:31.089441 5378 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/c2ff359c-cf92-4a5b-8610-f6eb41acbe73-systemd-units\") pod \"ovnkube-node-7x9jg\" (UID: \"c2ff359c-cf92-4a5b-8610-f6eb41acbe73\") " pod="openshift-ovn-kubernetes/ovnkube-node-7x9jg" Oct 30 20:38:31.090020 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:38:31.089452 5378 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/022c4cf0-9797-4428-8fa3-9f3112f2ccc5-system-cni-dir\") pod \"multus-dzbhx\" (UID: \"022c4cf0-9797-4428-8fa3-9f3112f2ccc5\") " pod="openshift-multus/multus-dzbhx" Oct 30 20:38:31.090020 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:38:31.089461 5378 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/666b6e91-4a60-45d9-9e3e-877867ded7fc-hosts-file\") pod \"node-resolver-nv2rk\" (UID: \"666b6e91-4a60-45d9-9e3e-877867ded7fc\") " pod="openshift-dns/node-resolver-nv2rk" Oct 30 20:38:31.090020 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:38:31.089474 5378 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/662e676d-c357-4ac8-a813-d34096051aff-sys-fs\") pod \"aws-ebs-csi-driver-node-svf4h\" (UID: \"662e676d-c357-4ac8-a813-d34096051aff\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-svf4h" Oct 30 20:38:31.090020 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:38:31.089478 5378 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/022c4cf0-9797-4428-8fa3-9f3112f2ccc5-host-run-k8s-cni-cncf-io\") pod \"multus-dzbhx\" (UID: \"022c4cf0-9797-4428-8fa3-9f3112f2ccc5\") " pod="openshift-multus/multus-dzbhx" Oct 30 20:38:31.090020 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:38:31.089494 5378 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/c2ff359c-cf92-4a5b-8610-f6eb41acbe73-log-socket\") pod \"ovnkube-node-7x9jg\" (UID: \"c2ff359c-cf92-4a5b-8610-f6eb41acbe73\") " pod="openshift-ovn-kubernetes/ovnkube-node-7x9jg" Oct 30 20:38:31.090020 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:38:31.089504 5378 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/23527d33-4e73-4c36-ba42-7f0a5b7f62c6-etc-sysctl-conf\") pod \"tuned-w9npq\" (UID: \"23527d33-4e73-4c36-ba42-7f0a5b7f62c6\") " pod="openshift-cluster-node-tuning-operator/tuned-w9npq" Oct 30 20:38:31.090020 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:38:31.089510 5378 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/662e676d-c357-4ac8-a813-d34096051aff-socket-dir\") pod \"aws-ebs-csi-driver-node-svf4h\" (UID: \"662e676d-c357-4ac8-a813-d34096051aff\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-svf4h" Oct 30 20:38:31.090020 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:38:31.089508 5378 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/c2ff359c-cf92-4a5b-8610-f6eb41acbe73-systemd-units\") pod \"ovnkube-node-7x9jg\" (UID: \"c2ff359c-cf92-4a5b-8610-f6eb41acbe73\") " pod="openshift-ovn-kubernetes/ovnkube-node-7x9jg" Oct 30 20:38:31.090020 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:38:31.089543 5378 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/23527d33-4e73-4c36-ba42-7f0a5b7f62c6-host\") pod \"tuned-w9npq\" (UID: \"23527d33-4e73-4c36-ba42-7f0a5b7f62c6\") " pod="openshift-cluster-node-tuning-operator/tuned-w9npq" Oct 30 20:38:31.090020 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:38:31.089548 5378 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/c2ff359c-cf92-4a5b-8610-f6eb41acbe73-log-socket\") pod \"ovnkube-node-7x9jg\" (UID: \"c2ff359c-cf92-4a5b-8610-f6eb41acbe73\") " pod="openshift-ovn-kubernetes/ovnkube-node-7x9jg" Oct 30 20:38:31.090020 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:38:31.089553 5378 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/662e676d-c357-4ac8-a813-d34096051aff-socket-dir\") pod \"aws-ebs-csi-driver-node-svf4h\" (UID: \"662e676d-c357-4ac8-a813-d34096051aff\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-svf4h" Oct 30 20:38:31.090020 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:38:31.089551 5378 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/022c4cf0-9797-4428-8fa3-9f3112f2ccc5-multus-conf-dir\") pod \"multus-dzbhx\" (UID: \"022c4cf0-9797-4428-8fa3-9f3112f2ccc5\") " pod="openshift-multus/multus-dzbhx" Oct 30 20:38:31.090020 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:38:31.089585 5378 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/662e676d-c357-4ac8-a813-d34096051aff-etc-selinux\") pod \"aws-ebs-csi-driver-node-svf4h\" (UID: \"662e676d-c357-4ac8-a813-d34096051aff\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-svf4h" Oct 30 20:38:31.090020 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:38:31.089592 5378 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/662e676d-c357-4ac8-a813-d34096051aff-sys-fs\") pod \"aws-ebs-csi-driver-node-svf4h\" (UID: \"662e676d-c357-4ac8-a813-d34096051aff\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-svf4h" Oct 30 20:38:31.090020 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:38:31.089602 5378 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/23527d33-4e73-4c36-ba42-7f0a5b7f62c6-host\") pod \"tuned-w9npq\" (UID: \"23527d33-4e73-4c36-ba42-7f0a5b7f62c6\") " pod="openshift-cluster-node-tuning-operator/tuned-w9npq" Oct 30 20:38:31.090020 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:38:31.089610 5378 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/23527d33-4e73-4c36-ba42-7f0a5b7f62c6-etc-kubernetes\") pod \"tuned-w9npq\" (UID: \"23527d33-4e73-4c36-ba42-7f0a5b7f62c6\") " pod="openshift-cluster-node-tuning-operator/tuned-w9npq" Oct 30 20:38:31.090020 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:38:31.089629 5378 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/662e676d-c357-4ac8-a813-d34096051aff-etc-selinux\") pod \"aws-ebs-csi-driver-node-svf4h\" (UID: \"662e676d-c357-4ac8-a813-d34096051aff\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-svf4h" Oct 30 20:38:31.090020 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:38:31.089631 5378 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/23527d33-4e73-4c36-ba42-7f0a5b7f62c6-var-lib-kubelet\") pod \"tuned-w9npq\" (UID: \"23527d33-4e73-4c36-ba42-7f0a5b7f62c6\") " pod="openshift-cluster-node-tuning-operator/tuned-w9npq" Oct 30 20:38:31.090020 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:38:31.089657 5378 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/23527d33-4e73-4c36-ba42-7f0a5b7f62c6-var-lib-kubelet\") pod \"tuned-w9npq\" (UID: \"23527d33-4e73-4c36-ba42-7f0a5b7f62c6\") " pod="openshift-cluster-node-tuning-operator/tuned-w9npq" Oct 30 20:38:31.090020 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:38:31.089670 5378 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/022c4cf0-9797-4428-8fa3-9f3112f2ccc5-etc-kubernetes\") pod \"multus-dzbhx\" (UID: \"022c4cf0-9797-4428-8fa3-9f3112f2ccc5\") " pod="openshift-multus/multus-dzbhx" Oct 30 20:38:31.090020 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:38:31.089677 5378 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/23527d33-4e73-4c36-ba42-7f0a5b7f62c6-etc-kubernetes\") pod \"tuned-w9npq\" (UID: \"23527d33-4e73-4c36-ba42-7f0a5b7f62c6\") " pod="openshift-cluster-node-tuning-operator/tuned-w9npq" Oct 30 20:38:31.090020 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:38:31.089695 5378 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/022c4cf0-9797-4428-8fa3-9f3112f2ccc5-host-var-lib-cni-multus\") pod \"multus-dzbhx\" (UID: \"022c4cf0-9797-4428-8fa3-9f3112f2ccc5\") " pod="openshift-multus/multus-dzbhx" Oct 30 20:38:31.090020 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:38:31.089720 5378 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/662e676d-c357-4ac8-a813-d34096051aff-kubelet-dir\") pod \"aws-ebs-csi-driver-node-svf4h\" (UID: \"662e676d-c357-4ac8-a813-d34096051aff\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-svf4h" Oct 30 20:38:31.090020 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:38:31.089744 5378 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/affdfb88-e208-4ca1-843d-efc8384c27a1-host-slash\") pod \"iptables-alerter-kvq2l\" (UID: \"affdfb88-e208-4ca1-843d-efc8384c27a1\") " pod="openshift-network-operator/iptables-alerter-kvq2l" Oct 30 20:38:31.090020 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:38:31.089748 5378 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/022c4cf0-9797-4428-8fa3-9f3112f2ccc5-etc-kubernetes\") pod \"multus-dzbhx\" (UID: \"022c4cf0-9797-4428-8fa3-9f3112f2ccc5\") " pod="openshift-multus/multus-dzbhx" Oct 30 20:38:31.090020 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:38:31.089766 5378 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/7df66b3c-abe2-4d20-abda-45320cd4af1b-root\") pod \"node-exporter-jzxqq\" (UID: \"7df66b3c-abe2-4d20-abda-45320cd4af1b\") " pod="openshift-monitoring/node-exporter-jzxqq" Oct 30 20:38:31.090020 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:38:31.089780 5378 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/662e676d-c357-4ac8-a813-d34096051aff-kubelet-dir\") pod \"aws-ebs-csi-driver-node-svf4h\" (UID: \"662e676d-c357-4ac8-a813-d34096051aff\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-svf4h" Oct 30 20:38:31.090020 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:38:31.089752 5378 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/022c4cf0-9797-4428-8fa3-9f3112f2ccc5-host-var-lib-cni-multus\") pod \"multus-dzbhx\" (UID: \"022c4cf0-9797-4428-8fa3-9f3112f2ccc5\") " pod="openshift-multus/multus-dzbhx" Oct 30 20:38:31.090020 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:38:31.089790 5378 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/23527d33-4e73-4c36-ba42-7f0a5b7f62c6-etc-systemd\") pod \"tuned-w9npq\" (UID: \"23527d33-4e73-4c36-ba42-7f0a5b7f62c6\") " pod="openshift-cluster-node-tuning-operator/tuned-w9npq" Oct 30 20:38:31.090020 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:38:31.089812 5378 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/23527d33-4e73-4c36-ba42-7f0a5b7f62c6-sys\") pod \"tuned-w9npq\" (UID: \"23527d33-4e73-4c36-ba42-7f0a5b7f62c6\") " pod="openshift-cluster-node-tuning-operator/tuned-w9npq" Oct 30 20:38:31.090020 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:38:31.089817 5378 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/7df66b3c-abe2-4d20-abda-45320cd4af1b-root\") pod \"node-exporter-jzxqq\" (UID: \"7df66b3c-abe2-4d20-abda-45320cd4af1b\") " pod="openshift-monitoring/node-exporter-jzxqq" Oct 30 20:38:31.090020 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:38:31.089836 5378 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/c2ff359c-cf92-4a5b-8610-f6eb41acbe73-run-systemd\") pod \"ovnkube-node-7x9jg\" (UID: \"c2ff359c-cf92-4a5b-8610-f6eb41acbe73\") " pod="openshift-ovn-kubernetes/ovnkube-node-7x9jg" Oct 30 20:38:31.090020 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:38:31.089853 5378 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/affdfb88-e208-4ca1-843d-efc8384c27a1-host-slash\") pod \"iptables-alerter-kvq2l\" (UID: \"affdfb88-e208-4ca1-843d-efc8384c27a1\") " pod="openshift-network-operator/iptables-alerter-kvq2l" Oct 30 20:38:31.090020 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:38:31.089863 5378 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/23527d33-4e73-4c36-ba42-7f0a5b7f62c6-etc-systemd\") pod \"tuned-w9npq\" (UID: \"23527d33-4e73-4c36-ba42-7f0a5b7f62c6\") " pod="openshift-cluster-node-tuning-operator/tuned-w9npq" Oct 30 20:38:31.090020 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:38:31.089865 5378 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/23527d33-4e73-4c36-ba42-7f0a5b7f62c6-sys\") pod \"tuned-w9npq\" (UID: \"23527d33-4e73-4c36-ba42-7f0a5b7f62c6\") " pod="openshift-cluster-node-tuning-operator/tuned-w9npq" Oct 30 20:38:31.090020 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:38:31.089866 5378 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/c2ff359c-cf92-4a5b-8610-f6eb41acbe73-host-cni-bin\") pod \"ovnkube-node-7x9jg\" (UID: \"c2ff359c-cf92-4a5b-8610-f6eb41acbe73\") " pod="openshift-ovn-kubernetes/ovnkube-node-7x9jg" Oct 30 20:38:31.090020 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:38:31.089890 5378 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/c2ff359c-cf92-4a5b-8610-f6eb41acbe73-run-systemd\") pod \"ovnkube-node-7x9jg\" (UID: \"c2ff359c-cf92-4a5b-8610-f6eb41acbe73\") " pod="openshift-ovn-kubernetes/ovnkube-node-7x9jg" Oct 30 20:38:31.090020 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:38:31.089898 5378 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/c2ff359c-cf92-4a5b-8610-f6eb41acbe73-host-cni-bin\") pod \"ovnkube-node-7x9jg\" (UID: \"c2ff359c-cf92-4a5b-8610-f6eb41acbe73\") " pod="openshift-ovn-kubernetes/ovnkube-node-7x9jg" Oct 30 20:38:31.090020 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:38:31.089904 5378 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/b7fa45b4-283a-4439-82b7-778abf2e9deb-host\") pod \"node-ca-tlch5\" (UID: \"b7fa45b4-283a-4439-82b7-778abf2e9deb\") " pod="openshift-image-registry/node-ca-tlch5" Oct 30 20:38:31.090020 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:38:31.089923 5378 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/23527d33-4e73-4c36-ba42-7f0a5b7f62c6-etc-modprobe-d\") pod \"tuned-w9npq\" (UID: \"23527d33-4e73-4c36-ba42-7f0a5b7f62c6\") " pod="openshift-cluster-node-tuning-operator/tuned-w9npq" Oct 30 20:38:31.090020 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:38:31.089938 5378 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/b7fa45b4-283a-4439-82b7-778abf2e9deb-host\") pod \"node-ca-tlch5\" (UID: \"b7fa45b4-283a-4439-82b7-778abf2e9deb\") " pod="openshift-image-registry/node-ca-tlch5" Oct 30 20:38:31.090020 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:38:31.089938 5378 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/c2ff359c-cf92-4a5b-8610-f6eb41acbe73-host-kubelet\") pod \"ovnkube-node-7x9jg\" (UID: \"c2ff359c-cf92-4a5b-8610-f6eb41acbe73\") " pod="openshift-ovn-kubernetes/ovnkube-node-7x9jg" Oct 30 20:38:31.090020 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:38:31.089955 5378 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/c2ff359c-cf92-4a5b-8610-f6eb41acbe73-host-kubelet\") pod \"ovnkube-node-7x9jg\" (UID: \"c2ff359c-cf92-4a5b-8610-f6eb41acbe73\") " pod="openshift-ovn-kubernetes/ovnkube-node-7x9jg" Oct 30 20:38:31.090020 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:38:31.089963 5378 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/022c4cf0-9797-4428-8fa3-9f3112f2ccc5-multus-socket-dir-parent\") pod \"multus-dzbhx\" (UID: \"022c4cf0-9797-4428-8fa3-9f3112f2ccc5\") " pod="openshift-multus/multus-dzbhx" Oct 30 20:38:31.090020 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:38:31.089980 5378 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/23527d33-4e73-4c36-ba42-7f0a5b7f62c6-etc-sysconfig\") pod \"tuned-w9npq\" (UID: \"23527d33-4e73-4c36-ba42-7f0a5b7f62c6\") " pod="openshift-cluster-node-tuning-operator/tuned-w9npq" Oct 30 20:38:31.090020 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:38:31.090000 5378 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c2ff359c-cf92-4a5b-8610-f6eb41acbe73-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-7x9jg\" (UID: \"c2ff359c-cf92-4a5b-8610-f6eb41acbe73\") " pod="openshift-ovn-kubernetes/ovnkube-node-7x9jg" Oct 30 20:38:31.090020 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:38:31.090015 5378 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/022c4cf0-9797-4428-8fa3-9f3112f2ccc5-multus-socket-dir-parent\") pod \"multus-dzbhx\" (UID: \"022c4cf0-9797-4428-8fa3-9f3112f2ccc5\") " pod="openshift-multus/multus-dzbhx" Oct 30 20:38:31.090020 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:38:31.090024 5378 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/23527d33-4e73-4c36-ba42-7f0a5b7f62c6-etc-modprobe-d\") pod \"tuned-w9npq\" (UID: \"23527d33-4e73-4c36-ba42-7f0a5b7f62c6\") " pod="openshift-cluster-node-tuning-operator/tuned-w9npq" Oct 30 20:38:31.090020 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:38:31.090019 5378 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/c2ff359c-cf92-4a5b-8610-f6eb41acbe73-var-lib-openvswitch\") pod \"ovnkube-node-7x9jg\" (UID: \"c2ff359c-cf92-4a5b-8610-f6eb41acbe73\") " pod="openshift-ovn-kubernetes/ovnkube-node-7x9jg" Oct 30 20:38:31.090020 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:38:31.090036 5378 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/c2ff359c-cf92-4a5b-8610-f6eb41acbe73-var-lib-openvswitch\") pod \"ovnkube-node-7x9jg\" (UID: \"c2ff359c-cf92-4a5b-8610-f6eb41acbe73\") " pod="openshift-ovn-kubernetes/ovnkube-node-7x9jg" Oct 30 20:38:31.090020 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:38:31.090030 5378 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/23527d33-4e73-4c36-ba42-7f0a5b7f62c6-etc-sysconfig\") pod \"tuned-w9npq\" (UID: \"23527d33-4e73-4c36-ba42-7f0a5b7f62c6\") " pod="openshift-cluster-node-tuning-operator/tuned-w9npq" Oct 30 20:38:31.090020 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:38:31.090048 5378 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c2ff359c-cf92-4a5b-8610-f6eb41acbe73-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-7x9jg\" (UID: \"c2ff359c-cf92-4a5b-8610-f6eb41acbe73\") " pod="openshift-ovn-kubernetes/ovnkube-node-7x9jg" Oct 30 20:38:31.090020 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:38:31.090050 5378 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/c2ff359c-cf92-4a5b-8610-f6eb41acbe73-etc-openvswitch\") pod \"ovnkube-node-7x9jg\" (UID: \"c2ff359c-cf92-4a5b-8610-f6eb41acbe73\") " pod="openshift-ovn-kubernetes/ovnkube-node-7x9jg" Oct 30 20:38:31.091931 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:38:31.090067 5378 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/c2ff359c-cf92-4a5b-8610-f6eb41acbe73-etc-openvswitch\") pod \"ovnkube-node-7x9jg\" (UID: \"c2ff359c-cf92-4a5b-8610-f6eb41acbe73\") " pod="openshift-ovn-kubernetes/ovnkube-node-7x9jg" Oct 30 20:38:31.091931 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:38:31.090072 5378 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/c2ff359c-cf92-4a5b-8610-f6eb41acbe73-run-ovn\") pod \"ovnkube-node-7x9jg\" (UID: \"c2ff359c-cf92-4a5b-8610-f6eb41acbe73\") " pod="openshift-ovn-kubernetes/ovnkube-node-7x9jg" Oct 30 20:38:31.091931 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:38:31.090087 5378 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/910c0d9f-c79b-4839-9a1b-f2d278d11fdb-tuning-conf-dir\") pod \"multus-additional-cni-plugins-blccx\" (UID: \"910c0d9f-c79b-4839-9a1b-f2d278d11fdb\") " pod="openshift-multus/multus-additional-cni-plugins-blccx" Oct 30 20:38:31.091931 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:38:31.090100 5378 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/0d6769f4-dc38-45ac-863b-c75fa672866a-dbus\") pod \"global-pull-secret-syncer-bcmwc\" (UID: \"0d6769f4-dc38-45ac-863b-c75fa672866a\") " pod="kube-system/global-pull-secret-syncer-bcmwc" Oct 30 20:38:31.091931 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:38:31.090114 5378 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/022c4cf0-9797-4428-8fa3-9f3112f2ccc5-hostroot\") pod \"multus-dzbhx\" (UID: \"022c4cf0-9797-4428-8fa3-9f3112f2ccc5\") " pod="openshift-multus/multus-dzbhx" Oct 30 20:38:31.091931 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:38:31.090130 5378 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/23527d33-4e73-4c36-ba42-7f0a5b7f62c6-lib-modules\") pod \"tuned-w9npq\" (UID: \"23527d33-4e73-4c36-ba42-7f0a5b7f62c6\") " pod="openshift-cluster-node-tuning-operator/tuned-w9npq" Oct 30 20:38:31.091931 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:38:31.090115 5378 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/c2ff359c-cf92-4a5b-8610-f6eb41acbe73-run-ovn\") pod \"ovnkube-node-7x9jg\" (UID: \"c2ff359c-cf92-4a5b-8610-f6eb41acbe73\") " pod="openshift-ovn-kubernetes/ovnkube-node-7x9jg" Oct 30 20:38:31.091931 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:38:31.090143 5378 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/910c0d9f-c79b-4839-9a1b-f2d278d11fdb-tuning-conf-dir\") pod \"multus-additional-cni-plugins-blccx\" (UID: \"910c0d9f-c79b-4839-9a1b-f2d278d11fdb\") " pod="openshift-multus/multus-additional-cni-plugins-blccx" Oct 30 20:38:31.091931 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:38:31.090151 5378 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/0d6769f4-dc38-45ac-863b-c75fa672866a-dbus\") pod \"global-pull-secret-syncer-bcmwc\" (UID: \"0d6769f4-dc38-45ac-863b-c75fa672866a\") " pod="kube-system/global-pull-secret-syncer-bcmwc" Oct 30 20:38:31.091931 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:38:31.090151 5378 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/c2ff359c-cf92-4a5b-8610-f6eb41acbe73-host-slash\") pod \"ovnkube-node-7x9jg\" (UID: \"c2ff359c-cf92-4a5b-8610-f6eb41acbe73\") " pod="openshift-ovn-kubernetes/ovnkube-node-7x9jg" Oct 30 20:38:31.091931 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:38:31.090167 5378 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/c2ff359c-cf92-4a5b-8610-f6eb41acbe73-host-slash\") pod \"ovnkube-node-7x9jg\" (UID: \"c2ff359c-cf92-4a5b-8610-f6eb41acbe73\") " pod="openshift-ovn-kubernetes/ovnkube-node-7x9jg" Oct 30 20:38:31.091931 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:38:31.090174 5378 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/022c4cf0-9797-4428-8fa3-9f3112f2ccc5-hostroot\") pod \"multus-dzbhx\" (UID: \"022c4cf0-9797-4428-8fa3-9f3112f2ccc5\") " pod="openshift-multus/multus-dzbhx" Oct 30 20:38:31.091931 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:38:31.090177 5378 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/59406b2a-a784-4d6e-aedf-96e11ff99f68-crio-socket\") pod \"insights-runtime-extractor-q6mx7\" (UID: \"59406b2a-a784-4d6e-aedf-96e11ff99f68\") " pod="openshift-insights/insights-runtime-extractor-q6mx7" Oct 30 20:38:31.091931 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:38:31.090194 5378 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/23527d33-4e73-4c36-ba42-7f0a5b7f62c6-etc-sysctl-d\") pod \"tuned-w9npq\" (UID: \"23527d33-4e73-4c36-ba42-7f0a5b7f62c6\") " pod="openshift-cluster-node-tuning-operator/tuned-w9npq" Oct 30 20:38:31.091931 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:38:31.090215 5378 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/910c0d9f-c79b-4839-9a1b-f2d278d11fdb-system-cni-dir\") pod \"multus-additional-cni-plugins-blccx\" (UID: \"910c0d9f-c79b-4839-9a1b-f2d278d11fdb\") " pod="openshift-multus/multus-additional-cni-plugins-blccx" Oct 30 20:38:31.091931 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:38:31.090233 5378 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/23527d33-4e73-4c36-ba42-7f0a5b7f62c6-lib-modules\") pod \"tuned-w9npq\" (UID: \"23527d33-4e73-4c36-ba42-7f0a5b7f62c6\") " pod="openshift-cluster-node-tuning-operator/tuned-w9npq" Oct 30 20:38:31.091931 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:38:31.090240 5378 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/59406b2a-a784-4d6e-aedf-96e11ff99f68-crio-socket\") pod \"insights-runtime-extractor-q6mx7\" (UID: \"59406b2a-a784-4d6e-aedf-96e11ff99f68\") " pod="openshift-insights/insights-runtime-extractor-q6mx7" Oct 30 20:38:31.091931 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:38:31.090242 5378 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/c2ff359c-cf92-4a5b-8610-f6eb41acbe73-host-cni-netd\") pod \"ovnkube-node-7x9jg\" (UID: \"c2ff359c-cf92-4a5b-8610-f6eb41acbe73\") " pod="openshift-ovn-kubernetes/ovnkube-node-7x9jg" Oct 30 20:38:31.091931 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:38:31.090279 5378 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/c2ff359c-cf92-4a5b-8610-f6eb41acbe73-host-cni-netd\") pod \"ovnkube-node-7x9jg\" (UID: \"c2ff359c-cf92-4a5b-8610-f6eb41acbe73\") " pod="openshift-ovn-kubernetes/ovnkube-node-7x9jg" Oct 30 20:38:31.091931 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:38:31.090300 5378 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/23527d33-4e73-4c36-ba42-7f0a5b7f62c6-etc-sysctl-d\") pod \"tuned-w9npq\" (UID: \"23527d33-4e73-4c36-ba42-7f0a5b7f62c6\") " pod="openshift-cluster-node-tuning-operator/tuned-w9npq" Oct 30 20:38:31.091931 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:38:31.090321 5378 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/910c0d9f-c79b-4839-9a1b-f2d278d11fdb-system-cni-dir\") pod \"multus-additional-cni-plugins-blccx\" (UID: \"910c0d9f-c79b-4839-9a1b-f2d278d11fdb\") " pod="openshift-multus/multus-additional-cni-plugins-blccx" Oct 30 20:38:31.091931 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:38:31.090337 5378 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/022c4cf0-9797-4428-8fa3-9f3112f2ccc5-cnibin\") pod \"multus-dzbhx\" (UID: \"022c4cf0-9797-4428-8fa3-9f3112f2ccc5\") " pod="openshift-multus/multus-dzbhx" Oct 30 20:38:31.091931 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:38:31.090387 5378 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/022c4cf0-9797-4428-8fa3-9f3112f2ccc5-cnibin\") pod \"multus-dzbhx\" (UID: \"022c4cf0-9797-4428-8fa3-9f3112f2ccc5\") " pod="openshift-multus/multus-dzbhx" Oct 30 20:38:31.120137 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:38:31.120120 5378 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 30 20:38:31.644679 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:38:31.644647 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 20:38:32.422048 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:38:32.422010 5378 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-ws655" Oct 30 20:38:32.533973 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:38:32.533940 5378 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/thanos-querier-7d8d77f945-zfsgx" Oct 30 20:38:32.539424 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:38:32.539339 5378 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/thanos-querier-7d8d77f945-zfsgx" Oct 30 20:38:32.818985 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:38:32.818901 5378 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6bfdc6f474-hj4hq" event={"ID":"6646b935-fe10-415a-be4b-a62a64ea45ec","Type":"ContainerStarted","Data":"c71886b8af6a606b11086a38e78a74cdf2f65ee75709915297f0edbc2c4c2429"} Oct 30 20:38:33.538558 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:38:33.538523 5378 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-6bfdc6f474-hj4hq" Oct 30 20:38:33.613085 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:38:33.613048 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 20:38:33.741188 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:38:33.741164 5378 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-8wwgv" Oct 30 20:38:33.755884 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:38:33.755857 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 20:38:33.892215 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:38:33.892175 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 20:38:34.885518 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:38:34.885482 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 20:38:35.008640 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:38:35.008599 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 20:38:35.079671 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:38:35.079643 5378 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-7x9jg" Oct 30 20:38:35.283360 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:38:35.283281 5378 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-7x9jg" Oct 30 20:38:35.698379 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:38:35.698327 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 20:38:35.831603 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:38:35.831560 5378 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Oct 30 20:38:35.859373 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:38:35.859340 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 20:38:36.486313 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:38:36.486280 5378 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/monitoring-plugin-77f4f76594-64rbk" Oct 30 20:38:36.492665 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:38:36.492641 5378 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/monitoring-plugin-77f4f76594-64rbk" Oct 30 20:38:37.286543 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:38:37.286505 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 20:38:37.419584 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:38:37.419542 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 20:38:38.009377 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:38:38.009347 5378 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console/console-7759588d67-q7dwh"] Oct 30 20:38:38.011503 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:38:38.011473 5378 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7759588d67-q7dwh" Oct 30 20:38:38.040721 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:38:38.040693 5378 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sfn6b\" (UniqueName: \"kubernetes.io/projected/ffc2a268-b4b5-40f3-8074-d7dd2d54eb05-kube-api-access-sfn6b\") pod \"console-7759588d67-q7dwh\" (UID: \"ffc2a268-b4b5-40f3-8074-d7dd2d54eb05\") " pod="openshift-console/console-7759588d67-q7dwh" Oct 30 20:38:38.040833 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:38:38.040737 5378 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/ffc2a268-b4b5-40f3-8074-d7dd2d54eb05-console-oauth-config\") pod \"console-7759588d67-q7dwh\" (UID: \"ffc2a268-b4b5-40f3-8074-d7dd2d54eb05\") " pod="openshift-console/console-7759588d67-q7dwh" Oct 30 20:38:38.040833 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:38:38.040761 5378 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/ffc2a268-b4b5-40f3-8074-d7dd2d54eb05-oauth-serving-cert\") pod \"console-7759588d67-q7dwh\" (UID: \"ffc2a268-b4b5-40f3-8074-d7dd2d54eb05\") " pod="openshift-console/console-7759588d67-q7dwh" Oct 30 20:38:38.040945 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:38:38.040914 5378 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/ffc2a268-b4b5-40f3-8074-d7dd2d54eb05-console-serving-cert\") pod \"console-7759588d67-q7dwh\" (UID: \"ffc2a268-b4b5-40f3-8074-d7dd2d54eb05\") " pod="openshift-console/console-7759588d67-q7dwh" Oct 30 20:38:38.040945 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:38:38.040940 5378 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/ffc2a268-b4b5-40f3-8074-d7dd2d54eb05-console-config\") pod \"console-7759588d67-q7dwh\" (UID: \"ffc2a268-b4b5-40f3-8074-d7dd2d54eb05\") " pod="openshift-console/console-7759588d67-q7dwh" Oct 30 20:38:38.041039 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:38:38.040957 5378 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/ffc2a268-b4b5-40f3-8074-d7dd2d54eb05-tmp\") pod \"console-7759588d67-q7dwh\" (UID: \"ffc2a268-b4b5-40f3-8074-d7dd2d54eb05\") " pod="openshift-console/console-7759588d67-q7dwh" Oct 30 20:38:38.041039 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:38:38.040971 5378 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/ffc2a268-b4b5-40f3-8074-d7dd2d54eb05-service-ca\") pod \"console-7759588d67-q7dwh\" (UID: \"ffc2a268-b4b5-40f3-8074-d7dd2d54eb05\") " pod="openshift-console/console-7759588d67-q7dwh" Oct 30 20:38:38.041039 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:38:38.040986 5378 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ffc2a268-b4b5-40f3-8074-d7dd2d54eb05-trusted-ca-bundle\") pod \"console-7759588d67-q7dwh\" (UID: \"ffc2a268-b4b5-40f3-8074-d7dd2d54eb05\") " pod="openshift-console/console-7759588d67-q7dwh" Oct 30 20:38:38.141406 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:38:38.141374 5378 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/ffc2a268-b4b5-40f3-8074-d7dd2d54eb05-console-serving-cert\") pod \"console-7759588d67-q7dwh\" (UID: \"ffc2a268-b4b5-40f3-8074-d7dd2d54eb05\") " pod="openshift-console/console-7759588d67-q7dwh" Oct 30 20:38:38.141406 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:38:38.141416 5378 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/ffc2a268-b4b5-40f3-8074-d7dd2d54eb05-console-config\") pod \"console-7759588d67-q7dwh\" (UID: \"ffc2a268-b4b5-40f3-8074-d7dd2d54eb05\") " pod="openshift-console/console-7759588d67-q7dwh" Oct 30 20:38:38.141645 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:38:38.141442 5378 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/ffc2a268-b4b5-40f3-8074-d7dd2d54eb05-tmp\") pod \"console-7759588d67-q7dwh\" (UID: \"ffc2a268-b4b5-40f3-8074-d7dd2d54eb05\") " pod="openshift-console/console-7759588d67-q7dwh" Oct 30 20:38:38.141645 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:38:38.141467 5378 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/ffc2a268-b4b5-40f3-8074-d7dd2d54eb05-service-ca\") pod \"console-7759588d67-q7dwh\" (UID: \"ffc2a268-b4b5-40f3-8074-d7dd2d54eb05\") " pod="openshift-console/console-7759588d67-q7dwh" Oct 30 20:38:38.141645 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:38:38.141491 5378 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ffc2a268-b4b5-40f3-8074-d7dd2d54eb05-trusted-ca-bundle\") pod \"console-7759588d67-q7dwh\" (UID: \"ffc2a268-b4b5-40f3-8074-d7dd2d54eb05\") " pod="openshift-console/console-7759588d67-q7dwh" Oct 30 20:38:38.141645 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:38:38.141574 5378 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sfn6b\" (UniqueName: \"kubernetes.io/projected/ffc2a268-b4b5-40f3-8074-d7dd2d54eb05-kube-api-access-sfn6b\") pod \"console-7759588d67-q7dwh\" (UID: \"ffc2a268-b4b5-40f3-8074-d7dd2d54eb05\") " pod="openshift-console/console-7759588d67-q7dwh" Oct 30 20:38:38.141645 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:38:38.141619 5378 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/ffc2a268-b4b5-40f3-8074-d7dd2d54eb05-console-oauth-config\") pod \"console-7759588d67-q7dwh\" (UID: \"ffc2a268-b4b5-40f3-8074-d7dd2d54eb05\") " pod="openshift-console/console-7759588d67-q7dwh" Oct 30 20:38:38.141645 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:38:38.141643 5378 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/ffc2a268-b4b5-40f3-8074-d7dd2d54eb05-oauth-serving-cert\") pod \"console-7759588d67-q7dwh\" (UID: \"ffc2a268-b4b5-40f3-8074-d7dd2d54eb05\") " pod="openshift-console/console-7759588d67-q7dwh" Oct 30 20:38:38.141943 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:38:38.141821 5378 swap_util.go:78] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Oct 30 20:38:38.142109 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:38:38.142081 5378 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/ffc2a268-b4b5-40f3-8074-d7dd2d54eb05-tmp\") pod \"console-7759588d67-q7dwh\" (UID: \"ffc2a268-b4b5-40f3-8074-d7dd2d54eb05\") " pod="openshift-console/console-7759588d67-q7dwh" Oct 30 20:38:38.142246 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:38:38.142175 5378 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/ffc2a268-b4b5-40f3-8074-d7dd2d54eb05-console-config\") pod \"console-7759588d67-q7dwh\" (UID: \"ffc2a268-b4b5-40f3-8074-d7dd2d54eb05\") " pod="openshift-console/console-7759588d67-q7dwh" Oct 30 20:38:38.142318 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:38:38.142294 5378 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/ffc2a268-b4b5-40f3-8074-d7dd2d54eb05-oauth-serving-cert\") pod \"console-7759588d67-q7dwh\" (UID: \"ffc2a268-b4b5-40f3-8074-d7dd2d54eb05\") " pod="openshift-console/console-7759588d67-q7dwh" Oct 30 20:38:38.142694 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:38:38.142671 5378 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/ffc2a268-b4b5-40f3-8074-d7dd2d54eb05-service-ca\") pod \"console-7759588d67-q7dwh\" (UID: \"ffc2a268-b4b5-40f3-8074-d7dd2d54eb05\") " pod="openshift-console/console-7759588d67-q7dwh" Oct 30 20:38:38.144508 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:38:38.144484 5378 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/ffc2a268-b4b5-40f3-8074-d7dd2d54eb05-console-serving-cert\") pod \"console-7759588d67-q7dwh\" (UID: \"ffc2a268-b4b5-40f3-8074-d7dd2d54eb05\") " pod="openshift-console/console-7759588d67-q7dwh" Oct 30 20:38:38.144698 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:38:38.144678 5378 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/ffc2a268-b4b5-40f3-8074-d7dd2d54eb05-console-oauth-config\") pod \"console-7759588d67-q7dwh\" (UID: \"ffc2a268-b4b5-40f3-8074-d7dd2d54eb05\") " pod="openshift-console/console-7759588d67-q7dwh" Oct 30 20:38:38.218001 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:38:38.217963 5378 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\"" Oct 30 20:38:38.218195 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:38:38.218176 5378 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-7759588d67-q7dwh"] Oct 30 20:38:38.219559 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:38:38.219536 5378 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sfn6b\" (UniqueName: \"kubernetes.io/projected/ffc2a268-b4b5-40f3-8074-d7dd2d54eb05-kube-api-access-sfn6b\") pod \"console-7759588d67-q7dwh\" (UID: \"ffc2a268-b4b5-40f3-8074-d7dd2d54eb05\") " pod="openshift-console/console-7759588d67-q7dwh" Oct 30 20:38:38.223678 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:38:38.223647 5378 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ffc2a268-b4b5-40f3-8074-d7dd2d54eb05-trusted-ca-bundle\") pod \"console-7759588d67-q7dwh\" (UID: \"ffc2a268-b4b5-40f3-8074-d7dd2d54eb05\") " pod="openshift-console/console-7759588d67-q7dwh" Oct 30 20:38:38.323513 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:38:38.323435 5378 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7759588d67-q7dwh" Oct 30 20:38:38.525621 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:38:38.525520 5378 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-6bfdc6f474-hj4hq" Oct 30 20:38:38.532004 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:38:38.531979 5378 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-6bfdc6f474-hj4hq" Oct 30 20:38:38.697436 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:38:38.697407 5378 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-7759588d67-q7dwh"] Oct 30 20:38:38.729861 ip-10-0-138-149 kubenswrapper[5378]: W1030 20:38:38.729829 5378 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podffc2a268_b4b5_40f3_8074_d7dd2d54eb05.slice/crio-99e1181265ff11a762b4d976bc3e8d0f36abda834c72aee8e346f2736c550721 WatchSource:0}: Error finding container 99e1181265ff11a762b4d976bc3e8d0f36abda834c72aee8e346f2736c550721: Status 404 returned error can't find the container with id 99e1181265ff11a762b4d976bc3e8d0f36abda834c72aee8e346f2736c550721 Oct 30 20:38:38.845226 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:38:38.845068 5378 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7759588d67-q7dwh" event={"ID":"ffc2a268-b4b5-40f3-8074-d7dd2d54eb05","Type":"ContainerStarted","Data":"cb67a07b2e9f6c66c3e308c96c10a9c7d43ecec89bd72e23592270f6445aaf2f"} Oct 30 20:38:38.845226 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:38:38.845130 5378 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7759588d67-q7dwh" event={"ID":"ffc2a268-b4b5-40f3-8074-d7dd2d54eb05","Type":"ContainerStarted","Data":"99e1181265ff11a762b4d976bc3e8d0f36abda834c72aee8e346f2736c550721"} Oct 30 20:38:38.849410 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:38:38.849360 5378 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-6bfdc6f474-hj4hq" Oct 30 20:38:38.962557 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:38:38.962356 5378 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-console/console-7759588d67-q7dwh" podStartSLOduration=1.962343088 podStartE2EDuration="1.962343088s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:38:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:38:38.961260425 +0000 UTC m=+9.389374345" watchObservedRunningTime="2026-10-30 20:38:38.962343088 +0000 UTC m=+9.390457004" Oct 30 20:38:40.435073 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:38:40.435031 5378 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-7759588d67-q7dwh" Oct 30 20:38:40.440399 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:38:40.440376 5378 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-7759588d67-q7dwh" Oct 30 20:38:41.858030 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:38:41.857977 5378 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Oct 30 20:38:41.885931 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:38:41.885906 5378 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-7x9jg" Oct 30 20:38:43.980870 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:38:43.980834 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 20:38:44.119066 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:38:44.119030 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 20:38:45.027079 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:38:45.027053 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 20:38:45.155599 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:38:45.155551 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 20:38:48.009657 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:38:48.009624 5378 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-7759588d67-q7dwh" Oct 30 20:38:48.010137 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:38:48.009757 5378 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Oct 30 20:38:48.013738 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:38:48.013710 5378 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-7759588d67-q7dwh" Oct 30 20:38:48.183962 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:38:48.183931 5378 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-6bfdc6f474-hj4hq"] Oct 30 20:38:48.268101 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:38:48.267961 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 20:38:48.409881 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:38:48.409838 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 20:38:48.840619 ip-10-0-138-149 kubenswrapper[5378]: E1030 20:38:48.840585 5378 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q6mx7" containerName="extractor" Oct 30 20:38:48.883505 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:38:48.883464 5378 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-6dd789c8b7-5pslw" event={"ID":"59046ab6-2f4c-4031-ba8c-21fb322a9d07","Type":"ContainerStarted","Data":"9853e1b0549f2fe3480c45d7d47bb979b2c926291ff06d8cd56f243068300d5d"} Oct 30 20:38:51.956126 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:38:51.956094 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 20:38:52.080006 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:38:52.079966 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 20:38:54.492082 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:38:54.492041 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 20:38:58.884797 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:38:58.884758 5378 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/downloads-6dd789c8b7-5pslw" Oct 30 20:38:58.886042 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:38:58.886022 5378 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/downloads-6dd789c8b7-5pslw" Oct 30 20:38:58.982992 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:38:58.982964 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 20:38:59.118376 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:38:59.118350 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 20:39:00.807531 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:39:00.807499 5378 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-cjrkj" Oct 30 20:39:02.424788 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:39:02.424757 5378 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-ws655" Oct 30 20:39:13.084374 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:39:13.084334 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 20:39:13.205111 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:39:13.205055 5378 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-console/console-6bfdc6f474-hj4hq" podUID="6646b935-fe10-415a-be4b-a62a64ea45ec" containerName="console" containerID="cri-o://c71886b8af6a606b11086a38e78a74cdf2f65ee75709915297f0edbc2c4c2429" gracePeriod=15 Oct 30 20:39:13.206973 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:39:13.206941 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 20:39:13.478948 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:39:13.478924 5378 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6bfdc6f474-hj4hq" Oct 30 20:39:13.625047 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:39:13.625023 5378 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6646b935-fe10-415a-be4b-a62a64ea45ec-console-config\" (UniqueName: \"kubernetes.io/configmap/6646b935-fe10-415a-be4b-a62a64ea45ec-console-config\") pod \"6646b935-fe10-415a-be4b-a62a64ea45ec\" (UID: \"6646b935-fe10-415a-be4b-a62a64ea45ec\") " Oct 30 20:39:13.625197 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:39:13.625062 5378 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6646b935-fe10-415a-be4b-a62a64ea45ec-console-oauth-config\" (UniqueName: \"kubernetes.io/secret/6646b935-fe10-415a-be4b-a62a64ea45ec-console-oauth-config\") pod \"6646b935-fe10-415a-be4b-a62a64ea45ec\" (UID: \"6646b935-fe10-415a-be4b-a62a64ea45ec\") " Oct 30 20:39:13.625197 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:39:13.625093 5378 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6646b935-fe10-415a-be4b-a62a64ea45ec-oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/6646b935-fe10-415a-be4b-a62a64ea45ec-oauth-serving-cert\") pod \"6646b935-fe10-415a-be4b-a62a64ea45ec\" (UID: \"6646b935-fe10-415a-be4b-a62a64ea45ec\") " Oct 30 20:39:13.625197 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:39:13.625175 5378 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6646b935-fe10-415a-be4b-a62a64ea45ec-service-ca\" (UniqueName: \"kubernetes.io/configmap/6646b935-fe10-415a-be4b-a62a64ea45ec-service-ca\") pod \"6646b935-fe10-415a-be4b-a62a64ea45ec\" (UID: \"6646b935-fe10-415a-be4b-a62a64ea45ec\") " Oct 30 20:39:13.625317 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:39:13.625201 5378 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6646b935-fe10-415a-be4b-a62a64ea45ec-console-serving-cert\" (UniqueName: \"kubernetes.io/secret/6646b935-fe10-415a-be4b-a62a64ea45ec-console-serving-cert\") pod \"6646b935-fe10-415a-be4b-a62a64ea45ec\" (UID: \"6646b935-fe10-415a-be4b-a62a64ea45ec\") " Oct 30 20:39:13.625317 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:39:13.625239 5378 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6646b935-fe10-415a-be4b-a62a64ea45ec-tmp\" (UniqueName: \"kubernetes.io/empty-dir/6646b935-fe10-415a-be4b-a62a64ea45ec-tmp\") pod \"6646b935-fe10-415a-be4b-a62a64ea45ec\" (UID: \"6646b935-fe10-415a-be4b-a62a64ea45ec\") " Oct 30 20:39:13.625317 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:39:13.625270 5378 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6646b935-fe10-415a-be4b-a62a64ea45ec-kube-api-access-fqlw7\" (UniqueName: \"kubernetes.io/projected/6646b935-fe10-415a-be4b-a62a64ea45ec-kube-api-access-fqlw7\") pod \"6646b935-fe10-415a-be4b-a62a64ea45ec\" (UID: \"6646b935-fe10-415a-be4b-a62a64ea45ec\") " Oct 30 20:39:13.625637 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:39:13.625604 5378 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6646b935-fe10-415a-be4b-a62a64ea45ec-console-config" pod "6646b935-fe10-415a-be4b-a62a64ea45ec" (UID: "6646b935-fe10-415a-be4b-a62a64ea45ec"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:39:13.625637 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:39:13.625605 5378 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6646b935-fe10-415a-be4b-a62a64ea45ec-tmp" pod "6646b935-fe10-415a-be4b-a62a64ea45ec" (UID: "6646b935-fe10-415a-be4b-a62a64ea45ec"). InnerVolumeSpecName "tmp". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:39:13.625794 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:39:13.625646 5378 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6646b935-fe10-415a-be4b-a62a64ea45ec-oauth-serving-cert" pod "6646b935-fe10-415a-be4b-a62a64ea45ec" (UID: "6646b935-fe10-415a-be4b-a62a64ea45ec"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:39:13.625794 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:39:13.625654 5378 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6646b935-fe10-415a-be4b-a62a64ea45ec-service-ca" pod "6646b935-fe10-415a-be4b-a62a64ea45ec" (UID: "6646b935-fe10-415a-be4b-a62a64ea45ec"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:39:13.627577 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:39:13.627545 5378 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6646b935-fe10-415a-be4b-a62a64ea45ec-console-oauth-config" pod "6646b935-fe10-415a-be4b-a62a64ea45ec" (UID: "6646b935-fe10-415a-be4b-a62a64ea45ec"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 20:39:13.627668 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:39:13.627607 5378 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6646b935-fe10-415a-be4b-a62a64ea45ec-kube-api-access-fqlw7" pod "6646b935-fe10-415a-be4b-a62a64ea45ec" (UID: "6646b935-fe10-415a-be4b-a62a64ea45ec"). InnerVolumeSpecName "kube-api-access-fqlw7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:39:13.627668 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:39:13.627613 5378 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6646b935-fe10-415a-be4b-a62a64ea45ec-console-serving-cert" pod "6646b935-fe10-415a-be4b-a62a64ea45ec" (UID: "6646b935-fe10-415a-be4b-a62a64ea45ec"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 20:39:13.726300 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:39:13.726268 5378 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/6646b935-fe10-415a-be4b-a62a64ea45ec-console-config\") on node \"ip-10-0-138-149.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:39:13.726300 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:39:13.726296 5378 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/6646b935-fe10-415a-be4b-a62a64ea45ec-console-oauth-config\") on node \"ip-10-0-138-149.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:39:13.726300 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:39:13.726306 5378 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/6646b935-fe10-415a-be4b-a62a64ea45ec-oauth-serving-cert\") on node \"ip-10-0-138-149.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:39:13.726555 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:39:13.726316 5378 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/6646b935-fe10-415a-be4b-a62a64ea45ec-service-ca\") on node \"ip-10-0-138-149.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:39:13.726555 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:39:13.726324 5378 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/6646b935-fe10-415a-be4b-a62a64ea45ec-console-serving-cert\") on node \"ip-10-0-138-149.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:39:13.726555 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:39:13.726334 5378 reconciler_common.go:299] "Volume detached for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/6646b935-fe10-415a-be4b-a62a64ea45ec-tmp\") on node \"ip-10-0-138-149.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:39:13.726555 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:39:13.726342 5378 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fqlw7\" (UniqueName: \"kubernetes.io/projected/6646b935-fe10-415a-be4b-a62a64ea45ec-kube-api-access-fqlw7\") on node \"ip-10-0-138-149.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:39:13.970369 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:39:13.970285 5378 generic.go:356] "Generic (PLEG): container finished" podID="6646b935-fe10-415a-be4b-a62a64ea45ec" containerID="c71886b8af6a606b11086a38e78a74cdf2f65ee75709915297f0edbc2c4c2429" exitCode=0 Oct 30 20:39:13.970369 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:39:13.970339 5378 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6bfdc6f474-hj4hq" Oct 30 20:39:13.970369 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:39:13.970358 5378 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6bfdc6f474-hj4hq" event={"ID":"6646b935-fe10-415a-be4b-a62a64ea45ec","Type":"ContainerDied","Data":"c71886b8af6a606b11086a38e78a74cdf2f65ee75709915297f0edbc2c4c2429"} Oct 30 20:39:13.970643 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:39:13.970404 5378 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6bfdc6f474-hj4hq" event={"ID":"6646b935-fe10-415a-be4b-a62a64ea45ec","Type":"ContainerDied","Data":"c1c69985245a3e90dd11045025945aa9e230f6a0cddc66aa01c9131cf8ce0d3b"} Oct 30 20:39:13.970643 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:39:13.970430 5378 scope.go:122] "RemoveContainer" containerID="c71886b8af6a606b11086a38e78a74cdf2f65ee75709915297f0edbc2c4c2429" Oct 30 20:39:13.978248 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:39:13.978231 5378 scope.go:122] "RemoveContainer" containerID="c71886b8af6a606b11086a38e78a74cdf2f65ee75709915297f0edbc2c4c2429" Oct 30 20:39:13.978505 ip-10-0-138-149 kubenswrapper[5378]: E1030 20:39:13.978482 5378 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c71886b8af6a606b11086a38e78a74cdf2f65ee75709915297f0edbc2c4c2429\": container with ID starting with c71886b8af6a606b11086a38e78a74cdf2f65ee75709915297f0edbc2c4c2429 not found: ID does not exist" containerID="c71886b8af6a606b11086a38e78a74cdf2f65ee75709915297f0edbc2c4c2429" Oct 30 20:39:13.978560 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:39:13.978512 5378 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c71886b8af6a606b11086a38e78a74cdf2f65ee75709915297f0edbc2c4c2429"} err="failed to get container status \"c71886b8af6a606b11086a38e78a74cdf2f65ee75709915297f0edbc2c4c2429\": rpc error: code = NotFound desc = could not find container \"c71886b8af6a606b11086a38e78a74cdf2f65ee75709915297f0edbc2c4c2429\": container with ID starting with c71886b8af6a606b11086a38e78a74cdf2f65ee75709915297f0edbc2c4c2429 not found: ID does not exist" Oct 30 20:39:14.133765 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:39:14.133740 5378 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-6bfdc6f474-hj4hq"] Oct 30 20:39:14.184352 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:39:14.184328 5378 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-6bfdc6f474-hj4hq"] Oct 30 20:39:15.737972 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:39:15.737941 5378 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6646b935-fe10-415a-be4b-a62a64ea45ec" path="/var/lib/kubelet/pods/6646b935-fe10-415a-be4b-a62a64ea45ec/volumes" Oct 30 20:39:18.836605 ip-10-0-138-149 kubenswrapper[5378]: E1030 20:39:18.835849 5378 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q6mx7" containerName="extractor" Oct 30 20:39:36.043551 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:39:36.043512 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 20:39:48.835456 ip-10-0-138-149 kubenswrapper[5378]: E1030 20:39:48.835420 5378 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q6mx7" containerName="extractor" Oct 30 20:39:48.857534 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:39:48.857502 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 20:39:49.214988 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:39:49.214950 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 20:39:49.889846 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:39:49.889815 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 20:39:50.017826 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:39:50.017797 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 20:39:59.056635 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:39:59.056598 5378 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console/console-7b796cd949-sd2d4"] Oct 30 20:39:59.057122 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:39:59.056976 5378 cpu_manager.go:414] "containerMap: removing container" podUID="6646b935-fe10-415a-be4b-a62a64ea45ec" containerName="console" Oct 30 20:39:59.057122 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:39:59.056994 5378 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6646b935-fe10-415a-be4b-a62a64ea45ec" containerName="console" Oct 30 20:39:59.057122 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:39:59.057068 5378 memory_manager.go:365] "RemoveStaleState removing state" podUID="6646b935-fe10-415a-be4b-a62a64ea45ec" containerName="console" Oct 30 20:39:59.058805 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:39:59.058782 5378 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7b796cd949-sd2d4" Oct 30 20:39:59.146983 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:39:59.146953 5378 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/14e39214-df20-411a-a433-f9dcd9d722c6-console-serving-cert\") pod \"console-7b796cd949-sd2d4\" (UID: \"14e39214-df20-411a-a433-f9dcd9d722c6\") " pod="openshift-console/console-7b796cd949-sd2d4" Oct 30 20:39:59.147147 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:39:59.147052 5378 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/14e39214-df20-411a-a433-f9dcd9d722c6-console-config\") pod \"console-7b796cd949-sd2d4\" (UID: \"14e39214-df20-411a-a433-f9dcd9d722c6\") " pod="openshift-console/console-7b796cd949-sd2d4" Oct 30 20:39:59.147147 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:39:59.147081 5378 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/14e39214-df20-411a-a433-f9dcd9d722c6-console-oauth-config\") pod \"console-7b796cd949-sd2d4\" (UID: \"14e39214-df20-411a-a433-f9dcd9d722c6\") " pod="openshift-console/console-7b796cd949-sd2d4" Oct 30 20:39:59.147147 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:39:59.147101 5378 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/14e39214-df20-411a-a433-f9dcd9d722c6-service-ca\") pod \"console-7b796cd949-sd2d4\" (UID: \"14e39214-df20-411a-a433-f9dcd9d722c6\") " pod="openshift-console/console-7b796cd949-sd2d4" Oct 30 20:39:59.147327 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:39:59.147210 5378 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/14e39214-df20-411a-a433-f9dcd9d722c6-oauth-serving-cert\") pod \"console-7b796cd949-sd2d4\" (UID: \"14e39214-df20-411a-a433-f9dcd9d722c6\") " pod="openshift-console/console-7b796cd949-sd2d4" Oct 30 20:39:59.147327 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:39:59.147238 5378 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/14e39214-df20-411a-a433-f9dcd9d722c6-trusted-ca-bundle\") pod \"console-7b796cd949-sd2d4\" (UID: \"14e39214-df20-411a-a433-f9dcd9d722c6\") " pod="openshift-console/console-7b796cd949-sd2d4" Oct 30 20:39:59.147327 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:39:59.147256 5378 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/14e39214-df20-411a-a433-f9dcd9d722c6-tmp\") pod \"console-7b796cd949-sd2d4\" (UID: \"14e39214-df20-411a-a433-f9dcd9d722c6\") " pod="openshift-console/console-7b796cd949-sd2d4" Oct 30 20:39:59.147327 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:39:59.147272 5378 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-plb8b\" (UniqueName: \"kubernetes.io/projected/14e39214-df20-411a-a433-f9dcd9d722c6-kube-api-access-plb8b\") pod \"console-7b796cd949-sd2d4\" (UID: \"14e39214-df20-411a-a433-f9dcd9d722c6\") " pod="openshift-console/console-7b796cd949-sd2d4" Oct 30 20:39:59.172859 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:39:59.172824 5378 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-7b796cd949-sd2d4"] Oct 30 20:39:59.248083 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:39:59.248049 5378 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/14e39214-df20-411a-a433-f9dcd9d722c6-console-config\") pod \"console-7b796cd949-sd2d4\" (UID: \"14e39214-df20-411a-a433-f9dcd9d722c6\") " pod="openshift-console/console-7b796cd949-sd2d4" Oct 30 20:39:59.248267 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:39:59.248097 5378 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/14e39214-df20-411a-a433-f9dcd9d722c6-console-oauth-config\") pod \"console-7b796cd949-sd2d4\" (UID: \"14e39214-df20-411a-a433-f9dcd9d722c6\") " pod="openshift-console/console-7b796cd949-sd2d4" Oct 30 20:39:59.248267 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:39:59.248117 5378 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/14e39214-df20-411a-a433-f9dcd9d722c6-service-ca\") pod \"console-7b796cd949-sd2d4\" (UID: \"14e39214-df20-411a-a433-f9dcd9d722c6\") " pod="openshift-console/console-7b796cd949-sd2d4" Oct 30 20:39:59.248267 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:39:59.248135 5378 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/14e39214-df20-411a-a433-f9dcd9d722c6-oauth-serving-cert\") pod \"console-7b796cd949-sd2d4\" (UID: \"14e39214-df20-411a-a433-f9dcd9d722c6\") " pod="openshift-console/console-7b796cd949-sd2d4" Oct 30 20:39:59.248267 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:39:59.248153 5378 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/14e39214-df20-411a-a433-f9dcd9d722c6-trusted-ca-bundle\") pod \"console-7b796cd949-sd2d4\" (UID: \"14e39214-df20-411a-a433-f9dcd9d722c6\") " pod="openshift-console/console-7b796cd949-sd2d4" Oct 30 20:39:59.248267 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:39:59.248202 5378 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/14e39214-df20-411a-a433-f9dcd9d722c6-tmp\") pod \"console-7b796cd949-sd2d4\" (UID: \"14e39214-df20-411a-a433-f9dcd9d722c6\") " pod="openshift-console/console-7b796cd949-sd2d4" Oct 30 20:39:59.248267 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:39:59.248235 5378 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-plb8b\" (UniqueName: \"kubernetes.io/projected/14e39214-df20-411a-a433-f9dcd9d722c6-kube-api-access-plb8b\") pod \"console-7b796cd949-sd2d4\" (UID: \"14e39214-df20-411a-a433-f9dcd9d722c6\") " pod="openshift-console/console-7b796cd949-sd2d4" Oct 30 20:39:59.248598 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:39:59.248271 5378 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/14e39214-df20-411a-a433-f9dcd9d722c6-console-serving-cert\") pod \"console-7b796cd949-sd2d4\" (UID: \"14e39214-df20-411a-a433-f9dcd9d722c6\") " pod="openshift-console/console-7b796cd949-sd2d4" Oct 30 20:39:59.248804 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:39:59.248752 5378 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/14e39214-df20-411a-a433-f9dcd9d722c6-tmp\") pod \"console-7b796cd949-sd2d4\" (UID: \"14e39214-df20-411a-a433-f9dcd9d722c6\") " pod="openshift-console/console-7b796cd949-sd2d4" Oct 30 20:39:59.249015 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:39:59.248975 5378 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/14e39214-df20-411a-a433-f9dcd9d722c6-console-config\") pod \"console-7b796cd949-sd2d4\" (UID: \"14e39214-df20-411a-a433-f9dcd9d722c6\") " pod="openshift-console/console-7b796cd949-sd2d4" Oct 30 20:39:59.249190 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:39:59.249166 5378 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/14e39214-df20-411a-a433-f9dcd9d722c6-service-ca\") pod \"console-7b796cd949-sd2d4\" (UID: \"14e39214-df20-411a-a433-f9dcd9d722c6\") " pod="openshift-console/console-7b796cd949-sd2d4" Oct 30 20:39:59.249285 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:39:59.249176 5378 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/14e39214-df20-411a-a433-f9dcd9d722c6-oauth-serving-cert\") pod \"console-7b796cd949-sd2d4\" (UID: \"14e39214-df20-411a-a433-f9dcd9d722c6\") " pod="openshift-console/console-7b796cd949-sd2d4" Oct 30 20:39:59.249348 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:39:59.249333 5378 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/14e39214-df20-411a-a433-f9dcd9d722c6-trusted-ca-bundle\") pod \"console-7b796cd949-sd2d4\" (UID: \"14e39214-df20-411a-a433-f9dcd9d722c6\") " pod="openshift-console/console-7b796cd949-sd2d4" Oct 30 20:39:59.250820 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:39:59.250801 5378 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/14e39214-df20-411a-a433-f9dcd9d722c6-console-serving-cert\") pod \"console-7b796cd949-sd2d4\" (UID: \"14e39214-df20-411a-a433-f9dcd9d722c6\") " pod="openshift-console/console-7b796cd949-sd2d4" Oct 30 20:39:59.250895 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:39:59.250848 5378 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/14e39214-df20-411a-a433-f9dcd9d722c6-console-oauth-config\") pod \"console-7b796cd949-sd2d4\" (UID: \"14e39214-df20-411a-a433-f9dcd9d722c6\") " pod="openshift-console/console-7b796cd949-sd2d4" Oct 30 20:39:59.306052 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:39:59.306026 5378 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-plb8b\" (UniqueName: \"kubernetes.io/projected/14e39214-df20-411a-a433-f9dcd9d722c6-kube-api-access-plb8b\") pod \"console-7b796cd949-sd2d4\" (UID: \"14e39214-df20-411a-a433-f9dcd9d722c6\") " pod="openshift-console/console-7b796cd949-sd2d4" Oct 30 20:39:59.312635 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:39:59.312585 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 20:39:59.368578 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:39:59.368536 5378 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7b796cd949-sd2d4" Oct 30 20:39:59.456968 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:39:59.456929 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 20:39:59.715125 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:39:59.715080 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 20:39:59.733734 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:39:59.733708 5378 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-7b796cd949-sd2d4"] Oct 30 20:39:59.765115 ip-10-0-138-149 kubenswrapper[5378]: W1030 20:39:59.765086 5378 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod14e39214_df20_411a_a433_f9dcd9d722c6.slice/crio-4990891c30b04aaf0520192661169c7a2b89211863c3026c7121608cc5558ccd WatchSource:0}: Error finding container 4990891c30b04aaf0520192661169c7a2b89211863c3026c7121608cc5558ccd: Status 404 returned error can't find the container with id 4990891c30b04aaf0520192661169c7a2b89211863c3026c7121608cc5558ccd Oct 30 20:39:59.948452 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:39:59.948418 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 20:40:00.121506 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:40:00.121469 5378 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7b796cd949-sd2d4" event={"ID":"14e39214-df20-411a-a433-f9dcd9d722c6","Type":"ContainerStarted","Data":"6d02b2264e0316008d1957de8bc21e8e04318fc9e0d9ec80b405faf02e87582d"} Oct 30 20:40:00.121878 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:40:00.121560 5378 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7b796cd949-sd2d4" event={"ID":"14e39214-df20-411a-a433-f9dcd9d722c6","Type":"ContainerStarted","Data":"4990891c30b04aaf0520192661169c7a2b89211863c3026c7121608cc5558ccd"} Oct 30 20:40:00.241930 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:40:00.241877 5378 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-console/console-7b796cd949-sd2d4" podStartSLOduration=2.241864062 podStartE2EDuration="2.241864062s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:39:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:40:00.239542688 +0000 UTC m=+90.667656619" watchObservedRunningTime="2026-10-30 20:40:00.241864062 +0000 UTC m=+90.669977977" Oct 30 20:40:09.369238 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:40:09.369154 5378 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-7b796cd949-sd2d4" Oct 30 20:40:09.369238 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:40:09.369218 5378 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-7b796cd949-sd2d4" Oct 30 20:40:09.373827 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:40:09.373803 5378 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-7b796cd949-sd2d4" Oct 30 20:40:10.158042 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:40:10.158009 5378 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-7b796cd949-sd2d4" Oct 30 20:40:10.326814 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:40:10.326784 5378 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-7759588d67-q7dwh"] Oct 30 20:40:17.842616 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:40:17.842573 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 20:40:18.835660 ip-10-0-138-149 kubenswrapper[5378]: E1030 20:40:18.835621 5378 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q6mx7" containerName="extractor" Oct 30 20:40:21.658931 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:40:21.658899 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 20:40:35.345492 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:40:35.345426 5378 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-console/console-7759588d67-q7dwh" podUID="ffc2a268-b4b5-40f3-8074-d7dd2d54eb05" containerName="console" containerID="cri-o://cb67a07b2e9f6c66c3e308c96c10a9c7d43ecec89bd72e23592270f6445aaf2f" gracePeriod=15 Oct 30 20:40:35.583073 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:40:35.583050 5378 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7759588d67-q7dwh" Oct 30 20:40:35.713750 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:40:35.713676 5378 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ffc2a268-b4b5-40f3-8074-d7dd2d54eb05-oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/ffc2a268-b4b5-40f3-8074-d7dd2d54eb05-oauth-serving-cert\") pod \"ffc2a268-b4b5-40f3-8074-d7dd2d54eb05\" (UID: \"ffc2a268-b4b5-40f3-8074-d7dd2d54eb05\") " Oct 30 20:40:35.713750 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:40:35.713730 5378 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ffc2a268-b4b5-40f3-8074-d7dd2d54eb05-service-ca\" (UniqueName: \"kubernetes.io/configmap/ffc2a268-b4b5-40f3-8074-d7dd2d54eb05-service-ca\") pod \"ffc2a268-b4b5-40f3-8074-d7dd2d54eb05\" (UID: \"ffc2a268-b4b5-40f3-8074-d7dd2d54eb05\") " Oct 30 20:40:35.713940 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:40:35.713763 5378 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ffc2a268-b4b5-40f3-8074-d7dd2d54eb05-kube-api-access-sfn6b\" (UniqueName: \"kubernetes.io/projected/ffc2a268-b4b5-40f3-8074-d7dd2d54eb05-kube-api-access-sfn6b\") pod \"ffc2a268-b4b5-40f3-8074-d7dd2d54eb05\" (UID: \"ffc2a268-b4b5-40f3-8074-d7dd2d54eb05\") " Oct 30 20:40:35.713940 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:40:35.713785 5378 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ffc2a268-b4b5-40f3-8074-d7dd2d54eb05-console-serving-cert\" (UniqueName: \"kubernetes.io/secret/ffc2a268-b4b5-40f3-8074-d7dd2d54eb05-console-serving-cert\") pod \"ffc2a268-b4b5-40f3-8074-d7dd2d54eb05\" (UID: \"ffc2a268-b4b5-40f3-8074-d7dd2d54eb05\") " Oct 30 20:40:35.713940 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:40:35.713807 5378 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ffc2a268-b4b5-40f3-8074-d7dd2d54eb05-console-config\" (UniqueName: \"kubernetes.io/configmap/ffc2a268-b4b5-40f3-8074-d7dd2d54eb05-console-config\") pod \"ffc2a268-b4b5-40f3-8074-d7dd2d54eb05\" (UID: \"ffc2a268-b4b5-40f3-8074-d7dd2d54eb05\") " Oct 30 20:40:35.713940 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:40:35.713835 5378 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ffc2a268-b4b5-40f3-8074-d7dd2d54eb05-tmp\" (UniqueName: \"kubernetes.io/empty-dir/ffc2a268-b4b5-40f3-8074-d7dd2d54eb05-tmp\") pod \"ffc2a268-b4b5-40f3-8074-d7dd2d54eb05\" (UID: \"ffc2a268-b4b5-40f3-8074-d7dd2d54eb05\") " Oct 30 20:40:35.713940 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:40:35.713894 5378 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ffc2a268-b4b5-40f3-8074-d7dd2d54eb05-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ffc2a268-b4b5-40f3-8074-d7dd2d54eb05-trusted-ca-bundle\") pod \"ffc2a268-b4b5-40f3-8074-d7dd2d54eb05\" (UID: \"ffc2a268-b4b5-40f3-8074-d7dd2d54eb05\") " Oct 30 20:40:35.713940 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:40:35.713933 5378 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ffc2a268-b4b5-40f3-8074-d7dd2d54eb05-console-oauth-config\" (UniqueName: \"kubernetes.io/secret/ffc2a268-b4b5-40f3-8074-d7dd2d54eb05-console-oauth-config\") pod \"ffc2a268-b4b5-40f3-8074-d7dd2d54eb05\" (UID: \"ffc2a268-b4b5-40f3-8074-d7dd2d54eb05\") " Oct 30 20:40:35.714229 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:40:35.714070 5378 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ffc2a268-b4b5-40f3-8074-d7dd2d54eb05-oauth-serving-cert" pod "ffc2a268-b4b5-40f3-8074-d7dd2d54eb05" (UID: "ffc2a268-b4b5-40f3-8074-d7dd2d54eb05"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:40:35.714229 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:40:35.714186 5378 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ffc2a268-b4b5-40f3-8074-d7dd2d54eb05-tmp" pod "ffc2a268-b4b5-40f3-8074-d7dd2d54eb05" (UID: "ffc2a268-b4b5-40f3-8074-d7dd2d54eb05"). InnerVolumeSpecName "tmp". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:40:35.714342 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:40:35.714277 5378 reconciler_common.go:299] "Volume detached for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/ffc2a268-b4b5-40f3-8074-d7dd2d54eb05-tmp\") on node \"ip-10-0-138-149.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:40:35.714342 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:40:35.714297 5378 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/ffc2a268-b4b5-40f3-8074-d7dd2d54eb05-oauth-serving-cert\") on node \"ip-10-0-138-149.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:40:35.714442 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:40:35.714373 5378 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ffc2a268-b4b5-40f3-8074-d7dd2d54eb05-service-ca" pod "ffc2a268-b4b5-40f3-8074-d7dd2d54eb05" (UID: "ffc2a268-b4b5-40f3-8074-d7dd2d54eb05"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:40:35.714442 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:40:35.714421 5378 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ffc2a268-b4b5-40f3-8074-d7dd2d54eb05-console-config" pod "ffc2a268-b4b5-40f3-8074-d7dd2d54eb05" (UID: "ffc2a268-b4b5-40f3-8074-d7dd2d54eb05"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:40:35.714515 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:40:35.714452 5378 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ffc2a268-b4b5-40f3-8074-d7dd2d54eb05-trusted-ca-bundle" pod "ffc2a268-b4b5-40f3-8074-d7dd2d54eb05" (UID: "ffc2a268-b4b5-40f3-8074-d7dd2d54eb05"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:40:35.715929 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:40:35.715909 5378 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ffc2a268-b4b5-40f3-8074-d7dd2d54eb05-kube-api-access-sfn6b" pod "ffc2a268-b4b5-40f3-8074-d7dd2d54eb05" (UID: "ffc2a268-b4b5-40f3-8074-d7dd2d54eb05"). InnerVolumeSpecName "kube-api-access-sfn6b". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:40:35.716404 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:40:35.716387 5378 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ffc2a268-b4b5-40f3-8074-d7dd2d54eb05-console-oauth-config" pod "ffc2a268-b4b5-40f3-8074-d7dd2d54eb05" (UID: "ffc2a268-b4b5-40f3-8074-d7dd2d54eb05"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 20:40:35.716446 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:40:35.716424 5378 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ffc2a268-b4b5-40f3-8074-d7dd2d54eb05-console-serving-cert" pod "ffc2a268-b4b5-40f3-8074-d7dd2d54eb05" (UID: "ffc2a268-b4b5-40f3-8074-d7dd2d54eb05"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 20:40:35.815421 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:40:35.815395 5378 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/ffc2a268-b4b5-40f3-8074-d7dd2d54eb05-service-ca\") on node \"ip-10-0-138-149.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:40:35.815421 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:40:35.815418 5378 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sfn6b\" (UniqueName: \"kubernetes.io/projected/ffc2a268-b4b5-40f3-8074-d7dd2d54eb05-kube-api-access-sfn6b\") on node \"ip-10-0-138-149.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:40:35.815556 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:40:35.815428 5378 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/ffc2a268-b4b5-40f3-8074-d7dd2d54eb05-console-serving-cert\") on node \"ip-10-0-138-149.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:40:35.815556 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:40:35.815437 5378 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/ffc2a268-b4b5-40f3-8074-d7dd2d54eb05-console-config\") on node \"ip-10-0-138-149.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:40:35.815556 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:40:35.815445 5378 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ffc2a268-b4b5-40f3-8074-d7dd2d54eb05-trusted-ca-bundle\") on node \"ip-10-0-138-149.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:40:35.815556 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:40:35.815454 5378 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/ffc2a268-b4b5-40f3-8074-d7dd2d54eb05-console-oauth-config\") on node \"ip-10-0-138-149.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:40:36.237035 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:40:36.237002 5378 generic.go:356] "Generic (PLEG): container finished" podID="ffc2a268-b4b5-40f3-8074-d7dd2d54eb05" containerID="cb67a07b2e9f6c66c3e308c96c10a9c7d43ecec89bd72e23592270f6445aaf2f" exitCode=0 Oct 30 20:40:36.237204 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:40:36.237048 5378 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7759588d67-q7dwh" Oct 30 20:40:36.237204 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:40:36.237070 5378 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7759588d67-q7dwh" event={"ID":"ffc2a268-b4b5-40f3-8074-d7dd2d54eb05","Type":"ContainerDied","Data":"cb67a07b2e9f6c66c3e308c96c10a9c7d43ecec89bd72e23592270f6445aaf2f"} Oct 30 20:40:36.237204 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:40:36.237109 5378 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7759588d67-q7dwh" event={"ID":"ffc2a268-b4b5-40f3-8074-d7dd2d54eb05","Type":"ContainerDied","Data":"99e1181265ff11a762b4d976bc3e8d0f36abda834c72aee8e346f2736c550721"} Oct 30 20:40:36.237204 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:40:36.237130 5378 scope.go:122] "RemoveContainer" containerID="cb67a07b2e9f6c66c3e308c96c10a9c7d43ecec89bd72e23592270f6445aaf2f" Oct 30 20:40:36.244630 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:40:36.244615 5378 scope.go:122] "RemoveContainer" containerID="cb67a07b2e9f6c66c3e308c96c10a9c7d43ecec89bd72e23592270f6445aaf2f" Oct 30 20:40:36.244896 ip-10-0-138-149 kubenswrapper[5378]: E1030 20:40:36.244875 5378 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cb67a07b2e9f6c66c3e308c96c10a9c7d43ecec89bd72e23592270f6445aaf2f\": container with ID starting with cb67a07b2e9f6c66c3e308c96c10a9c7d43ecec89bd72e23592270f6445aaf2f not found: ID does not exist" containerID="cb67a07b2e9f6c66c3e308c96c10a9c7d43ecec89bd72e23592270f6445aaf2f" Oct 30 20:40:36.244946 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:40:36.244907 5378 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cb67a07b2e9f6c66c3e308c96c10a9c7d43ecec89bd72e23592270f6445aaf2f"} err="failed to get container status \"cb67a07b2e9f6c66c3e308c96c10a9c7d43ecec89bd72e23592270f6445aaf2f\": rpc error: code = NotFound desc = could not find container \"cb67a07b2e9f6c66c3e308c96c10a9c7d43ecec89bd72e23592270f6445aaf2f\": container with ID starting with cb67a07b2e9f6c66c3e308c96c10a9c7d43ecec89bd72e23592270f6445aaf2f not found: ID does not exist" Oct 30 20:40:36.386403 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:40:36.386348 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 20:40:36.401046 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:40:36.401021 5378 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-7759588d67-q7dwh"] Oct 30 20:40:36.451412 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:40:36.451384 5378 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-7759588d67-q7dwh"] Oct 30 20:40:37.738281 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:40:37.738238 5378 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ffc2a268-b4b5-40f3-8074-d7dd2d54eb05" path="/var/lib/kubelet/pods/ffc2a268-b4b5-40f3-8074-d7dd2d54eb05/volumes" Oct 30 20:40:48.835912 ip-10-0-138-149 kubenswrapper[5378]: E1030 20:40:48.835875 5378 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q6mx7" containerName="extractor" Oct 30 20:41:06.719745 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:41:06.719711 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 20:41:06.891167 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:41:06.891126 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 20:41:08.854735 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:41:08.854695 5378 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console/console-56746464c8-dhx6m"] Oct 30 20:41:08.855220 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:41:08.855077 5378 cpu_manager.go:414] "containerMap: removing container" podUID="ffc2a268-b4b5-40f3-8074-d7dd2d54eb05" containerName="console" Oct 30 20:41:08.855220 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:41:08.855094 5378 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ffc2a268-b4b5-40f3-8074-d7dd2d54eb05" containerName="console" Oct 30 20:41:08.855220 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:41:08.855154 5378 memory_manager.go:365] "RemoveStaleState removing state" podUID="ffc2a268-b4b5-40f3-8074-d7dd2d54eb05" containerName="console" Oct 30 20:41:08.856953 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:41:08.856930 5378 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-56746464c8-dhx6m" Oct 30 20:41:08.957236 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:41:08.957206 5378 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-56746464c8-dhx6m"] Oct 30 20:41:09.048265 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:41:09.048233 5378 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/1b476d09-e912-4121-ac44-f8428c36bf26-service-ca\") pod \"console-56746464c8-dhx6m\" (UID: \"1b476d09-e912-4121-ac44-f8428c36bf26\") " pod="openshift-console/console-56746464c8-dhx6m" Oct 30 20:41:09.048400 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:41:09.048285 5378 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/1b476d09-e912-4121-ac44-f8428c36bf26-tmp\") pod \"console-56746464c8-dhx6m\" (UID: \"1b476d09-e912-4121-ac44-f8428c36bf26\") " pod="openshift-console/console-56746464c8-dhx6m" Oct 30 20:41:09.048400 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:41:09.048375 5378 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/1b476d09-e912-4121-ac44-f8428c36bf26-console-config\") pod \"console-56746464c8-dhx6m\" (UID: \"1b476d09-e912-4121-ac44-f8428c36bf26\") " pod="openshift-console/console-56746464c8-dhx6m" Oct 30 20:41:09.048483 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:41:09.048403 5378 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/1b476d09-e912-4121-ac44-f8428c36bf26-console-serving-cert\") pod \"console-56746464c8-dhx6m\" (UID: \"1b476d09-e912-4121-ac44-f8428c36bf26\") " pod="openshift-console/console-56746464c8-dhx6m" Oct 30 20:41:09.048483 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:41:09.048428 5378 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v8qp5\" (UniqueName: \"kubernetes.io/projected/1b476d09-e912-4121-ac44-f8428c36bf26-kube-api-access-v8qp5\") pod \"console-56746464c8-dhx6m\" (UID: \"1b476d09-e912-4121-ac44-f8428c36bf26\") " pod="openshift-console/console-56746464c8-dhx6m" Oct 30 20:41:09.048674 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:41:09.048476 5378 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1b476d09-e912-4121-ac44-f8428c36bf26-trusted-ca-bundle\") pod \"console-56746464c8-dhx6m\" (UID: \"1b476d09-e912-4121-ac44-f8428c36bf26\") " pod="openshift-console/console-56746464c8-dhx6m" Oct 30 20:41:09.048674 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:41:09.048537 5378 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/1b476d09-e912-4121-ac44-f8428c36bf26-oauth-serving-cert\") pod \"console-56746464c8-dhx6m\" (UID: \"1b476d09-e912-4121-ac44-f8428c36bf26\") " pod="openshift-console/console-56746464c8-dhx6m" Oct 30 20:41:09.048674 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:41:09.048595 5378 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/1b476d09-e912-4121-ac44-f8428c36bf26-console-oauth-config\") pod \"console-56746464c8-dhx6m\" (UID: \"1b476d09-e912-4121-ac44-f8428c36bf26\") " pod="openshift-console/console-56746464c8-dhx6m" Oct 30 20:41:09.116932 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:41:09.116868 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 20:41:09.149282 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:41:09.149260 5378 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/1b476d09-e912-4121-ac44-f8428c36bf26-console-serving-cert\") pod \"console-56746464c8-dhx6m\" (UID: \"1b476d09-e912-4121-ac44-f8428c36bf26\") " pod="openshift-console/console-56746464c8-dhx6m" Oct 30 20:41:09.149373 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:41:09.149288 5378 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v8qp5\" (UniqueName: \"kubernetes.io/projected/1b476d09-e912-4121-ac44-f8428c36bf26-kube-api-access-v8qp5\") pod \"console-56746464c8-dhx6m\" (UID: \"1b476d09-e912-4121-ac44-f8428c36bf26\") " pod="openshift-console/console-56746464c8-dhx6m" Oct 30 20:41:09.149373 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:41:09.149310 5378 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1b476d09-e912-4121-ac44-f8428c36bf26-trusted-ca-bundle\") pod \"console-56746464c8-dhx6m\" (UID: \"1b476d09-e912-4121-ac44-f8428c36bf26\") " pod="openshift-console/console-56746464c8-dhx6m" Oct 30 20:41:09.149373 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:41:09.149324 5378 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/1b476d09-e912-4121-ac44-f8428c36bf26-oauth-serving-cert\") pod \"console-56746464c8-dhx6m\" (UID: \"1b476d09-e912-4121-ac44-f8428c36bf26\") " pod="openshift-console/console-56746464c8-dhx6m" Oct 30 20:41:09.149373 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:41:09.149338 5378 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/1b476d09-e912-4121-ac44-f8428c36bf26-console-oauth-config\") pod \"console-56746464c8-dhx6m\" (UID: \"1b476d09-e912-4121-ac44-f8428c36bf26\") " pod="openshift-console/console-56746464c8-dhx6m" Oct 30 20:41:09.149553 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:41:09.149461 5378 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/1b476d09-e912-4121-ac44-f8428c36bf26-service-ca\") pod \"console-56746464c8-dhx6m\" (UID: \"1b476d09-e912-4121-ac44-f8428c36bf26\") " pod="openshift-console/console-56746464c8-dhx6m" Oct 30 20:41:09.149553 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:41:09.149539 5378 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/1b476d09-e912-4121-ac44-f8428c36bf26-tmp\") pod \"console-56746464c8-dhx6m\" (UID: \"1b476d09-e912-4121-ac44-f8428c36bf26\") " pod="openshift-console/console-56746464c8-dhx6m" Oct 30 20:41:09.149679 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:41:09.149601 5378 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/1b476d09-e912-4121-ac44-f8428c36bf26-console-config\") pod \"console-56746464c8-dhx6m\" (UID: \"1b476d09-e912-4121-ac44-f8428c36bf26\") " pod="openshift-console/console-56746464c8-dhx6m" Oct 30 20:41:09.149895 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:41:09.149872 5378 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/1b476d09-e912-4121-ac44-f8428c36bf26-tmp\") pod \"console-56746464c8-dhx6m\" (UID: \"1b476d09-e912-4121-ac44-f8428c36bf26\") " pod="openshift-console/console-56746464c8-dhx6m" Oct 30 20:41:09.150147 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:41:09.150129 5378 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/1b476d09-e912-4121-ac44-f8428c36bf26-oauth-serving-cert\") pod \"console-56746464c8-dhx6m\" (UID: \"1b476d09-e912-4121-ac44-f8428c36bf26\") " pod="openshift-console/console-56746464c8-dhx6m" Oct 30 20:41:09.150312 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:41:09.150288 5378 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1b476d09-e912-4121-ac44-f8428c36bf26-trusted-ca-bundle\") pod \"console-56746464c8-dhx6m\" (UID: \"1b476d09-e912-4121-ac44-f8428c36bf26\") " pod="openshift-console/console-56746464c8-dhx6m" Oct 30 20:41:09.150744 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:41:09.150722 5378 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/1b476d09-e912-4121-ac44-f8428c36bf26-console-config\") pod \"console-56746464c8-dhx6m\" (UID: \"1b476d09-e912-4121-ac44-f8428c36bf26\") " pod="openshift-console/console-56746464c8-dhx6m" Oct 30 20:41:09.150831 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:41:09.150723 5378 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/1b476d09-e912-4121-ac44-f8428c36bf26-service-ca\") pod \"console-56746464c8-dhx6m\" (UID: \"1b476d09-e912-4121-ac44-f8428c36bf26\") " pod="openshift-console/console-56746464c8-dhx6m" Oct 30 20:41:09.151960 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:41:09.151942 5378 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/1b476d09-e912-4121-ac44-f8428c36bf26-console-serving-cert\") pod \"console-56746464c8-dhx6m\" (UID: \"1b476d09-e912-4121-ac44-f8428c36bf26\") " pod="openshift-console/console-56746464c8-dhx6m" Oct 30 20:41:09.152099 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:41:09.152069 5378 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/1b476d09-e912-4121-ac44-f8428c36bf26-console-oauth-config\") pod \"console-56746464c8-dhx6m\" (UID: \"1b476d09-e912-4121-ac44-f8428c36bf26\") " pod="openshift-console/console-56746464c8-dhx6m" Oct 30 20:41:09.206889 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:41:09.206862 5378 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v8qp5\" (UniqueName: \"kubernetes.io/projected/1b476d09-e912-4121-ac44-f8428c36bf26-kube-api-access-v8qp5\") pod \"console-56746464c8-dhx6m\" (UID: \"1b476d09-e912-4121-ac44-f8428c36bf26\") " pod="openshift-console/console-56746464c8-dhx6m" Oct 30 20:41:09.256278 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:41:09.256235 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 20:41:09.466415 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:41:09.466343 5378 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-56746464c8-dhx6m" Oct 30 20:41:09.495279 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:41:09.495248 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 20:41:09.612751 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:41:09.612719 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 20:41:09.832591 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:41:09.832512 5378 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-56746464c8-dhx6m"] Oct 30 20:41:09.867069 ip-10-0-138-149 kubenswrapper[5378]: W1030 20:41:09.867045 5378 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1b476d09_e912_4121_ac44_f8428c36bf26.slice/crio-45b56680b515dfaba8feda3df5bfd1c4e6841eb293928787dafdd52c52cd1b89 WatchSource:0}: Error finding container 45b56680b515dfaba8feda3df5bfd1c4e6841eb293928787dafdd52c52cd1b89: Status 404 returned error can't find the container with id 45b56680b515dfaba8feda3df5bfd1c4e6841eb293928787dafdd52c52cd1b89 Oct 30 20:41:10.346300 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:41:10.346261 5378 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-56746464c8-dhx6m" event={"ID":"1b476d09-e912-4121-ac44-f8428c36bf26","Type":"ContainerStarted","Data":"db7017f08ee866c3076d6f19cbf1693dd551fa6de69772a7c291e22a993d1ec0"} Oct 30 20:41:10.346484 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:41:10.346316 5378 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-56746464c8-dhx6m" event={"ID":"1b476d09-e912-4121-ac44-f8428c36bf26","Type":"ContainerStarted","Data":"45b56680b515dfaba8feda3df5bfd1c4e6841eb293928787dafdd52c52cd1b89"} Oct 30 20:41:10.463780 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:41:10.463730 5378 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-console/console-56746464c8-dhx6m" podStartSLOduration=2.463716425 podStartE2EDuration="2.463716425s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:41:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:41:10.46343552 +0000 UTC m=+160.891549439" watchObservedRunningTime="2026-10-30 20:41:10.463716425 +0000 UTC m=+160.891830341" Oct 30 20:41:18.835163 ip-10-0-138-149 kubenswrapper[5378]: E1030 20:41:18.835126 5378 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q6mx7" containerName="extractor" Oct 30 20:41:19.350073 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:41:19.350044 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 20:41:19.467280 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:41:19.467247 5378 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-56746464c8-dhx6m" Oct 30 20:41:19.467445 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:41:19.467327 5378 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-56746464c8-dhx6m" Oct 30 20:41:19.471918 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:41:19.471899 5378 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-56746464c8-dhx6m" Oct 30 20:41:19.473941 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:41:19.473924 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 20:41:20.382201 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:41:20.382165 5378 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-56746464c8-dhx6m" Oct 30 20:41:20.567358 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:41:20.567329 5378 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-7b796cd949-sd2d4"] Oct 30 20:41:29.012965 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:41:29.012901 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 20:41:29.148079 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:41:29.148047 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 20:41:36.712977 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:41:36.712942 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 20:41:45.586133 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:41:45.586029 5378 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-console/console-7b796cd949-sd2d4" podUID="14e39214-df20-411a-a433-f9dcd9d722c6" containerName="console" containerID="cri-o://6d02b2264e0316008d1957de8bc21e8e04318fc9e0d9ec80b405faf02e87582d" gracePeriod=15 Oct 30 20:41:45.826863 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:41:45.826841 5378 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7b796cd949-sd2d4" Oct 30 20:41:45.912626 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:41:45.912599 5378 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/14e39214-df20-411a-a433-f9dcd9d722c6-tmp\" (UniqueName: \"kubernetes.io/empty-dir/14e39214-df20-411a-a433-f9dcd9d722c6-tmp\") pod \"14e39214-df20-411a-a433-f9dcd9d722c6\" (UID: \"14e39214-df20-411a-a433-f9dcd9d722c6\") " Oct 30 20:41:45.912750 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:41:45.912637 5378 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/14e39214-df20-411a-a433-f9dcd9d722c6-service-ca\" (UniqueName: \"kubernetes.io/configmap/14e39214-df20-411a-a433-f9dcd9d722c6-service-ca\") pod \"14e39214-df20-411a-a433-f9dcd9d722c6\" (UID: \"14e39214-df20-411a-a433-f9dcd9d722c6\") " Oct 30 20:41:45.912750 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:41:45.912692 5378 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/14e39214-df20-411a-a433-f9dcd9d722c6-console-serving-cert\" (UniqueName: \"kubernetes.io/secret/14e39214-df20-411a-a433-f9dcd9d722c6-console-serving-cert\") pod \"14e39214-df20-411a-a433-f9dcd9d722c6\" (UID: \"14e39214-df20-411a-a433-f9dcd9d722c6\") " Oct 30 20:41:45.912750 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:41:45.912723 5378 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/14e39214-df20-411a-a433-f9dcd9d722c6-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/14e39214-df20-411a-a433-f9dcd9d722c6-trusted-ca-bundle\") pod \"14e39214-df20-411a-a433-f9dcd9d722c6\" (UID: \"14e39214-df20-411a-a433-f9dcd9d722c6\") " Oct 30 20:41:45.912909 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:41:45.912760 5378 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/14e39214-df20-411a-a433-f9dcd9d722c6-console-config\" (UniqueName: \"kubernetes.io/configmap/14e39214-df20-411a-a433-f9dcd9d722c6-console-config\") pod \"14e39214-df20-411a-a433-f9dcd9d722c6\" (UID: \"14e39214-df20-411a-a433-f9dcd9d722c6\") " Oct 30 20:41:45.912909 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:41:45.912824 5378 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/14e39214-df20-411a-a433-f9dcd9d722c6-kube-api-access-plb8b\" (UniqueName: \"kubernetes.io/projected/14e39214-df20-411a-a433-f9dcd9d722c6-kube-api-access-plb8b\") pod \"14e39214-df20-411a-a433-f9dcd9d722c6\" (UID: \"14e39214-df20-411a-a433-f9dcd9d722c6\") " Oct 30 20:41:45.912909 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:41:45.912856 5378 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/14e39214-df20-411a-a433-f9dcd9d722c6-console-oauth-config\" (UniqueName: \"kubernetes.io/secret/14e39214-df20-411a-a433-f9dcd9d722c6-console-oauth-config\") pod \"14e39214-df20-411a-a433-f9dcd9d722c6\" (UID: \"14e39214-df20-411a-a433-f9dcd9d722c6\") " Oct 30 20:41:45.912909 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:41:45.912883 5378 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/14e39214-df20-411a-a433-f9dcd9d722c6-oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/14e39214-df20-411a-a433-f9dcd9d722c6-oauth-serving-cert\") pod \"14e39214-df20-411a-a433-f9dcd9d722c6\" (UID: \"14e39214-df20-411a-a433-f9dcd9d722c6\") " Oct 30 20:41:45.912909 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:41:45.912852 5378 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/14e39214-df20-411a-a433-f9dcd9d722c6-tmp" pod "14e39214-df20-411a-a433-f9dcd9d722c6" (UID: "14e39214-df20-411a-a433-f9dcd9d722c6"). InnerVolumeSpecName "tmp". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:41:45.913151 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:41:45.913035 5378 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/14e39214-df20-411a-a433-f9dcd9d722c6-service-ca" pod "14e39214-df20-411a-a433-f9dcd9d722c6" (UID: "14e39214-df20-411a-a433-f9dcd9d722c6"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:41:45.913220 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:41:45.913089 5378 reconciler_common.go:299] "Volume detached for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/14e39214-df20-411a-a433-f9dcd9d722c6-tmp\") on node \"ip-10-0-138-149.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:41:45.913220 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:41:45.913216 5378 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/14e39214-df20-411a-a433-f9dcd9d722c6-service-ca\") on node \"ip-10-0-138-149.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:41:45.913305 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:41:45.913278 5378 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/14e39214-df20-411a-a433-f9dcd9d722c6-trusted-ca-bundle" pod "14e39214-df20-411a-a433-f9dcd9d722c6" (UID: "14e39214-df20-411a-a433-f9dcd9d722c6"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:41:45.913419 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:41:45.913399 5378 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/14e39214-df20-411a-a433-f9dcd9d722c6-oauth-serving-cert" pod "14e39214-df20-411a-a433-f9dcd9d722c6" (UID: "14e39214-df20-411a-a433-f9dcd9d722c6"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:41:45.913480 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:41:45.913394 5378 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/14e39214-df20-411a-a433-f9dcd9d722c6-console-config" pod "14e39214-df20-411a-a433-f9dcd9d722c6" (UID: "14e39214-df20-411a-a433-f9dcd9d722c6"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:41:45.914930 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:41:45.914905 5378 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/14e39214-df20-411a-a433-f9dcd9d722c6-console-serving-cert" pod "14e39214-df20-411a-a433-f9dcd9d722c6" (UID: "14e39214-df20-411a-a433-f9dcd9d722c6"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 20:41:45.915042 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:41:45.914974 5378 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/14e39214-df20-411a-a433-f9dcd9d722c6-console-oauth-config" pod "14e39214-df20-411a-a433-f9dcd9d722c6" (UID: "14e39214-df20-411a-a433-f9dcd9d722c6"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 20:41:45.915042 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:41:45.914973 5378 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/14e39214-df20-411a-a433-f9dcd9d722c6-kube-api-access-plb8b" pod "14e39214-df20-411a-a433-f9dcd9d722c6" (UID: "14e39214-df20-411a-a433-f9dcd9d722c6"). InnerVolumeSpecName "kube-api-access-plb8b". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:41:46.013786 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:41:46.013760 5378 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/14e39214-df20-411a-a433-f9dcd9d722c6-console-serving-cert\") on node \"ip-10-0-138-149.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:41:46.013786 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:41:46.013783 5378 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/14e39214-df20-411a-a433-f9dcd9d722c6-trusted-ca-bundle\") on node \"ip-10-0-138-149.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:41:46.013936 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:41:46.013793 5378 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/14e39214-df20-411a-a433-f9dcd9d722c6-console-config\") on node \"ip-10-0-138-149.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:41:46.013936 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:41:46.013802 5378 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-plb8b\" (UniqueName: \"kubernetes.io/projected/14e39214-df20-411a-a433-f9dcd9d722c6-kube-api-access-plb8b\") on node \"ip-10-0-138-149.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:41:46.013936 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:41:46.013813 5378 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/14e39214-df20-411a-a433-f9dcd9d722c6-console-oauth-config\") on node \"ip-10-0-138-149.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:41:46.013936 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:41:46.013821 5378 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/14e39214-df20-411a-a433-f9dcd9d722c6-oauth-serving-cert\") on node \"ip-10-0-138-149.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:41:46.463349 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:41:46.463313 5378 generic.go:356] "Generic (PLEG): container finished" podID="14e39214-df20-411a-a433-f9dcd9d722c6" containerID="6d02b2264e0316008d1957de8bc21e8e04318fc9e0d9ec80b405faf02e87582d" exitCode=0 Oct 30 20:41:46.463506 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:41:46.463362 5378 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7b796cd949-sd2d4" Oct 30 20:41:46.463506 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:41:46.463358 5378 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7b796cd949-sd2d4" event={"ID":"14e39214-df20-411a-a433-f9dcd9d722c6","Type":"ContainerDied","Data":"6d02b2264e0316008d1957de8bc21e8e04318fc9e0d9ec80b405faf02e87582d"} Oct 30 20:41:46.463506 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:41:46.463487 5378 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7b796cd949-sd2d4" event={"ID":"14e39214-df20-411a-a433-f9dcd9d722c6","Type":"ContainerDied","Data":"4990891c30b04aaf0520192661169c7a2b89211863c3026c7121608cc5558ccd"} Oct 30 20:41:46.463627 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:41:46.463515 5378 scope.go:122] "RemoveContainer" containerID="6d02b2264e0316008d1957de8bc21e8e04318fc9e0d9ec80b405faf02e87582d" Oct 30 20:41:46.471550 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:41:46.471533 5378 scope.go:122] "RemoveContainer" containerID="6d02b2264e0316008d1957de8bc21e8e04318fc9e0d9ec80b405faf02e87582d" Oct 30 20:41:46.471832 ip-10-0-138-149 kubenswrapper[5378]: E1030 20:41:46.471815 5378 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6d02b2264e0316008d1957de8bc21e8e04318fc9e0d9ec80b405faf02e87582d\": container with ID starting with 6d02b2264e0316008d1957de8bc21e8e04318fc9e0d9ec80b405faf02e87582d not found: ID does not exist" containerID="6d02b2264e0316008d1957de8bc21e8e04318fc9e0d9ec80b405faf02e87582d" Oct 30 20:41:46.471898 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:41:46.471843 5378 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6d02b2264e0316008d1957de8bc21e8e04318fc9e0d9ec80b405faf02e87582d"} err="failed to get container status \"6d02b2264e0316008d1957de8bc21e8e04318fc9e0d9ec80b405faf02e87582d\": rpc error: code = NotFound desc = could not find container \"6d02b2264e0316008d1957de8bc21e8e04318fc9e0d9ec80b405faf02e87582d\": container with ID starting with 6d02b2264e0316008d1957de8bc21e8e04318fc9e0d9ec80b405faf02e87582d not found: ID does not exist" Oct 30 20:41:46.627576 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:41:46.627535 5378 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-7b796cd949-sd2d4"] Oct 30 20:41:46.678004 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:41:46.677975 5378 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-7b796cd949-sd2d4"] Oct 30 20:41:47.737928 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:41:47.737899 5378 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="14e39214-df20-411a-a433-f9dcd9d722c6" path="/var/lib/kubelet/pods/14e39214-df20-411a-a433-f9dcd9d722c6/volumes" Oct 30 20:41:48.834548 ip-10-0-138-149 kubenswrapper[5378]: E1030 20:41:48.834514 5378 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q6mx7" containerName="extractor" Oct 30 20:42:18.835610 ip-10-0-138-149 kubenswrapper[5378]: E1030 20:42:18.835561 5378 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q6mx7" containerName="extractor" Oct 30 20:42:37.063921 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:42:37.063883 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 20:42:48.835657 ip-10-0-138-149 kubenswrapper[5378]: E1030 20:42:48.835620 5378 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q6mx7" containerName="extractor" Oct 30 20:43:14.188513 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:43:14.188432 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 20:43:18.835667 ip-10-0-138-149 kubenswrapper[5378]: E1030 20:43:18.835632 5378 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q6mx7" containerName="extractor" Oct 30 20:43:25.939034 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:43:25.938999 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 20:43:26.237170 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:43:26.237086 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 20:43:27.798888 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:43:27.798855 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 20:43:27.805982 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:43:27.805956 5378 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7876bfcb74-dq8zm"] Oct 30 20:43:27.806242 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:43:27.806229 5378 cpu_manager.go:414] "containerMap: removing container" podUID="14e39214-df20-411a-a433-f9dcd9d722c6" containerName="console" Oct 30 20:43:27.806289 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:43:27.806243 5378 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="14e39214-df20-411a-a433-f9dcd9d722c6" containerName="console" Oct 30 20:43:27.806323 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:43:27.806301 5378 memory_manager.go:365] "RemoveStaleState removing state" podUID="14e39214-df20-411a-a433-f9dcd9d722c6" containerName="console" Oct 30 20:43:27.809037 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:43:27.809021 5378 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-dq8zm" Oct 30 20:43:27.870411 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:43:27.870384 5378 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\"" Oct 30 20:43:27.870521 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:43:27.870432 5378 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\"" Oct 30 20:43:27.916045 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:43:27.916021 5378 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\"" Oct 30 20:43:27.916133 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:43:27.916059 5378 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\"" Oct 30 20:43:27.916133 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:43:27.916077 5378 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\"" Oct 30 20:43:27.916133 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:43:27.916104 5378 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\"" Oct 30 20:43:27.916254 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:43:27.916139 5378 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-pwj8h\"" Oct 30 20:43:27.918980 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:43:27.918956 5378 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7876bfcb74-dq8zm"] Oct 30 20:43:27.922651 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:43:27.922626 5378 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jr6p2\" (UniqueName: \"kubernetes.io/projected/ad251643-5dd1-42dd-8044-f42d88b0b32d-kube-api-access-jr6p2\") pod \"istiod-openshift-gateway-7876bfcb74-dq8zm\" (UID: \"ad251643-5dd1-42dd-8044-f42d88b0b32d\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-dq8zm" Oct 30 20:43:27.922751 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:43:27.922670 5378 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/ad251643-5dd1-42dd-8044-f42d88b0b32d-cacerts\") pod \"istiod-openshift-gateway-7876bfcb74-dq8zm\" (UID: \"ad251643-5dd1-42dd-8044-f42d88b0b32d\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-dq8zm" Oct 30 20:43:27.922751 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:43:27.922708 5378 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ad251643-5dd1-42dd-8044-f42d88b0b32d-istio-token\") pod \"istiod-openshift-gateway-7876bfcb74-dq8zm\" (UID: \"ad251643-5dd1-42dd-8044-f42d88b0b32d\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-dq8zm" Oct 30 20:43:27.922839 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:43:27.922813 5378 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/ad251643-5dd1-42dd-8044-f42d88b0b32d-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7876bfcb74-dq8zm\" (UID: \"ad251643-5dd1-42dd-8044-f42d88b0b32d\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-dq8zm" Oct 30 20:43:27.922893 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:43:27.922849 5378 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/ad251643-5dd1-42dd-8044-f42d88b0b32d-local-certs\") pod \"istiod-openshift-gateway-7876bfcb74-dq8zm\" (UID: \"ad251643-5dd1-42dd-8044-f42d88b0b32d\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-dq8zm" Oct 30 20:43:27.922893 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:43:27.922869 5378 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/ad251643-5dd1-42dd-8044-f42d88b0b32d-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7876bfcb74-dq8zm\" (UID: \"ad251643-5dd1-42dd-8044-f42d88b0b32d\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-dq8zm" Oct 30 20:43:27.922893 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:43:27.922888 5378 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/ad251643-5dd1-42dd-8044-f42d88b0b32d-istio-kubeconfig\") pod \"istiod-openshift-gateway-7876bfcb74-dq8zm\" (UID: \"ad251643-5dd1-42dd-8044-f42d88b0b32d\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-dq8zm" Oct 30 20:43:27.959077 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:43:27.959054 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 20:43:28.024167 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:43:28.024139 5378 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ad251643-5dd1-42dd-8044-f42d88b0b32d-istio-token\") pod \"istiod-openshift-gateway-7876bfcb74-dq8zm\" (UID: \"ad251643-5dd1-42dd-8044-f42d88b0b32d\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-dq8zm" Oct 30 20:43:28.024315 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:43:28.024174 5378 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/ad251643-5dd1-42dd-8044-f42d88b0b32d-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7876bfcb74-dq8zm\" (UID: \"ad251643-5dd1-42dd-8044-f42d88b0b32d\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-dq8zm" Oct 30 20:43:28.024315 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:43:28.024195 5378 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/ad251643-5dd1-42dd-8044-f42d88b0b32d-local-certs\") pod \"istiod-openshift-gateway-7876bfcb74-dq8zm\" (UID: \"ad251643-5dd1-42dd-8044-f42d88b0b32d\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-dq8zm" Oct 30 20:43:28.024315 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:43:28.024219 5378 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/ad251643-5dd1-42dd-8044-f42d88b0b32d-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7876bfcb74-dq8zm\" (UID: \"ad251643-5dd1-42dd-8044-f42d88b0b32d\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-dq8zm" Oct 30 20:43:28.024315 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:43:28.024299 5378 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/ad251643-5dd1-42dd-8044-f42d88b0b32d-istio-kubeconfig\") pod \"istiod-openshift-gateway-7876bfcb74-dq8zm\" (UID: \"ad251643-5dd1-42dd-8044-f42d88b0b32d\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-dq8zm" Oct 30 20:43:28.024526 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:43:28.024365 5378 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jr6p2\" (UniqueName: \"kubernetes.io/projected/ad251643-5dd1-42dd-8044-f42d88b0b32d-kube-api-access-jr6p2\") pod \"istiod-openshift-gateway-7876bfcb74-dq8zm\" (UID: \"ad251643-5dd1-42dd-8044-f42d88b0b32d\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-dq8zm" Oct 30 20:43:28.024526 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:43:28.024410 5378 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/ad251643-5dd1-42dd-8044-f42d88b0b32d-cacerts\") pod \"istiod-openshift-gateway-7876bfcb74-dq8zm\" (UID: \"ad251643-5dd1-42dd-8044-f42d88b0b32d\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-dq8zm" Oct 30 20:43:28.024909 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:43:28.024887 5378 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/ad251643-5dd1-42dd-8044-f42d88b0b32d-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7876bfcb74-dq8zm\" (UID: \"ad251643-5dd1-42dd-8044-f42d88b0b32d\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-dq8zm" Oct 30 20:43:28.026559 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:43:28.026534 5378 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/ad251643-5dd1-42dd-8044-f42d88b0b32d-local-certs\") pod \"istiod-openshift-gateway-7876bfcb74-dq8zm\" (UID: \"ad251643-5dd1-42dd-8044-f42d88b0b32d\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-dq8zm" Oct 30 20:43:28.026722 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:43:28.026691 5378 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/ad251643-5dd1-42dd-8044-f42d88b0b32d-istio-kubeconfig\") pod \"istiod-openshift-gateway-7876bfcb74-dq8zm\" (UID: \"ad251643-5dd1-42dd-8044-f42d88b0b32d\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-dq8zm" Oct 30 20:43:28.026955 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:43:28.026937 5378 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/ad251643-5dd1-42dd-8044-f42d88b0b32d-cacerts\") pod \"istiod-openshift-gateway-7876bfcb74-dq8zm\" (UID: \"ad251643-5dd1-42dd-8044-f42d88b0b32d\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-dq8zm" Oct 30 20:43:28.027036 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:43:28.027007 5378 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/ad251643-5dd1-42dd-8044-f42d88b0b32d-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7876bfcb74-dq8zm\" (UID: \"ad251643-5dd1-42dd-8044-f42d88b0b32d\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-dq8zm" Oct 30 20:43:28.084398 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:43:28.084345 5378 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jr6p2\" (UniqueName: \"kubernetes.io/projected/ad251643-5dd1-42dd-8044-f42d88b0b32d-kube-api-access-jr6p2\") pod \"istiod-openshift-gateway-7876bfcb74-dq8zm\" (UID: \"ad251643-5dd1-42dd-8044-f42d88b0b32d\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-dq8zm" Oct 30 20:43:28.084398 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:43:28.084370 5378 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ad251643-5dd1-42dd-8044-f42d88b0b32d-istio-token\") pod \"istiod-openshift-gateway-7876bfcb74-dq8zm\" (UID: \"ad251643-5dd1-42dd-8044-f42d88b0b32d\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-dq8zm" Oct 30 20:43:28.119285 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:43:28.119266 5378 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-dq8zm" Oct 30 20:43:28.540994 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:43:28.540963 5378 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7876bfcb74-dq8zm"] Oct 30 20:43:28.544224 ip-10-0-138-149 kubenswrapper[5378]: W1030 20:43:28.544198 5378 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podad251643_5dd1_42dd_8044_f42d88b0b32d.slice/crio-3dde6e773287427695d008dcca8de7501c4e0e4f26b9adbf1f7d766032436d20 WatchSource:0}: Error finding container 3dde6e773287427695d008dcca8de7501c4e0e4f26b9adbf1f7d766032436d20: Status 404 returned error can't find the container with id 3dde6e773287427695d008dcca8de7501c4e0e4f26b9adbf1f7d766032436d20 Oct 30 20:43:28.789766 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:43:28.789729 5378 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-dq8zm" event={"ID":"ad251643-5dd1-42dd-8044-f42d88b0b32d","Type":"ContainerStarted","Data":"3dde6e773287427695d008dcca8de7501c4e0e4f26b9adbf1f7d766032436d20"} Oct 30 20:43:29.232028 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:43:29.231984 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 20:43:29.754600 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:43:29.754557 5378 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-7x9jg_c2ff359c-cf92-4a5b-8610-f6eb41acbe73/ovn-acl-logging/0.log" Oct 30 20:43:29.756766 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:43:29.756740 5378 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-7x9jg_c2ff359c-cf92-4a5b-8610-f6eb41acbe73/ovn-acl-logging/0.log" Oct 30 20:43:32.617806 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:43:32.617773 5378 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 20:43:32.618224 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:43:32.617846 5378 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 20:43:32.804146 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:43:32.804110 5378 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-dq8zm" event={"ID":"ad251643-5dd1-42dd-8044-f42d88b0b32d","Type":"ContainerStarted","Data":"a3dab19580f4a3c1e4e0324073bfb2d6ce63d8a0eae553d25b0ccbe7e4b1fa2e"} Oct 30 20:43:32.804323 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:43:32.804239 5378 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-dq8zm" Oct 30 20:43:33.023902 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:43:33.023791 5378 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-dq8zm" podStartSLOduration=1.952949976 podStartE2EDuration="6.023778005s" totalImagesPullingTime="4.070828029s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:43:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:43:32.973205653 +0000 UTC m=+303.401319571" watchObservedRunningTime="2026-10-30 20:43:33.023778005 +0000 UTC m=+303.451891919" Oct 30 20:43:33.809721 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:43:33.809684 5378 patch_prober.go:29] interesting pod/istiod-openshift-gateway-7876bfcb74-dq8zm container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 20:43:33.810177 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:43:33.809748 5378 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-dq8zm" podUID="ad251643-5dd1-42dd-8044-f42d88b0b32d" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 20:43:33.977780 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:43:33.977741 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 20:43:34.121581 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:43:34.121540 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 20:43:34.543530 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:43:34.543456 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 20:43:34.730322 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:43:34.730294 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 20:43:34.865756 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:43:34.865723 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 20:43:36.809195 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:43:36.809167 5378 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-dq8zm" Oct 30 20:43:37.397554 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:43:37.397492 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 20:43:38.607671 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:43:38.607633 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 20:43:48.835742 ip-10-0-138-149 kubenswrapper[5378]: E1030 20:43:48.835696 5378 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q6mx7" containerName="extractor" Oct 30 20:44:02.099660 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:44:02.099626 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 20:44:02.984890 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:44:02.984853 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 20:44:04.086251 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:44:04.086210 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 20:44:05.718464 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:44:05.718429 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 20:44:05.846258 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:44:05.846207 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 20:44:09.001249 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:44:09.001214 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 20:44:09.248291 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:44:09.248255 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 20:44:09.417181 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:44:09.417124 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 20:44:10.090368 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:44:10.090337 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 20:44:10.409072 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:44:10.409039 5378 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-57f958cfdb-rfvmm"] Oct 30 20:44:10.412121 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:44:10.412105 5378 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-57f958cfdb-rfvmm" Oct 30 20:44:10.482295 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:44:10.482266 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 20:44:10.519100 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:44:10.519073 5378 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-manager-service-cert\"" Oct 30 20:44:10.519100 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:44:10.519073 5378 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-webhook-cert\"" Oct 30 20:44:10.519307 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:44:10.519110 5378 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub-operator-system\"/\"openshift-service-ca.crt\"" Oct 30 20:44:10.519307 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:44:10.519142 5378 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-metrics-tls\"" Oct 30 20:44:10.519307 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:44:10.519207 5378 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub-operator-system\"/\"kube-root-ca.crt\"" Oct 30 20:44:10.519307 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:44:10.519205 5378 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-manager-dockercfg-wrjh7\"" Oct 30 20:44:10.521714 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:44:10.521687 5378 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-57f958cfdb-rfvmm"] Oct 30 20:44:10.544648 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:44:10.544626 5378 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/c861fe92-6105-4377-8ab6-6ab62e86ebce-webhook-cert\") pod \"opendatahub-operator-controller-manager-57f958cfdb-rfvmm\" (UID: \"c861fe92-6105-4377-8ab6-6ab62e86ebce\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-57f958cfdb-rfvmm" Oct 30 20:44:10.544756 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:44:10.544660 5378 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/c861fe92-6105-4377-8ab6-6ab62e86ebce-tmp\") pod \"opendatahub-operator-controller-manager-57f958cfdb-rfvmm\" (UID: \"c861fe92-6105-4377-8ab6-6ab62e86ebce\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-57f958cfdb-rfvmm" Oct 30 20:44:10.544756 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:44:10.544691 5378 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/c861fe92-6105-4377-8ab6-6ab62e86ebce-manifests\") pod \"opendatahub-operator-controller-manager-57f958cfdb-rfvmm\" (UID: \"c861fe92-6105-4377-8ab6-6ab62e86ebce\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-57f958cfdb-rfvmm" Oct 30 20:44:10.544830 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:44:10.544753 5378 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/c861fe92-6105-4377-8ab6-6ab62e86ebce-apiservice-cert\") pod \"opendatahub-operator-controller-manager-57f958cfdb-rfvmm\" (UID: \"c861fe92-6105-4377-8ab6-6ab62e86ebce\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-57f958cfdb-rfvmm" Oct 30 20:44:10.544862 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:44:10.544843 5378 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bfv7p\" (UniqueName: \"kubernetes.io/projected/c861fe92-6105-4377-8ab6-6ab62e86ebce-kube-api-access-bfv7p\") pod \"opendatahub-operator-controller-manager-57f958cfdb-rfvmm\" (UID: \"c861fe92-6105-4377-8ab6-6ab62e86ebce\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-57f958cfdb-rfvmm" Oct 30 20:44:10.544895 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:44:10.544873 5378 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/c861fe92-6105-4377-8ab6-6ab62e86ebce-metrics-cert\") pod \"opendatahub-operator-controller-manager-57f958cfdb-rfvmm\" (UID: \"c861fe92-6105-4377-8ab6-6ab62e86ebce\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-57f958cfdb-rfvmm" Oct 30 20:44:10.645670 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:44:10.645642 5378 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/c861fe92-6105-4377-8ab6-6ab62e86ebce-manifests\") pod \"opendatahub-operator-controller-manager-57f958cfdb-rfvmm\" (UID: \"c861fe92-6105-4377-8ab6-6ab62e86ebce\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-57f958cfdb-rfvmm" Oct 30 20:44:10.645670 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:44:10.645673 5378 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/c861fe92-6105-4377-8ab6-6ab62e86ebce-apiservice-cert\") pod \"opendatahub-operator-controller-manager-57f958cfdb-rfvmm\" (UID: \"c861fe92-6105-4377-8ab6-6ab62e86ebce\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-57f958cfdb-rfvmm" Oct 30 20:44:10.645836 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:44:10.645699 5378 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bfv7p\" (UniqueName: \"kubernetes.io/projected/c861fe92-6105-4377-8ab6-6ab62e86ebce-kube-api-access-bfv7p\") pod \"opendatahub-operator-controller-manager-57f958cfdb-rfvmm\" (UID: \"c861fe92-6105-4377-8ab6-6ab62e86ebce\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-57f958cfdb-rfvmm" Oct 30 20:44:10.645876 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:44:10.645856 5378 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/c861fe92-6105-4377-8ab6-6ab62e86ebce-metrics-cert\") pod \"opendatahub-operator-controller-manager-57f958cfdb-rfvmm\" (UID: \"c861fe92-6105-4377-8ab6-6ab62e86ebce\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-57f958cfdb-rfvmm" Oct 30 20:44:10.645967 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:44:10.645943 5378 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/c861fe92-6105-4377-8ab6-6ab62e86ebce-webhook-cert\") pod \"opendatahub-operator-controller-manager-57f958cfdb-rfvmm\" (UID: \"c861fe92-6105-4377-8ab6-6ab62e86ebce\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-57f958cfdb-rfvmm" Oct 30 20:44:10.646040 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:44:10.645994 5378 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/c861fe92-6105-4377-8ab6-6ab62e86ebce-tmp\") pod \"opendatahub-operator-controller-manager-57f958cfdb-rfvmm\" (UID: \"c861fe92-6105-4377-8ab6-6ab62e86ebce\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-57f958cfdb-rfvmm" Oct 30 20:44:10.646307 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:44:10.646281 5378 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/c861fe92-6105-4377-8ab6-6ab62e86ebce-manifests\") pod \"opendatahub-operator-controller-manager-57f958cfdb-rfvmm\" (UID: \"c861fe92-6105-4377-8ab6-6ab62e86ebce\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-57f958cfdb-rfvmm" Oct 30 20:44:10.646646 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:44:10.646623 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 20:44:10.648248 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:44:10.648221 5378 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/c861fe92-6105-4377-8ab6-6ab62e86ebce-tmp\") pod \"opendatahub-operator-controller-manager-57f958cfdb-rfvmm\" (UID: \"c861fe92-6105-4377-8ab6-6ab62e86ebce\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-57f958cfdb-rfvmm" Oct 30 20:44:10.648363 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:44:10.648344 5378 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/c861fe92-6105-4377-8ab6-6ab62e86ebce-apiservice-cert\") pod \"opendatahub-operator-controller-manager-57f958cfdb-rfvmm\" (UID: \"c861fe92-6105-4377-8ab6-6ab62e86ebce\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-57f958cfdb-rfvmm" Oct 30 20:44:10.648435 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:44:10.648393 5378 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/c861fe92-6105-4377-8ab6-6ab62e86ebce-metrics-cert\") pod \"opendatahub-operator-controller-manager-57f958cfdb-rfvmm\" (UID: \"c861fe92-6105-4377-8ab6-6ab62e86ebce\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-57f958cfdb-rfvmm" Oct 30 20:44:10.648708 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:44:10.648688 5378 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/c861fe92-6105-4377-8ab6-6ab62e86ebce-webhook-cert\") pod \"opendatahub-operator-controller-manager-57f958cfdb-rfvmm\" (UID: \"c861fe92-6105-4377-8ab6-6ab62e86ebce\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-57f958cfdb-rfvmm" Oct 30 20:44:10.723754 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:44:10.723697 5378 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bfv7p\" (UniqueName: \"kubernetes.io/projected/c861fe92-6105-4377-8ab6-6ab62e86ebce-kube-api-access-bfv7p\") pod \"opendatahub-operator-controller-manager-57f958cfdb-rfvmm\" (UID: \"c861fe92-6105-4377-8ab6-6ab62e86ebce\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-57f958cfdb-rfvmm" Oct 30 20:44:10.726512 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:44:10.726496 5378 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-57f958cfdb-rfvmm" Oct 30 20:44:10.843040 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:44:10.843011 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 20:44:10.984956 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:44:10.984891 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 20:44:11.138380 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:44:11.138351 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 20:44:11.209469 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:44:11.209443 5378 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-57f958cfdb-rfvmm"] Oct 30 20:44:11.212304 ip-10-0-138-149 kubenswrapper[5378]: W1030 20:44:11.212278 5378 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc861fe92_6105_4377_8ab6_6ab62e86ebce.slice/crio-506ba32d102b23c2f39127801840236112d297cc9ffb0f0e2123c3d3795d6a1e WatchSource:0}: Error finding container 506ba32d102b23c2f39127801840236112d297cc9ffb0f0e2123c3d3795d6a1e: Status 404 returned error can't find the container with id 506ba32d102b23c2f39127801840236112d297cc9ffb0f0e2123c3d3795d6a1e Oct 30 20:44:11.213992 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:44:11.213972 5378 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 30 20:44:11.290719 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:44:11.290624 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 20:44:11.531204 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:44:11.531175 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 20:44:11.933846 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:44:11.933812 5378 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-57f958cfdb-rfvmm" event={"ID":"c861fe92-6105-4377-8ab6-6ab62e86ebce","Type":"ContainerStarted","Data":"506ba32d102b23c2f39127801840236112d297cc9ffb0f0e2123c3d3795d6a1e"} Oct 30 20:44:12.008777 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:44:12.008743 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 20:44:14.946221 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:44:14.946181 5378 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-57f958cfdb-rfvmm" event={"ID":"c861fe92-6105-4377-8ab6-6ab62e86ebce","Type":"ContainerStarted","Data":"5460bfd40c0f5d2df6e8284bda076b0966e9c91e11ad0e45855a50a32a3e8766"} Oct 30 20:44:15.149932 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:44:15.149901 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 20:44:15.951341 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:44:15.951299 5378 generic.go:356] "Generic (PLEG): container finished" podID="c861fe92-6105-4377-8ab6-6ab62e86ebce" containerID="5460bfd40c0f5d2df6e8284bda076b0966e9c91e11ad0e45855a50a32a3e8766" exitCode=0 Oct 30 20:44:15.951776 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:44:15.951370 5378 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-57f958cfdb-rfvmm" event={"ID":"c861fe92-6105-4377-8ab6-6ab62e86ebce","Type":"ContainerDied","Data":"5460bfd40c0f5d2df6e8284bda076b0966e9c91e11ad0e45855a50a32a3e8766"} Oct 30 20:44:16.956302 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:44:16.956268 5378 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-57f958cfdb-rfvmm" event={"ID":"c861fe92-6105-4377-8ab6-6ab62e86ebce","Type":"ContainerStarted","Data":"319b23b657d83095990cdc5b9c5f95648d2cfbd60e38fc1984f3106b1f875382"} Oct 30 20:44:16.956714 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:44:16.956391 5378 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-57f958cfdb-rfvmm" Oct 30 20:44:17.192820 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:44:17.192771 5378 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-57f958cfdb-rfvmm" podStartSLOduration=2.280659586 podStartE2EDuration="7.192756678s" totalImagesPullingTime="4.430267077s" totalInitContainerRuntime="481.830015ms" isStatefulPod=false podCreationTimestamp="2026-10-30 20:44:10 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:44:17.142153964 +0000 UTC m=+347.570267901" watchObservedRunningTime="2026-10-30 20:44:17.192756678 +0000 UTC m=+347.620870597" Oct 30 20:44:18.079540 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:44:18.079505 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 20:44:18.227168 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:44:18.227130 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 20:44:18.835222 ip-10-0-138-149 kubenswrapper[5378]: E1030 20:44:18.835183 5378 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q6mx7" containerName="extractor" Oct 30 20:44:19.037311 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:44:19.037262 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 20:44:19.910681 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:44:19.910647 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 20:44:20.511588 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:44:20.511530 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 20:44:21.243578 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:44:21.243517 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 20:44:21.383246 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:44:21.383218 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 20:44:23.927872 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:44:23.927844 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 20:44:24.604291 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:44:24.604260 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 20:44:24.980969 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:44:24.980882 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 20:44:25.133216 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:44:25.133178 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 20:44:27.566787 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:44:27.566748 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 20:44:27.733686 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:44:27.733629 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 20:44:27.961708 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:44:27.961678 5378 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-57f958cfdb-rfvmm" Oct 30 20:44:29.718724 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:44:29.718686 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 20:44:30.197740 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:44:30.197704 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 20:44:30.399580 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:44:30.399511 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 20:44:30.552813 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:44:30.552728 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 20:44:33.158389 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:44:33.158333 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 20:44:33.313184 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:44:33.313153 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 20:44:34.136328 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:44:34.136278 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 20:44:35.941401 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:44:35.941373 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 20:44:36.097052 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:44:36.097014 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 20:44:36.343185 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:44:36.343094 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 20:44:37.734409 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:44:37.734370 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 20:44:38.119077 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:44:38.119036 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 20:44:38.289509 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:44:38.289472 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 20:44:40.442410 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:44:40.442375 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 20:44:40.490768 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:44:40.490732 5378 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66c4fb575d-b2sv2"] Oct 30 20:44:40.493766 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:44:40.493750 5378 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-b2sv2" Oct 30 20:44:40.564494 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:44:40.564458 5378 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/dd4ee482-64ad-45a3-bad6-2ad6e4a3693d-nginx-conf\") pod \"kuadrant-console-plugin-66c4fb575d-b2sv2\" (UID: \"dd4ee482-64ad-45a3-bad6-2ad6e4a3693d\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-b2sv2" Oct 30 20:44:40.564494 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:44:40.564493 5378 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mw2dd\" (UniqueName: \"kubernetes.io/projected/dd4ee482-64ad-45a3-bad6-2ad6e4a3693d-kube-api-access-mw2dd\") pod \"kuadrant-console-plugin-66c4fb575d-b2sv2\" (UID: \"dd4ee482-64ad-45a3-bad6-2ad6e4a3693d\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-b2sv2" Oct 30 20:44:40.564767 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:44:40.564616 5378 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/dd4ee482-64ad-45a3-bad6-2ad6e4a3693d-plugin-serving-cert\") pod \"kuadrant-console-plugin-66c4fb575d-b2sv2\" (UID: \"dd4ee482-64ad-45a3-bad6-2ad6e4a3693d\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-b2sv2" Oct 30 20:44:40.608191 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:44:40.608159 5378 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kuadrant-console-nginx-conf\"" Oct 30 20:44:40.608191 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:44:40.608159 5378 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\"" Oct 30 20:44:40.608403 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:44:40.608166 5378 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\"" Oct 30 20:44:40.609748 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:44:40.609729 5378 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-pfc5q\"" Oct 30 20:44:40.609886 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:44:40.609862 5378 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"plugin-serving-cert\"" Oct 30 20:44:40.610543 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:44:40.610525 5378 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66c4fb575d-b2sv2"] Oct 30 20:44:40.665391 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:44:40.665353 5378 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mw2dd\" (UniqueName: \"kubernetes.io/projected/dd4ee482-64ad-45a3-bad6-2ad6e4a3693d-kube-api-access-mw2dd\") pod \"kuadrant-console-plugin-66c4fb575d-b2sv2\" (UID: \"dd4ee482-64ad-45a3-bad6-2ad6e4a3693d\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-b2sv2" Oct 30 20:44:40.665623 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:44:40.665417 5378 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/dd4ee482-64ad-45a3-bad6-2ad6e4a3693d-plugin-serving-cert\") pod \"kuadrant-console-plugin-66c4fb575d-b2sv2\" (UID: \"dd4ee482-64ad-45a3-bad6-2ad6e4a3693d\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-b2sv2" Oct 30 20:44:40.665623 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:44:40.665456 5378 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/dd4ee482-64ad-45a3-bad6-2ad6e4a3693d-nginx-conf\") pod \"kuadrant-console-plugin-66c4fb575d-b2sv2\" (UID: \"dd4ee482-64ad-45a3-bad6-2ad6e4a3693d\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-b2sv2" Oct 30 20:44:40.666066 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:44:40.666043 5378 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/dd4ee482-64ad-45a3-bad6-2ad6e4a3693d-nginx-conf\") pod \"kuadrant-console-plugin-66c4fb575d-b2sv2\" (UID: \"dd4ee482-64ad-45a3-bad6-2ad6e4a3693d\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-b2sv2" Oct 30 20:44:40.667918 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:44:40.667895 5378 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/dd4ee482-64ad-45a3-bad6-2ad6e4a3693d-plugin-serving-cert\") pod \"kuadrant-console-plugin-66c4fb575d-b2sv2\" (UID: \"dd4ee482-64ad-45a3-bad6-2ad6e4a3693d\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-b2sv2" Oct 30 20:44:40.694592 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:44:40.694505 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 20:44:40.729851 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:44:40.729816 5378 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mw2dd\" (UniqueName: \"kubernetes.io/projected/dd4ee482-64ad-45a3-bad6-2ad6e4a3693d-kube-api-access-mw2dd\") pod \"kuadrant-console-plugin-66c4fb575d-b2sv2\" (UID: \"dd4ee482-64ad-45a3-bad6-2ad6e4a3693d\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-b2sv2" Oct 30 20:44:40.813718 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:44:40.813678 5378 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-b2sv2" Oct 30 20:44:41.181631 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:44:41.181601 5378 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66c4fb575d-b2sv2"] Oct 30 20:44:41.184705 ip-10-0-138-149 kubenswrapper[5378]: W1030 20:44:41.184676 5378 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poddd4ee482_64ad_45a3_bad6_2ad6e4a3693d.slice/crio-6478cc8870a57811ecd11fad979753e1b3f1984c40418c642876164c40a71113 WatchSource:0}: Error finding container 6478cc8870a57811ecd11fad979753e1b3f1984c40418c642876164c40a71113: Status 404 returned error can't find the container with id 6478cc8870a57811ecd11fad979753e1b3f1984c40418c642876164c40a71113 Oct 30 20:44:42.040379 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:44:42.040343 5378 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-b2sv2" event={"ID":"dd4ee482-64ad-45a3-bad6-2ad6e4a3693d","Type":"ContainerStarted","Data":"6478cc8870a57811ecd11fad979753e1b3f1984c40418c642876164c40a71113"} Oct 30 20:44:42.789044 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:44:42.789013 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 20:44:42.924898 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:44:42.924867 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 20:44:43.057553 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:44:43.057473 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 20:44:43.700888 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:44:43.700848 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 20:44:43.831186 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:44:43.831047 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 20:44:44.507503 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:44:44.507468 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 20:44:44.651176 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:44:44.651145 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 20:44:45.402125 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:44:45.402095 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 20:44:45.530747 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:44:45.530714 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 20:44:45.901045 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:44:45.901012 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 20:44:47.545953 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:44:47.545899 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 20:44:47.631536 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:44:47.631505 5378 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/kube-auth-proxy-7988bdb8dc-9bf7w"] Oct 30 20:44:47.635132 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:44:47.635111 5378 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-7988bdb8dc-9bf7w" Oct 30 20:44:47.721630 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:44:47.721584 5378 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p64bf\" (UniqueName: \"kubernetes.io/projected/31b20f62-29dc-4320-b844-92916489fb18-kube-api-access-p64bf\") pod \"kube-auth-proxy-7988bdb8dc-9bf7w\" (UID: \"31b20f62-29dc-4320-b844-92916489fb18\") " pod="openshift-ingress/kube-auth-proxy-7988bdb8dc-9bf7w" Oct 30 20:44:47.721790 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:44:47.721649 5378 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/31b20f62-29dc-4320-b844-92916489fb18-tls-certs\") pod \"kube-auth-proxy-7988bdb8dc-9bf7w\" (UID: \"31b20f62-29dc-4320-b844-92916489fb18\") " pod="openshift-ingress/kube-auth-proxy-7988bdb8dc-9bf7w" Oct 30 20:44:47.721853 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:44:47.721780 5378 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/31b20f62-29dc-4320-b844-92916489fb18-tmp\") pod \"kube-auth-proxy-7988bdb8dc-9bf7w\" (UID: \"31b20f62-29dc-4320-b844-92916489fb18\") " pod="openshift-ingress/kube-auth-proxy-7988bdb8dc-9bf7w" Oct 30 20:44:47.741207 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:44:47.741181 5378 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\"" Oct 30 20:44:47.741370 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:44:47.741337 5378 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\"" Oct 30 20:44:47.741436 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:44:47.741419 5378 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-dockercfg-zk9dx\"" Oct 30 20:44:47.742403 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:44:47.742379 5378 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-7988bdb8dc-9bf7w"] Oct 30 20:44:47.772649 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:44:47.772614 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 20:44:47.822988 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:44:47.822905 5378 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/31b20f62-29dc-4320-b844-92916489fb18-tls-certs\") pod \"kube-auth-proxy-7988bdb8dc-9bf7w\" (UID: \"31b20f62-29dc-4320-b844-92916489fb18\") " pod="openshift-ingress/kube-auth-proxy-7988bdb8dc-9bf7w" Oct 30 20:44:47.823133 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:44:47.822996 5378 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/31b20f62-29dc-4320-b844-92916489fb18-tmp\") pod \"kube-auth-proxy-7988bdb8dc-9bf7w\" (UID: \"31b20f62-29dc-4320-b844-92916489fb18\") " pod="openshift-ingress/kube-auth-proxy-7988bdb8dc-9bf7w" Oct 30 20:44:47.823181 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:44:47.823137 5378 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p64bf\" (UniqueName: \"kubernetes.io/projected/31b20f62-29dc-4320-b844-92916489fb18-kube-api-access-p64bf\") pod \"kube-auth-proxy-7988bdb8dc-9bf7w\" (UID: \"31b20f62-29dc-4320-b844-92916489fb18\") " pod="openshift-ingress/kube-auth-proxy-7988bdb8dc-9bf7w" Oct 30 20:44:47.826021 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:44:47.825993 5378 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/31b20f62-29dc-4320-b844-92916489fb18-tmp\") pod \"kube-auth-proxy-7988bdb8dc-9bf7w\" (UID: \"31b20f62-29dc-4320-b844-92916489fb18\") " pod="openshift-ingress/kube-auth-proxy-7988bdb8dc-9bf7w" Oct 30 20:44:47.826267 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:44:47.826246 5378 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/31b20f62-29dc-4320-b844-92916489fb18-tls-certs\") pod \"kube-auth-proxy-7988bdb8dc-9bf7w\" (UID: \"31b20f62-29dc-4320-b844-92916489fb18\") " pod="openshift-ingress/kube-auth-proxy-7988bdb8dc-9bf7w" Oct 30 20:44:47.881281 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:44:47.881253 5378 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p64bf\" (UniqueName: \"kubernetes.io/projected/31b20f62-29dc-4320-b844-92916489fb18-kube-api-access-p64bf\") pod \"kube-auth-proxy-7988bdb8dc-9bf7w\" (UID: \"31b20f62-29dc-4320-b844-92916489fb18\") " pod="openshift-ingress/kube-auth-proxy-7988bdb8dc-9bf7w" Oct 30 20:44:47.914441 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:44:47.914417 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 20:44:47.948895 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:44:47.948866 5378 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-7988bdb8dc-9bf7w" Oct 30 20:44:48.228389 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:44:48.228352 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 20:44:48.325534 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:44:48.325498 5378 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-7988bdb8dc-9bf7w"] Oct 30 20:44:48.329368 ip-10-0-138-149 kubenswrapper[5378]: W1030 20:44:48.329344 5378 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod31b20f62_29dc_4320_b844_92916489fb18.slice/crio-4015f7dfd69cb9b4b2a8428a2853e754062f0a741d80eac84d1fa7ca9b06cf37 WatchSource:0}: Error finding container 4015f7dfd69cb9b4b2a8428a2853e754062f0a741d80eac84d1fa7ca9b06cf37: Status 404 returned error can't find the container with id 4015f7dfd69cb9b4b2a8428a2853e754062f0a741d80eac84d1fa7ca9b06cf37 Oct 30 20:44:48.356056 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:44:48.356012 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 20:44:48.491474 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:44:48.491449 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 20:44:48.672669 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:44:48.672633 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 20:44:48.836247 ip-10-0-138-149 kubenswrapper[5378]: E1030 20:44:48.836173 5378 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q6mx7" containerName="extractor" Oct 30 20:44:49.068547 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:44:49.068516 5378 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-b2sv2" event={"ID":"dd4ee482-64ad-45a3-bad6-2ad6e4a3693d","Type":"ContainerStarted","Data":"1d422bd0db29e533183b05ea52c495cd5c0046f9f9fee08b12712dc9419d38e6"} Oct 30 20:44:49.069896 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:44:49.069870 5378 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-7988bdb8dc-9bf7w" event={"ID":"31b20f62-29dc-4320-b844-92916489fb18","Type":"ContainerStarted","Data":"4015f7dfd69cb9b4b2a8428a2853e754062f0a741d80eac84d1fa7ca9b06cf37"} Oct 30 20:44:49.193310 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:44:49.193262 5378 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-b2sv2" podStartSLOduration=1.925442006 podStartE2EDuration="9.193248058s" totalImagesPullingTime="7.267806052s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:44:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:44:49.192290257 +0000 UTC m=+379.620404176" watchObservedRunningTime="2026-10-30 20:44:49.193248058 +0000 UTC m=+379.621361977" Oct 30 20:44:50.066665 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:44:50.066629 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 20:44:50.420847 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:44:50.420810 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 20:44:50.594108 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:44:50.594074 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 20:44:51.625601 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:44:51.625555 5378 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-57f7f8659f-zz5wh"] Oct 30 20:44:51.628624 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:44:51.628606 5378 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-57f7f8659f-zz5wh" Oct 30 20:44:51.658348 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:44:51.658319 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 20:44:51.687083 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:44:51.687052 5378 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-57f7f8659f-zz5wh"] Oct 30 20:44:51.759045 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:44:51.759011 5378 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9cglr\" (UniqueName: \"kubernetes.io/projected/9b8afd50-8989-4785-ada2-e31c4db91421-kube-api-access-9cglr\") pod \"authorino-57f7f8659f-zz5wh\" (UID: \"9b8afd50-8989-4785-ada2-e31c4db91421\") " pod="kuadrant-system/authorino-57f7f8659f-zz5wh" Oct 30 20:44:51.787875 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:44:51.787852 5378 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-qrtn9\"" Oct 30 20:44:51.797522 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:44:51.797499 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 20:44:51.860369 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:44:51.860337 5378 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9cglr\" (UniqueName: \"kubernetes.io/projected/9b8afd50-8989-4785-ada2-e31c4db91421-kube-api-access-9cglr\") pod \"authorino-57f7f8659f-zz5wh\" (UID: \"9b8afd50-8989-4785-ada2-e31c4db91421\") " pod="kuadrant-system/authorino-57f7f8659f-zz5wh" Oct 30 20:44:51.919012 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:44:51.918952 5378 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9cglr\" (UniqueName: \"kubernetes.io/projected/9b8afd50-8989-4785-ada2-e31c4db91421-kube-api-access-9cglr\") pod \"authorino-57f7f8659f-zz5wh\" (UID: \"9b8afd50-8989-4785-ada2-e31c4db91421\") " pod="kuadrant-system/authorino-57f7f8659f-zz5wh" Oct 30 20:44:51.934212 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:44:51.934187 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 20:44:51.940073 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:44:51.940054 5378 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-57f7f8659f-zz5wh" Oct 30 20:44:52.082696 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:44:52.082665 5378 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-7988bdb8dc-9bf7w" event={"ID":"31b20f62-29dc-4320-b844-92916489fb18","Type":"ContainerStarted","Data":"d79d6f2db13a1711e6bb4ce1ffcc5860abae5a60e55abc707c84fe71182b7c1e"} Oct 30 20:44:52.199366 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:44:52.199281 5378 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/kube-auth-proxy-7988bdb8dc-9bf7w" podStartSLOduration=2.306807966 podStartE2EDuration="5.199266772s" totalImagesPullingTime="2.892458806s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:44:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:44:52.19869659 +0000 UTC m=+382.626810509" watchObservedRunningTime="2026-10-30 20:44:52.199266772 +0000 UTC m=+382.627380690" Oct 30 20:44:52.315291 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:44:52.315260 5378 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-57f7f8659f-zz5wh"] Oct 30 20:44:52.318016 ip-10-0-138-149 kubenswrapper[5378]: W1030 20:44:52.317986 5378 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod9b8afd50_8989_4785_ada2_e31c4db91421.slice/crio-fc3dc8d53ed8c022940e189239b6754c038e3e3dae278aa74c72e74a4d33807a WatchSource:0}: Error finding container fc3dc8d53ed8c022940e189239b6754c038e3e3dae278aa74c72e74a4d33807a: Status 404 returned error can't find the container with id fc3dc8d53ed8c022940e189239b6754c038e3e3dae278aa74c72e74a4d33807a Oct 30 20:44:53.086886 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:44:53.086846 5378 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-57f7f8659f-zz5wh" event={"ID":"9b8afd50-8989-4785-ada2-e31c4db91421","Type":"ContainerStarted","Data":"fc3dc8d53ed8c022940e189239b6754c038e3e3dae278aa74c72e74a4d33807a"} Oct 30 20:44:54.518443 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:44:54.518408 5378 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-r9g59"] Oct 30 20:44:54.522139 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:44:54.522115 5378 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-7c9b87d574-r9g59" Oct 30 20:44:54.577715 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:44:54.577688 5378 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"limitador-limits-config-limitador\"" Oct 30 20:44:54.584079 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:44:54.584037 5378 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sxkzc\" (UniqueName: \"kubernetes.io/projected/c980ce98-3651-4f7c-bc05-467c9e14f9c2-kube-api-access-sxkzc\") pod \"limitador-limitador-7c9b87d574-r9g59\" (UID: \"c980ce98-3651-4f7c-bc05-467c9e14f9c2\") " pod="kuadrant-system/limitador-limitador-7c9b87d574-r9g59" Oct 30 20:44:54.584200 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:44:54.584157 5378 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/c980ce98-3651-4f7c-bc05-467c9e14f9c2-config-file\") pod \"limitador-limitador-7c9b87d574-r9g59\" (UID: \"c980ce98-3651-4f7c-bc05-467c9e14f9c2\") " pod="kuadrant-system/limitador-limitador-7c9b87d574-r9g59" Oct 30 20:44:54.627984 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:44:54.627949 5378 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-r9g59"] Oct 30 20:44:54.685629 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:44:54.684719 5378 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sxkzc\" (UniqueName: \"kubernetes.io/projected/c980ce98-3651-4f7c-bc05-467c9e14f9c2-kube-api-access-sxkzc\") pod \"limitador-limitador-7c9b87d574-r9g59\" (UID: \"c980ce98-3651-4f7c-bc05-467c9e14f9c2\") " pod="kuadrant-system/limitador-limitador-7c9b87d574-r9g59" Oct 30 20:44:54.685629 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:44:54.684796 5378 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/c980ce98-3651-4f7c-bc05-467c9e14f9c2-config-file\") pod \"limitador-limitador-7c9b87d574-r9g59\" (UID: \"c980ce98-3651-4f7c-bc05-467c9e14f9c2\") " pod="kuadrant-system/limitador-limitador-7c9b87d574-r9g59" Oct 30 20:44:54.685629 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:44:54.685578 5378 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/c980ce98-3651-4f7c-bc05-467c9e14f9c2-config-file\") pod \"limitador-limitador-7c9b87d574-r9g59\" (UID: \"c980ce98-3651-4f7c-bc05-467c9e14f9c2\") " pod="kuadrant-system/limitador-limitador-7c9b87d574-r9g59" Oct 30 20:44:54.750493 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:44:54.750458 5378 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sxkzc\" (UniqueName: \"kubernetes.io/projected/c980ce98-3651-4f7c-bc05-467c9e14f9c2-kube-api-access-sxkzc\") pod \"limitador-limitador-7c9b87d574-r9g59\" (UID: \"c980ce98-3651-4f7c-bc05-467c9e14f9c2\") " pod="kuadrant-system/limitador-limitador-7c9b87d574-r9g59" Oct 30 20:44:54.768301 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:44:54.768274 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 20:44:54.837286 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:44:54.837201 5378 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-7c9b87d574-r9g59" Oct 30 20:44:54.846159 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:44:54.846133 5378 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-r9g59"] Oct 30 20:44:55.101349 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:44:55.101324 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 20:44:55.235096 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:44:55.235067 5378 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-r9g59"] Oct 30 20:44:55.237484 ip-10-0-138-149 kubenswrapper[5378]: W1030 20:44:55.237444 5378 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc980ce98_3651_4f7c_bc05_467c9e14f9c2.slice/crio-9ed1103bfad65e71d4ba24cb1e7573c2384eedefaa844bc9b886cf70cfb1d962 WatchSource:0}: Error finding container 9ed1103bfad65e71d4ba24cb1e7573c2384eedefaa844bc9b886cf70cfb1d962: Status 404 returned error can't find the container with id 9ed1103bfad65e71d4ba24cb1e7573c2384eedefaa844bc9b886cf70cfb1d962 Oct 30 20:44:55.488802 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:44:55.488703 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 20:44:56.099789 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:44:56.099749 5378 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-7c9b87d574-r9g59" event={"ID":"c980ce98-3651-4f7c-bc05-467c9e14f9c2","Type":"ContainerStarted","Data":"9ed1103bfad65e71d4ba24cb1e7573c2384eedefaa844bc9b886cf70cfb1d962"} Oct 30 20:44:57.105976 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:44:57.105941 5378 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-57f7f8659f-zz5wh" event={"ID":"9b8afd50-8989-4785-ada2-e31c4db91421","Type":"ContainerStarted","Data":"054bbabb16ff949acd99f3cda11d4bbcb64df1b0b8f89a90827ef63bc40f41a0"} Oct 30 20:44:57.223300 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:44:57.223245 5378 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/authorino-57f7f8659f-zz5wh" podStartSLOduration=2.217457819 podStartE2EDuration="6.223230913s" totalImagesPullingTime="4.005773094s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:44:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:44:57.222677155 +0000 UTC m=+387.650791075" watchObservedRunningTime="2026-10-30 20:44:57.223230913 +0000 UTC m=+387.651344832" Oct 30 20:44:57.250578 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:44:57.250534 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 20:44:57.383406 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:44:57.383297 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 20:44:58.111094 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:44:58.111059 5378 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-7c9b87d574-r9g59" event={"ID":"c980ce98-3651-4f7c-bc05-467c9e14f9c2","Type":"ContainerStarted","Data":"69eeb7f911732010943947ced76659c085b276545645551ea3a402034ed485ad"} Oct 30 20:44:58.111488 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:44:58.111340 5378 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-7c9b87d574-r9g59" Oct 30 20:44:58.226035 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:44:58.225986 5378 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-7c9b87d574-r9g59" podStartSLOduration=2.219828521 podStartE2EDuration="4.225972628s" totalImagesPullingTime="2.006144107s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:44:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:44:58.225587533 +0000 UTC m=+388.653701452" watchObservedRunningTime="2026-10-30 20:44:58.225972628 +0000 UTC m=+388.654086550" Oct 30 20:44:59.367289 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:44:59.367249 5378 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-57f7f8659f-zz5wh"] Oct 30 20:44:59.367782 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:44:59.367444 5378 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/authorino-57f7f8659f-zz5wh" podUID="9b8afd50-8989-4785-ada2-e31c4db91421" containerName="authorino" containerID="cri-o://054bbabb16ff949acd99f3cda11d4bbcb64df1b0b8f89a90827ef63bc40f41a0" gracePeriod=30 Oct 30 20:44:59.607697 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:44:59.607672 5378 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-57f7f8659f-zz5wh" Oct 30 20:44:59.722185 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:44:59.722120 5378 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9b8afd50-8989-4785-ada2-e31c4db91421-kube-api-access-9cglr\" (UniqueName: \"kubernetes.io/projected/9b8afd50-8989-4785-ada2-e31c4db91421-kube-api-access-9cglr\") pod \"9b8afd50-8989-4785-ada2-e31c4db91421\" (UID: \"9b8afd50-8989-4785-ada2-e31c4db91421\") " Oct 30 20:44:59.724189 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:44:59.724166 5378 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9b8afd50-8989-4785-ada2-e31c4db91421-kube-api-access-9cglr" pod "9b8afd50-8989-4785-ada2-e31c4db91421" (UID: "9b8afd50-8989-4785-ada2-e31c4db91421"). InnerVolumeSpecName "kube-api-access-9cglr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:44:59.823516 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:44:59.823485 5378 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9cglr\" (UniqueName: \"kubernetes.io/projected/9b8afd50-8989-4785-ada2-e31c4db91421-kube-api-access-9cglr\") on node \"ip-10-0-138-149.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:45:00.119283 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:45:00.119249 5378 generic.go:356] "Generic (PLEG): container finished" podID="9b8afd50-8989-4785-ada2-e31c4db91421" containerID="054bbabb16ff949acd99f3cda11d4bbcb64df1b0b8f89a90827ef63bc40f41a0" exitCode=0 Oct 30 20:45:00.119449 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:45:00.119296 5378 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-57f7f8659f-zz5wh" Oct 30 20:45:00.119449 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:45:00.119316 5378 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-57f7f8659f-zz5wh" event={"ID":"9b8afd50-8989-4785-ada2-e31c4db91421","Type":"ContainerDied","Data":"054bbabb16ff949acd99f3cda11d4bbcb64df1b0b8f89a90827ef63bc40f41a0"} Oct 30 20:45:00.119449 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:45:00.119356 5378 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-57f7f8659f-zz5wh" event={"ID":"9b8afd50-8989-4785-ada2-e31c4db91421","Type":"ContainerDied","Data":"fc3dc8d53ed8c022940e189239b6754c038e3e3dae278aa74c72e74a4d33807a"} Oct 30 20:45:00.119449 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:45:00.119379 5378 scope.go:122] "RemoveContainer" containerID="054bbabb16ff949acd99f3cda11d4bbcb64df1b0b8f89a90827ef63bc40f41a0" Oct 30 20:45:00.127174 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:45:00.127158 5378 scope.go:122] "RemoveContainer" containerID="054bbabb16ff949acd99f3cda11d4bbcb64df1b0b8f89a90827ef63bc40f41a0" Oct 30 20:45:00.127441 ip-10-0-138-149 kubenswrapper[5378]: E1030 20:45:00.127425 5378 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"054bbabb16ff949acd99f3cda11d4bbcb64df1b0b8f89a90827ef63bc40f41a0\": container with ID starting with 054bbabb16ff949acd99f3cda11d4bbcb64df1b0b8f89a90827ef63bc40f41a0 not found: ID does not exist" containerID="054bbabb16ff949acd99f3cda11d4bbcb64df1b0b8f89a90827ef63bc40f41a0" Oct 30 20:45:00.127477 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:45:00.127450 5378 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"054bbabb16ff949acd99f3cda11d4bbcb64df1b0b8f89a90827ef63bc40f41a0"} err="failed to get container status \"054bbabb16ff949acd99f3cda11d4bbcb64df1b0b8f89a90827ef63bc40f41a0\": rpc error: code = NotFound desc = could not find container \"054bbabb16ff949acd99f3cda11d4bbcb64df1b0b8f89a90827ef63bc40f41a0\": container with ID starting with 054bbabb16ff949acd99f3cda11d4bbcb64df1b0b8f89a90827ef63bc40f41a0 not found: ID does not exist" Oct 30 20:45:00.282866 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:45:00.282833 5378 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-57f7f8659f-zz5wh"] Oct 30 20:45:00.289082 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:45:00.289058 5378 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-57f7f8659f-zz5wh"] Oct 30 20:45:01.738808 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:45:01.738767 5378 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9b8afd50-8989-4785-ada2-e31c4db91421" path="/var/lib/kubelet/pods/9b8afd50-8989-4785-ada2-e31c4db91421/volumes" Oct 30 20:45:02.649175 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:45:02.649145 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 20:45:02.771887 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:45:02.771852 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 20:45:03.042129 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:45:03.042051 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 20:45:03.159792 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:45:03.159759 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 20:45:03.441204 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:45:03.441169 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 20:45:03.581685 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:45:03.581650 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 20:45:09.116280 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:45:09.116249 5378 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-7c9b87d574-r9g59" Oct 30 20:45:11.445963 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:45:11.445930 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 20:45:11.580187 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:45:11.580155 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 20:45:14.461234 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:45:14.461200 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 20:45:14.594737 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:45:14.594704 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 20:45:17.166758 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:45:17.166725 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 20:45:17.300476 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:45:17.300444 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 20:45:18.121672 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:45:18.121642 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 20:45:18.270295 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:45:18.270264 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 20:45:18.836233 ip-10-0-138-149 kubenswrapper[5378]: E1030 20:45:18.836187 5378 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q6mx7" containerName="extractor" Oct 30 20:45:18.947130 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:45:18.947099 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 20:45:24.795018 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:45:24.794984 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 20:45:24.931508 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:45:24.931477 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 20:45:38.090195 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:45:38.090158 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 20:45:40.256157 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:45:40.256110 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 20:45:40.334891 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:45:40.334860 5378 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-579cf7c6d-4fqbt"] Oct 30 20:45:40.335196 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:45:40.335183 5378 cpu_manager.go:414] "containerMap: removing container" podUID="9b8afd50-8989-4785-ada2-e31c4db91421" containerName="authorino" Oct 30 20:45:40.335237 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:45:40.335198 5378 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9b8afd50-8989-4785-ada2-e31c4db91421" containerName="authorino" Oct 30 20:45:40.335270 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:45:40.335242 5378 memory_manager.go:365] "RemoveStaleState removing state" podUID="9b8afd50-8989-4785-ada2-e31c4db91421" containerName="authorino" Oct 30 20:45:40.337810 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:45:40.337783 5378 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-579cf7c6d-4fqbt" Oct 30 20:45:40.390796 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:45:40.390762 5378 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-metrics-tls\"" Oct 30 20:45:40.393663 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:45:40.393641 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 20:45:40.437842 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:45:40.437817 5378 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3bca99b1-5b6a-4a68-b78e-b27c4b2dfece-metrics-certs\") pod \"maas-controller-579cf7c6d-4fqbt\" (UID: \"3bca99b1-5b6a-4a68-b78e-b27c4b2dfece\") " pod="opendatahub/maas-controller-579cf7c6d-4fqbt" Oct 30 20:45:40.437955 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:45:40.437853 5378 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-55gfn\" (UniqueName: \"kubernetes.io/projected/3bca99b1-5b6a-4a68-b78e-b27c4b2dfece-kube-api-access-55gfn\") pod \"maas-controller-579cf7c6d-4fqbt\" (UID: \"3bca99b1-5b6a-4a68-b78e-b27c4b2dfece\") " pod="opendatahub/maas-controller-579cf7c6d-4fqbt" Oct 30 20:45:40.437955 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:45:40.437872 5378 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/3bca99b1-5b6a-4a68-b78e-b27c4b2dfece-webhook-cert\") pod \"maas-controller-579cf7c6d-4fqbt\" (UID: \"3bca99b1-5b6a-4a68-b78e-b27c4b2dfece\") " pod="opendatahub/maas-controller-579cf7c6d-4fqbt" Oct 30 20:45:40.441318 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:45:40.441298 5378 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\"" Oct 30 20:45:40.441411 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:45:40.441330 5378 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-kpvgd\"" Oct 30 20:45:40.441411 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:45:40.441344 5378 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\"" Oct 30 20:45:40.441411 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:45:40.441371 5378 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\"" Oct 30 20:45:40.441411 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:45:40.441406 5378 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\"" Oct 30 20:45:40.442320 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:45:40.442302 5378 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-579cf7c6d-4fqbt"] Oct 30 20:45:40.532671 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:45:40.532601 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 20:45:40.538719 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:45:40.538700 5378 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3bca99b1-5b6a-4a68-b78e-b27c4b2dfece-metrics-certs\") pod \"maas-controller-579cf7c6d-4fqbt\" (UID: \"3bca99b1-5b6a-4a68-b78e-b27c4b2dfece\") " pod="opendatahub/maas-controller-579cf7c6d-4fqbt" Oct 30 20:45:40.538791 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:45:40.538737 5378 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-55gfn\" (UniqueName: \"kubernetes.io/projected/3bca99b1-5b6a-4a68-b78e-b27c4b2dfece-kube-api-access-55gfn\") pod \"maas-controller-579cf7c6d-4fqbt\" (UID: \"3bca99b1-5b6a-4a68-b78e-b27c4b2dfece\") " pod="opendatahub/maas-controller-579cf7c6d-4fqbt" Oct 30 20:45:40.538791 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:45:40.538757 5378 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/3bca99b1-5b6a-4a68-b78e-b27c4b2dfece-webhook-cert\") pod \"maas-controller-579cf7c6d-4fqbt\" (UID: \"3bca99b1-5b6a-4a68-b78e-b27c4b2dfece\") " pod="opendatahub/maas-controller-579cf7c6d-4fqbt" Oct 30 20:45:40.538901 ip-10-0-138-149 kubenswrapper[5378]: E1030 20:45:40.538890 5378 secret.go:189] Couldn't get secret opendatahub/maas-controller-webhook-cert: secret "maas-controller-webhook-cert" not found Oct 30 20:45:40.538979 ip-10-0-138-149 kubenswrapper[5378]: E1030 20:45:40.538967 5378 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3bca99b1-5b6a-4a68-b78e-b27c4b2dfece-webhook-cert podName:3bca99b1-5b6a-4a68-b78e-b27c4b2dfece nodeName:}" failed. No retries permitted until 2026-10-30 20:45:41.038947545 +0000 UTC m=+431.467061457 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "webhook-cert" (UniqueName: "kubernetes.io/secret/3bca99b1-5b6a-4a68-b78e-b27c4b2dfece-webhook-cert") pod "maas-controller-579cf7c6d-4fqbt" (UID: "3bca99b1-5b6a-4a68-b78e-b27c4b2dfece") : secret "maas-controller-webhook-cert" not found Oct 30 20:45:40.541174 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:45:40.541150 5378 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3bca99b1-5b6a-4a68-b78e-b27c4b2dfece-metrics-certs\") pod \"maas-controller-579cf7c6d-4fqbt\" (UID: \"3bca99b1-5b6a-4a68-b78e-b27c4b2dfece\") " pod="opendatahub/maas-controller-579cf7c6d-4fqbt" Oct 30 20:45:40.596462 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:45:40.596434 5378 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-55gfn\" (UniqueName: \"kubernetes.io/projected/3bca99b1-5b6a-4a68-b78e-b27c4b2dfece-kube-api-access-55gfn\") pod \"maas-controller-579cf7c6d-4fqbt\" (UID: \"3bca99b1-5b6a-4a68-b78e-b27c4b2dfece\") " pod="opendatahub/maas-controller-579cf7c6d-4fqbt" Oct 30 20:45:40.650908 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:45:40.650880 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 20:45:40.762989 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:45:40.762961 5378 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-6cb6cd7964-gj5sw"] Oct 30 20:45:40.766092 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:45:40.766076 5378 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-6cb6cd7964-gj5sw" Oct 30 20:45:40.771512 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:45:40.771489 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 20:45:40.827596 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:45:40.827523 5378 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-6cb6cd7964-gj5sw"] Oct 30 20:45:40.840859 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:45:40.840839 5378 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/62138235-e57b-42e6-a892-67123e9b2478-webhook-cert\") pod \"maas-controller-6cb6cd7964-gj5sw\" (UID: \"62138235-e57b-42e6-a892-67123e9b2478\") " pod="opendatahub/maas-controller-6cb6cd7964-gj5sw" Oct 30 20:45:40.840967 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:45:40.840864 5378 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/62138235-e57b-42e6-a892-67123e9b2478-metrics-certs\") pod \"maas-controller-6cb6cd7964-gj5sw\" (UID: \"62138235-e57b-42e6-a892-67123e9b2478\") " pod="opendatahub/maas-controller-6cb6cd7964-gj5sw" Oct 30 20:45:40.840967 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:45:40.840883 5378 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2g2kz\" (UniqueName: \"kubernetes.io/projected/62138235-e57b-42e6-a892-67123e9b2478-kube-api-access-2g2kz\") pod \"maas-controller-6cb6cd7964-gj5sw\" (UID: \"62138235-e57b-42e6-a892-67123e9b2478\") " pod="opendatahub/maas-controller-6cb6cd7964-gj5sw" Oct 30 20:45:40.941534 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:45:40.941510 5378 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/62138235-e57b-42e6-a892-67123e9b2478-webhook-cert\") pod \"maas-controller-6cb6cd7964-gj5sw\" (UID: \"62138235-e57b-42e6-a892-67123e9b2478\") " pod="opendatahub/maas-controller-6cb6cd7964-gj5sw" Oct 30 20:45:40.941658 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:45:40.941538 5378 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/62138235-e57b-42e6-a892-67123e9b2478-metrics-certs\") pod \"maas-controller-6cb6cd7964-gj5sw\" (UID: \"62138235-e57b-42e6-a892-67123e9b2478\") " pod="opendatahub/maas-controller-6cb6cd7964-gj5sw" Oct 30 20:45:40.941658 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:45:40.941554 5378 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2g2kz\" (UniqueName: \"kubernetes.io/projected/62138235-e57b-42e6-a892-67123e9b2478-kube-api-access-2g2kz\") pod \"maas-controller-6cb6cd7964-gj5sw\" (UID: \"62138235-e57b-42e6-a892-67123e9b2478\") " pod="opendatahub/maas-controller-6cb6cd7964-gj5sw" Oct 30 20:45:40.944127 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:45:40.944106 5378 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/62138235-e57b-42e6-a892-67123e9b2478-webhook-cert\") pod \"maas-controller-6cb6cd7964-gj5sw\" (UID: \"62138235-e57b-42e6-a892-67123e9b2478\") " pod="opendatahub/maas-controller-6cb6cd7964-gj5sw" Oct 30 20:45:40.944225 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:45:40.944138 5378 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/62138235-e57b-42e6-a892-67123e9b2478-metrics-certs\") pod \"maas-controller-6cb6cd7964-gj5sw\" (UID: \"62138235-e57b-42e6-a892-67123e9b2478\") " pod="opendatahub/maas-controller-6cb6cd7964-gj5sw" Oct 30 20:45:40.999533 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:45:40.999502 5378 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2g2kz\" (UniqueName: \"kubernetes.io/projected/62138235-e57b-42e6-a892-67123e9b2478-kube-api-access-2g2kz\") pod \"maas-controller-6cb6cd7964-gj5sw\" (UID: \"62138235-e57b-42e6-a892-67123e9b2478\") " pod="opendatahub/maas-controller-6cb6cd7964-gj5sw" Oct 30 20:45:41.042683 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:45:41.042657 5378 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/3bca99b1-5b6a-4a68-b78e-b27c4b2dfece-webhook-cert\") pod \"maas-controller-579cf7c6d-4fqbt\" (UID: \"3bca99b1-5b6a-4a68-b78e-b27c4b2dfece\") " pod="opendatahub/maas-controller-579cf7c6d-4fqbt" Oct 30 20:45:41.044658 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:45:41.044641 5378 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/3bca99b1-5b6a-4a68-b78e-b27c4b2dfece-webhook-cert\") pod \"maas-controller-579cf7c6d-4fqbt\" (UID: \"3bca99b1-5b6a-4a68-b78e-b27c4b2dfece\") " pod="opendatahub/maas-controller-579cf7c6d-4fqbt" Oct 30 20:45:41.078587 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:45:41.078520 5378 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-6cb6cd7964-gj5sw" Oct 30 20:45:41.249554 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:45:41.249515 5378 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-579cf7c6d-4fqbt" Oct 30 20:45:41.446101 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:45:41.445952 5378 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-6cb6cd7964-gj5sw"] Oct 30 20:45:41.448671 ip-10-0-138-149 kubenswrapper[5378]: W1030 20:45:41.448646 5378 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod62138235_e57b_42e6_a892_67123e9b2478.slice/crio-6871633649c7b49e7f98cd3ed6e811492c31f93f811d77cec9173906ead6c3ef WatchSource:0}: Error finding container 6871633649c7b49e7f98cd3ed6e811492c31f93f811d77cec9173906ead6c3ef: Status 404 returned error can't find the container with id 6871633649c7b49e7f98cd3ed6e811492c31f93f811d77cec9173906ead6c3ef Oct 30 20:45:41.617462 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:45:41.617440 5378 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-579cf7c6d-4fqbt"] Oct 30 20:45:41.619630 ip-10-0-138-149 kubenswrapper[5378]: W1030 20:45:41.619602 5378 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3bca99b1_5b6a_4a68_b78e_b27c4b2dfece.slice/crio-66fb5f5f6e1786cdc50796b4136231de1aa60bbd4354c6c2226820f8887d9b08 WatchSource:0}: Error finding container 66fb5f5f6e1786cdc50796b4136231de1aa60bbd4354c6c2226820f8887d9b08: Status 404 returned error can't find the container with id 66fb5f5f6e1786cdc50796b4136231de1aa60bbd4354c6c2226820f8887d9b08 Oct 30 20:45:42.273408 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:45:42.273366 5378 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-6cb6cd7964-gj5sw" event={"ID":"62138235-e57b-42e6-a892-67123e9b2478","Type":"ContainerStarted","Data":"6871633649c7b49e7f98cd3ed6e811492c31f93f811d77cec9173906ead6c3ef"} Oct 30 20:45:42.274484 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:45:42.274468 5378 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-579cf7c6d-4fqbt" event={"ID":"3bca99b1-5b6a-4a68-b78e-b27c4b2dfece","Type":"ContainerStarted","Data":"66fb5f5f6e1786cdc50796b4136231de1aa60bbd4354c6c2226820f8887d9b08"} Oct 30 20:45:45.288342 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:45:45.288253 5378 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-579cf7c6d-4fqbt" event={"ID":"3bca99b1-5b6a-4a68-b78e-b27c4b2dfece","Type":"ContainerStarted","Data":"fe15b7b32b4f15cf299eb0e436fb5f70de321f15f5efedda64ef291aea1367f9"} Oct 30 20:45:45.288803 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:45:45.288363 5378 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-579cf7c6d-4fqbt" Oct 30 20:45:45.289616 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:45:45.289591 5378 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-6cb6cd7964-gj5sw" event={"ID":"62138235-e57b-42e6-a892-67123e9b2478","Type":"ContainerStarted","Data":"b1900f549a152d00785af325909d9c6af17e2c07763d7433baf6aabcbaa0e3a3"} Oct 30 20:45:45.289731 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:45:45.289647 5378 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-6cb6cd7964-gj5sw" Oct 30 20:45:45.385558 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:45:45.385523 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 20:45:45.456324 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:45:45.456276 5378 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/maas-controller-579cf7c6d-4fqbt" podStartSLOduration=1.955664109 podStartE2EDuration="5.456262545s" totalImagesPullingTime="3.500598436s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:45:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:45:45.455227038 +0000 UTC m=+435.883340968" watchObservedRunningTime="2026-10-30 20:45:45.456262545 +0000 UTC m=+435.884376519" Oct 30 20:45:45.541311 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:45:45.541236 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 20:45:45.572689 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:45:45.572645 5378 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/maas-controller-6cb6cd7964-gj5sw" podStartSLOduration=2.020008835 podStartE2EDuration="5.572631669s" totalImagesPullingTime="3.552622834s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:45:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:45:45.572013544 +0000 UTC m=+436.000127463" watchObservedRunningTime="2026-10-30 20:45:45.572631669 +0000 UTC m=+436.000745579" Oct 30 20:45:48.409497 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:45:48.409467 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 20:45:48.560316 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:45:48.560287 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 20:45:48.836029 ip-10-0-138-149 kubenswrapper[5378]: E1030 20:45:48.835954 5378 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q6mx7" containerName="extractor" Oct 30 20:45:56.294834 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:45:56.294803 5378 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-579cf7c6d-4fqbt" Oct 30 20:45:56.295250 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:45:56.294876 5378 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-6cb6cd7964-gj5sw" Oct 30 20:45:56.477035 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:45:56.476995 5378 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-579cf7c6d-4fqbt"] Oct 30 20:45:56.477284 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:45:56.477236 5378 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/maas-controller-579cf7c6d-4fqbt" podUID="3bca99b1-5b6a-4a68-b78e-b27c4b2dfece" containerName="manager" containerID="cri-o://fe15b7b32b4f15cf299eb0e436fb5f70de321f15f5efedda64ef291aea1367f9" gracePeriod=30 Oct 30 20:45:56.715340 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:45:56.715313 5378 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-579cf7c6d-4fqbt" Oct 30 20:45:56.772801 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:45:56.772770 5378 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3bca99b1-5b6a-4a68-b78e-b27c4b2dfece-webhook-cert\" (UniqueName: \"kubernetes.io/secret/3bca99b1-5b6a-4a68-b78e-b27c4b2dfece-webhook-cert\") pod \"3bca99b1-5b6a-4a68-b78e-b27c4b2dfece\" (UID: \"3bca99b1-5b6a-4a68-b78e-b27c4b2dfece\") " Oct 30 20:45:56.772939 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:45:56.772880 5378 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3bca99b1-5b6a-4a68-b78e-b27c4b2dfece-kube-api-access-55gfn\" (UniqueName: \"kubernetes.io/projected/3bca99b1-5b6a-4a68-b78e-b27c4b2dfece-kube-api-access-55gfn\") pod \"3bca99b1-5b6a-4a68-b78e-b27c4b2dfece\" (UID: \"3bca99b1-5b6a-4a68-b78e-b27c4b2dfece\") " Oct 30 20:45:56.772939 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:45:56.772914 5378 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3bca99b1-5b6a-4a68-b78e-b27c4b2dfece-metrics-certs\" (UniqueName: \"kubernetes.io/secret/3bca99b1-5b6a-4a68-b78e-b27c4b2dfece-metrics-certs\") pod \"3bca99b1-5b6a-4a68-b78e-b27c4b2dfece\" (UID: \"3bca99b1-5b6a-4a68-b78e-b27c4b2dfece\") " Oct 30 20:45:56.774934 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:45:56.774901 5378 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3bca99b1-5b6a-4a68-b78e-b27c4b2dfece-webhook-cert" pod "3bca99b1-5b6a-4a68-b78e-b27c4b2dfece" (UID: "3bca99b1-5b6a-4a68-b78e-b27c4b2dfece"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 20:45:56.775050 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:45:56.774945 5378 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3bca99b1-5b6a-4a68-b78e-b27c4b2dfece-metrics-certs" pod "3bca99b1-5b6a-4a68-b78e-b27c4b2dfece" (UID: "3bca99b1-5b6a-4a68-b78e-b27c4b2dfece"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 20:45:56.775050 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:45:56.775003 5378 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3bca99b1-5b6a-4a68-b78e-b27c4b2dfece-kube-api-access-55gfn" pod "3bca99b1-5b6a-4a68-b78e-b27c4b2dfece" (UID: "3bca99b1-5b6a-4a68-b78e-b27c4b2dfece"). InnerVolumeSpecName "kube-api-access-55gfn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:45:56.873656 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:45:56.873626 5378 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-55gfn\" (UniqueName: \"kubernetes.io/projected/3bca99b1-5b6a-4a68-b78e-b27c4b2dfece-kube-api-access-55gfn\") on node \"ip-10-0-138-149.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:45:56.873656 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:45:56.873654 5378 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3bca99b1-5b6a-4a68-b78e-b27c4b2dfece-metrics-certs\") on node \"ip-10-0-138-149.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:45:56.873834 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:45:56.873664 5378 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/3bca99b1-5b6a-4a68-b78e-b27c4b2dfece-webhook-cert\") on node \"ip-10-0-138-149.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:45:57.339330 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:45:57.339251 5378 generic.go:356] "Generic (PLEG): container finished" podID="3bca99b1-5b6a-4a68-b78e-b27c4b2dfece" containerID="fe15b7b32b4f15cf299eb0e436fb5f70de321f15f5efedda64ef291aea1367f9" exitCode=0 Oct 30 20:45:57.339330 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:45:57.339296 5378 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-579cf7c6d-4fqbt" Oct 30 20:45:57.339775 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:45:57.339335 5378 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-579cf7c6d-4fqbt" event={"ID":"3bca99b1-5b6a-4a68-b78e-b27c4b2dfece","Type":"ContainerDied","Data":"fe15b7b32b4f15cf299eb0e436fb5f70de321f15f5efedda64ef291aea1367f9"} Oct 30 20:45:57.339775 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:45:57.339387 5378 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-579cf7c6d-4fqbt" event={"ID":"3bca99b1-5b6a-4a68-b78e-b27c4b2dfece","Type":"ContainerDied","Data":"66fb5f5f6e1786cdc50796b4136231de1aa60bbd4354c6c2226820f8887d9b08"} Oct 30 20:45:57.339775 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:45:57.339408 5378 scope.go:122] "RemoveContainer" containerID="fe15b7b32b4f15cf299eb0e436fb5f70de321f15f5efedda64ef291aea1367f9" Oct 30 20:45:57.348132 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:45:57.348118 5378 scope.go:122] "RemoveContainer" containerID="fe15b7b32b4f15cf299eb0e436fb5f70de321f15f5efedda64ef291aea1367f9" Oct 30 20:45:57.348350 ip-10-0-138-149 kubenswrapper[5378]: E1030 20:45:57.348331 5378 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fe15b7b32b4f15cf299eb0e436fb5f70de321f15f5efedda64ef291aea1367f9\": container with ID starting with fe15b7b32b4f15cf299eb0e436fb5f70de321f15f5efedda64ef291aea1367f9 not found: ID does not exist" containerID="fe15b7b32b4f15cf299eb0e436fb5f70de321f15f5efedda64ef291aea1367f9" Oct 30 20:45:57.348414 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:45:57.348363 5378 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fe15b7b32b4f15cf299eb0e436fb5f70de321f15f5efedda64ef291aea1367f9"} err="failed to get container status \"fe15b7b32b4f15cf299eb0e436fb5f70de321f15f5efedda64ef291aea1367f9\": rpc error: code = NotFound desc = could not find container \"fe15b7b32b4f15cf299eb0e436fb5f70de321f15f5efedda64ef291aea1367f9\": container with ID starting with fe15b7b32b4f15cf299eb0e436fb5f70de321f15f5efedda64ef291aea1367f9 not found: ID does not exist" Oct 30 20:45:57.505321 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:45:57.505289 5378 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-579cf7c6d-4fqbt"] Oct 30 20:45:57.555858 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:45:57.555825 5378 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-579cf7c6d-4fqbt"] Oct 30 20:45:57.738033 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:45:57.738001 5378 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3bca99b1-5b6a-4a68-b78e-b27c4b2dfece" path="/var/lib/kubelet/pods/3bca99b1-5b6a-4a68-b78e-b27c4b2dfece/volumes" Oct 30 20:46:02.769107 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:46:02.769075 5378 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-5h7rm"] Oct 30 20:46:02.769518 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:46:02.769362 5378 cpu_manager.go:414] "containerMap: removing container" podUID="3bca99b1-5b6a-4a68-b78e-b27c4b2dfece" containerName="manager" Oct 30 20:46:02.769518 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:46:02.769373 5378 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3bca99b1-5b6a-4a68-b78e-b27c4b2dfece" containerName="manager" Oct 30 20:46:02.769518 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:46:02.769448 5378 memory_manager.go:365] "RemoveStaleState removing state" podUID="3bca99b1-5b6a-4a68-b78e-b27c4b2dfece" containerName="manager" Oct 30 20:46:02.773609 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:46:02.773588 5378 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-5h7rm" Oct 30 20:46:02.824109 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:46:02.824073 5378 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/7a313ddc-716a-4e94-83a5-0e2550272032-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-5h7rm\" (UID: \"7a313ddc-716a-4e94-83a5-0e2550272032\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-5h7rm" Oct 30 20:46:02.824257 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:46:02.824112 5378 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-78chx\" (UniqueName: \"kubernetes.io/projected/7a313ddc-716a-4e94-83a5-0e2550272032-kube-api-access-78chx\") pod \"istiod-openshift-gateway-7ddd9cfdc8-5h7rm\" (UID: \"7a313ddc-716a-4e94-83a5-0e2550272032\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-5h7rm" Oct 30 20:46:02.824257 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:46:02.824144 5378 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/7a313ddc-716a-4e94-83a5-0e2550272032-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-5h7rm\" (UID: \"7a313ddc-716a-4e94-83a5-0e2550272032\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-5h7rm" Oct 30 20:46:02.824327 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:46:02.824260 5378 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/7a313ddc-716a-4e94-83a5-0e2550272032-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-5h7rm\" (UID: \"7a313ddc-716a-4e94-83a5-0e2550272032\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-5h7rm" Oct 30 20:46:02.824327 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:46:02.824315 5378 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/7a313ddc-716a-4e94-83a5-0e2550272032-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-5h7rm\" (UID: \"7a313ddc-716a-4e94-83a5-0e2550272032\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-5h7rm" Oct 30 20:46:02.824388 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:46:02.824366 5378 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7a313ddc-716a-4e94-83a5-0e2550272032-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-5h7rm\" (UID: \"7a313ddc-716a-4e94-83a5-0e2550272032\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-5h7rm" Oct 30 20:46:02.824418 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:46:02.824401 5378 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/7a313ddc-716a-4e94-83a5-0e2550272032-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-5h7rm\" (UID: \"7a313ddc-716a-4e94-83a5-0e2550272032\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-5h7rm" Oct 30 20:46:02.876494 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:46:02.876465 5378 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-5h7rm"] Oct 30 20:46:02.925354 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:46:02.925318 5378 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/7a313ddc-716a-4e94-83a5-0e2550272032-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-5h7rm\" (UID: \"7a313ddc-716a-4e94-83a5-0e2550272032\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-5h7rm" Oct 30 20:46:02.925502 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:46:02.925364 5378 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/7a313ddc-716a-4e94-83a5-0e2550272032-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-5h7rm\" (UID: \"7a313ddc-716a-4e94-83a5-0e2550272032\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-5h7rm" Oct 30 20:46:02.925502 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:46:02.925395 5378 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-78chx\" (UniqueName: \"kubernetes.io/projected/7a313ddc-716a-4e94-83a5-0e2550272032-kube-api-access-78chx\") pod \"istiod-openshift-gateway-7ddd9cfdc8-5h7rm\" (UID: \"7a313ddc-716a-4e94-83a5-0e2550272032\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-5h7rm" Oct 30 20:46:02.925502 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:46:02.925426 5378 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/7a313ddc-716a-4e94-83a5-0e2550272032-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-5h7rm\" (UID: \"7a313ddc-716a-4e94-83a5-0e2550272032\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-5h7rm" Oct 30 20:46:02.925502 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:46:02.925468 5378 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/7a313ddc-716a-4e94-83a5-0e2550272032-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-5h7rm\" (UID: \"7a313ddc-716a-4e94-83a5-0e2550272032\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-5h7rm" Oct 30 20:46:02.925756 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:46:02.925513 5378 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/7a313ddc-716a-4e94-83a5-0e2550272032-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-5h7rm\" (UID: \"7a313ddc-716a-4e94-83a5-0e2550272032\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-5h7rm" Oct 30 20:46:02.925756 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:46:02.925546 5378 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7a313ddc-716a-4e94-83a5-0e2550272032-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-5h7rm\" (UID: \"7a313ddc-716a-4e94-83a5-0e2550272032\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-5h7rm" Oct 30 20:46:02.926357 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:46:02.926325 5378 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/7a313ddc-716a-4e94-83a5-0e2550272032-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-5h7rm\" (UID: \"7a313ddc-716a-4e94-83a5-0e2550272032\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-5h7rm" Oct 30 20:46:02.927748 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:46:02.927717 5378 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/7a313ddc-716a-4e94-83a5-0e2550272032-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-5h7rm\" (UID: \"7a313ddc-716a-4e94-83a5-0e2550272032\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-5h7rm" Oct 30 20:46:02.927898 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:46:02.927877 5378 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/7a313ddc-716a-4e94-83a5-0e2550272032-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-5h7rm\" (UID: \"7a313ddc-716a-4e94-83a5-0e2550272032\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-5h7rm" Oct 30 20:46:02.927954 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:46:02.927900 5378 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/7a313ddc-716a-4e94-83a5-0e2550272032-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-5h7rm\" (UID: \"7a313ddc-716a-4e94-83a5-0e2550272032\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-5h7rm" Oct 30 20:46:02.927954 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:46:02.927939 5378 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/7a313ddc-716a-4e94-83a5-0e2550272032-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-5h7rm\" (UID: \"7a313ddc-716a-4e94-83a5-0e2550272032\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-5h7rm" Oct 30 20:46:03.006993 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:46:03.006959 5378 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-78chx\" (UniqueName: \"kubernetes.io/projected/7a313ddc-716a-4e94-83a5-0e2550272032-kube-api-access-78chx\") pod \"istiod-openshift-gateway-7ddd9cfdc8-5h7rm\" (UID: \"7a313ddc-716a-4e94-83a5-0e2550272032\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-5h7rm" Oct 30 20:46:03.007523 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:46:03.007501 5378 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7a313ddc-716a-4e94-83a5-0e2550272032-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-5h7rm\" (UID: \"7a313ddc-716a-4e94-83a5-0e2550272032\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-5h7rm" Oct 30 20:46:03.084643 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:46:03.084580 5378 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-5h7rm" Oct 30 20:46:03.513317 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:46:03.513291 5378 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-5h7rm"] Oct 30 20:46:03.515987 ip-10-0-138-149 kubenswrapper[5378]: W1030 20:46:03.515953 5378 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7a313ddc_716a_4e94_83a5_0e2550272032.slice/crio-86144767b7b2a1eed6554d43b4ddf105919d7ae17e8300faffd42896f2ad748f WatchSource:0}: Error finding container 86144767b7b2a1eed6554d43b4ddf105919d7ae17e8300faffd42896f2ad748f: Status 404 returned error can't find the container with id 86144767b7b2a1eed6554d43b4ddf105919d7ae17e8300faffd42896f2ad748f Oct 30 20:46:03.518005 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:46:03.517971 5378 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 20:46:03.518124 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:46:03.518041 5378 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 20:46:04.367480 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:46:04.367449 5378 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-5h7rm" event={"ID":"7a313ddc-716a-4e94-83a5-0e2550272032","Type":"ContainerStarted","Data":"0c3e2d9d6714ac0261821f05f875c9616a203001aafae746b516f0009221cba3"} Oct 30 20:46:04.367891 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:46:04.367497 5378 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-5h7rm" event={"ID":"7a313ddc-716a-4e94-83a5-0e2550272032","Type":"ContainerStarted","Data":"86144767b7b2a1eed6554d43b4ddf105919d7ae17e8300faffd42896f2ad748f"} Oct 30 20:46:04.367891 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:46:04.367610 5378 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-5h7rm" Oct 30 20:46:04.369225 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:46:04.369201 5378 patch_prober.go:29] interesting pod/istiod-openshift-gateway-7ddd9cfdc8-5h7rm container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 20:46:04.369324 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:46:04.369244 5378 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-5h7rm" podUID="7a313ddc-716a-4e94-83a5-0e2550272032" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 20:46:04.582535 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:46:04.582480 5378 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-5h7rm" podStartSLOduration=2.582466673 podStartE2EDuration="2.582466673s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:46:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:46:04.542199472 +0000 UTC m=+454.970313396" watchObservedRunningTime="2026-10-30 20:46:04.582466673 +0000 UTC m=+455.010580589" Oct 30 20:46:05.371962 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:46:05.371935 5378 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-5h7rm" Oct 30 20:46:05.593907 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:46:05.593877 5378 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7876bfcb74-dq8zm"] Oct 30 20:46:05.594114 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:46:05.594076 5378 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-dq8zm" podUID="ad251643-5dd1-42dd-8044-f42d88b0b32d" containerName="discovery" containerID="cri-o://a3dab19580f4a3c1e4e0324073bfb2d6ce63d8a0eae553d25b0ccbe7e4b1fa2e" gracePeriod=30 Oct 30 20:46:05.839350 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:46:05.839325 5378 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-dq8zm" Oct 30 20:46:05.952423 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:46:05.952355 5378 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ad251643-5dd1-42dd-8044-f42d88b0b32d-kube-api-access-jr6p2\" (UniqueName: \"kubernetes.io/projected/ad251643-5dd1-42dd-8044-f42d88b0b32d-kube-api-access-jr6p2\") pod \"ad251643-5dd1-42dd-8044-f42d88b0b32d\" (UID: \"ad251643-5dd1-42dd-8044-f42d88b0b32d\") " Oct 30 20:46:05.952423 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:46:05.952394 5378 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ad251643-5dd1-42dd-8044-f42d88b0b32d-istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/ad251643-5dd1-42dd-8044-f42d88b0b32d-istio-csr-dns-cert\") pod \"ad251643-5dd1-42dd-8044-f42d88b0b32d\" (UID: \"ad251643-5dd1-42dd-8044-f42d88b0b32d\") " Oct 30 20:46:05.952604 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:46:05.952436 5378 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ad251643-5dd1-42dd-8044-f42d88b0b32d-local-certs\" (UniqueName: \"kubernetes.io/empty-dir/ad251643-5dd1-42dd-8044-f42d88b0b32d-local-certs\") pod \"ad251643-5dd1-42dd-8044-f42d88b0b32d\" (UID: \"ad251643-5dd1-42dd-8044-f42d88b0b32d\") " Oct 30 20:46:05.952656 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:46:05.952634 5378 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ad251643-5dd1-42dd-8044-f42d88b0b32d-istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/ad251643-5dd1-42dd-8044-f42d88b0b32d-istio-csr-ca-configmap\") pod \"ad251643-5dd1-42dd-8044-f42d88b0b32d\" (UID: \"ad251643-5dd1-42dd-8044-f42d88b0b32d\") " Oct 30 20:46:05.952721 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:46:05.952700 5378 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ad251643-5dd1-42dd-8044-f42d88b0b32d-istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/ad251643-5dd1-42dd-8044-f42d88b0b32d-istio-kubeconfig\") pod \"ad251643-5dd1-42dd-8044-f42d88b0b32d\" (UID: \"ad251643-5dd1-42dd-8044-f42d88b0b32d\") " Oct 30 20:46:05.952768 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:46:05.952739 5378 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ad251643-5dd1-42dd-8044-f42d88b0b32d-istio-token\" (UniqueName: \"kubernetes.io/projected/ad251643-5dd1-42dd-8044-f42d88b0b32d-istio-token\") pod \"ad251643-5dd1-42dd-8044-f42d88b0b32d\" (UID: \"ad251643-5dd1-42dd-8044-f42d88b0b32d\") " Oct 30 20:46:05.952838 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:46:05.952794 5378 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ad251643-5dd1-42dd-8044-f42d88b0b32d-cacerts\" (UniqueName: \"kubernetes.io/secret/ad251643-5dd1-42dd-8044-f42d88b0b32d-cacerts\") pod \"ad251643-5dd1-42dd-8044-f42d88b0b32d\" (UID: \"ad251643-5dd1-42dd-8044-f42d88b0b32d\") " Oct 30 20:46:05.953033 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:46:05.953004 5378 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ad251643-5dd1-42dd-8044-f42d88b0b32d-istio-csr-ca-configmap" pod "ad251643-5dd1-42dd-8044-f42d88b0b32d" (UID: "ad251643-5dd1-42dd-8044-f42d88b0b32d"). InnerVolumeSpecName "istio-csr-ca-configmap". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:46:05.954813 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:46:05.954772 5378 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ad251643-5dd1-42dd-8044-f42d88b0b32d-istio-kubeconfig" pod "ad251643-5dd1-42dd-8044-f42d88b0b32d" (UID: "ad251643-5dd1-42dd-8044-f42d88b0b32d"). InnerVolumeSpecName "istio-kubeconfig". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 20:46:05.954935 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:46:05.954812 5378 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ad251643-5dd1-42dd-8044-f42d88b0b32d-local-certs" pod "ad251643-5dd1-42dd-8044-f42d88b0b32d" (UID: "ad251643-5dd1-42dd-8044-f42d88b0b32d"). InnerVolumeSpecName "local-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:46:05.955072 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:46:05.955042 5378 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ad251643-5dd1-42dd-8044-f42d88b0b32d-kube-api-access-jr6p2" pod "ad251643-5dd1-42dd-8044-f42d88b0b32d" (UID: "ad251643-5dd1-42dd-8044-f42d88b0b32d"). InnerVolumeSpecName "kube-api-access-jr6p2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:46:05.955228 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:46:05.955209 5378 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ad251643-5dd1-42dd-8044-f42d88b0b32d-istio-csr-dns-cert" pod "ad251643-5dd1-42dd-8044-f42d88b0b32d" (UID: "ad251643-5dd1-42dd-8044-f42d88b0b32d"). InnerVolumeSpecName "istio-csr-dns-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 20:46:05.955338 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:46:05.955317 5378 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ad251643-5dd1-42dd-8044-f42d88b0b32d-istio-token" pod "ad251643-5dd1-42dd-8044-f42d88b0b32d" (UID: "ad251643-5dd1-42dd-8044-f42d88b0b32d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:46:05.955398 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:46:05.955320 5378 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ad251643-5dd1-42dd-8044-f42d88b0b32d-cacerts" pod "ad251643-5dd1-42dd-8044-f42d88b0b32d" (UID: "ad251643-5dd1-42dd-8044-f42d88b0b32d"). InnerVolumeSpecName "cacerts". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 20:46:06.054074 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:46:06.054048 5378 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ad251643-5dd1-42dd-8044-f42d88b0b32d-istio-token\") on node \"ip-10-0-138-149.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:46:06.054074 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:46:06.054072 5378 reconciler_common.go:299] "Volume detached for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/ad251643-5dd1-42dd-8044-f42d88b0b32d-cacerts\") on node \"ip-10-0-138-149.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:46:06.054332 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:46:06.054083 5378 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jr6p2\" (UniqueName: \"kubernetes.io/projected/ad251643-5dd1-42dd-8044-f42d88b0b32d-kube-api-access-jr6p2\") on node \"ip-10-0-138-149.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:46:06.054332 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:46:06.054094 5378 reconciler_common.go:299] "Volume detached for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/ad251643-5dd1-42dd-8044-f42d88b0b32d-istio-csr-dns-cert\") on node \"ip-10-0-138-149.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:46:06.054332 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:46:06.054105 5378 reconciler_common.go:299] "Volume detached for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/ad251643-5dd1-42dd-8044-f42d88b0b32d-local-certs\") on node \"ip-10-0-138-149.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:46:06.054332 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:46:06.054114 5378 reconciler_common.go:299] "Volume detached for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/ad251643-5dd1-42dd-8044-f42d88b0b32d-istio-csr-ca-configmap\") on node \"ip-10-0-138-149.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:46:06.054332 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:46:06.054122 5378 reconciler_common.go:299] "Volume detached for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/ad251643-5dd1-42dd-8044-f42d88b0b32d-istio-kubeconfig\") on node \"ip-10-0-138-149.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:46:06.376016 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:46:06.375982 5378 generic.go:356] "Generic (PLEG): container finished" podID="ad251643-5dd1-42dd-8044-f42d88b0b32d" containerID="a3dab19580f4a3c1e4e0324073bfb2d6ce63d8a0eae553d25b0ccbe7e4b1fa2e" exitCode=0 Oct 30 20:46:06.376497 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:46:06.376037 5378 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-dq8zm" Oct 30 20:46:06.376497 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:46:06.376072 5378 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-dq8zm" event={"ID":"ad251643-5dd1-42dd-8044-f42d88b0b32d","Type":"ContainerDied","Data":"a3dab19580f4a3c1e4e0324073bfb2d6ce63d8a0eae553d25b0ccbe7e4b1fa2e"} Oct 30 20:46:06.376497 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:46:06.376149 5378 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-dq8zm" event={"ID":"ad251643-5dd1-42dd-8044-f42d88b0b32d","Type":"ContainerDied","Data":"3dde6e773287427695d008dcca8de7501c4e0e4f26b9adbf1f7d766032436d20"} Oct 30 20:46:06.376497 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:46:06.376171 5378 scope.go:122] "RemoveContainer" containerID="a3dab19580f4a3c1e4e0324073bfb2d6ce63d8a0eae553d25b0ccbe7e4b1fa2e" Oct 30 20:46:06.384941 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:46:06.384928 5378 scope.go:122] "RemoveContainer" containerID="a3dab19580f4a3c1e4e0324073bfb2d6ce63d8a0eae553d25b0ccbe7e4b1fa2e" Oct 30 20:46:06.385190 ip-10-0-138-149 kubenswrapper[5378]: E1030 20:46:06.385170 5378 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a3dab19580f4a3c1e4e0324073bfb2d6ce63d8a0eae553d25b0ccbe7e4b1fa2e\": container with ID starting with a3dab19580f4a3c1e4e0324073bfb2d6ce63d8a0eae553d25b0ccbe7e4b1fa2e not found: ID does not exist" containerID="a3dab19580f4a3c1e4e0324073bfb2d6ce63d8a0eae553d25b0ccbe7e4b1fa2e" Oct 30 20:46:06.385252 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:46:06.385198 5378 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a3dab19580f4a3c1e4e0324073bfb2d6ce63d8a0eae553d25b0ccbe7e4b1fa2e"} err="failed to get container status \"a3dab19580f4a3c1e4e0324073bfb2d6ce63d8a0eae553d25b0ccbe7e4b1fa2e\": rpc error: code = NotFound desc = could not find container \"a3dab19580f4a3c1e4e0324073bfb2d6ce63d8a0eae553d25b0ccbe7e4b1fa2e\": container with ID starting with a3dab19580f4a3c1e4e0324073bfb2d6ce63d8a0eae553d25b0ccbe7e4b1fa2e not found: ID does not exist" Oct 30 20:46:06.545966 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:46:06.545936 5378 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7876bfcb74-dq8zm"] Oct 30 20:46:06.596300 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:46:06.596268 5378 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7876bfcb74-dq8zm"] Oct 30 20:46:07.738318 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:46:07.738285 5378 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ad251643-5dd1-42dd-8044-f42d88b0b32d" path="/var/lib/kubelet/pods/ad251643-5dd1-42dd-8044-f42d88b0b32d/volumes" Oct 30 20:46:07.957304 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:46:07.957270 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 20:46:08.088440 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:46:08.088352 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 20:46:08.218366 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:46:08.218338 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 20:46:08.345327 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:46:08.345294 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 20:46:08.482705 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:46:08.482624 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 20:46:08.626367 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:46:08.626286 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 20:46:08.781076 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:46:08.781045 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 20:46:08.851835 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:46:08.851805 5378 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/kserve-controller-manager-776cc66d47-vk9qt"] Oct 30 20:46:08.852109 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:46:08.852097 5378 cpu_manager.go:414] "containerMap: removing container" podUID="ad251643-5dd1-42dd-8044-f42d88b0b32d" containerName="discovery" Oct 30 20:46:08.852151 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:46:08.852110 5378 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ad251643-5dd1-42dd-8044-f42d88b0b32d" containerName="discovery" Oct 30 20:46:08.852182 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:46:08.852176 5378 memory_manager.go:365] "RemoveStaleState removing state" podUID="ad251643-5dd1-42dd-8044-f42d88b0b32d" containerName="discovery" Oct 30 20:46:08.856273 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:46:08.856246 5378 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-controller-manager-776cc66d47-vk9qt" Oct 30 20:46:08.948938 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:46:08.948864 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 20:46:08.960926 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:46:08.960897 5378 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-controller-manager-dockercfg-slkq5\"" Oct 30 20:46:08.961054 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:46:08.960939 5378 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-webhook-server-cert\"" Oct 30 20:46:08.961757 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:46:08.961741 5378 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-controller-manager-776cc66d47-vk9qt"] Oct 30 20:46:08.975282 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:46:08.975253 5378 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/7bdaf64e-4296-4533-890f-c664325df4af-cert\") pod \"kserve-controller-manager-776cc66d47-vk9qt\" (UID: \"7bdaf64e-4296-4533-890f-c664325df4af\") " pod="opendatahub/kserve-controller-manager-776cc66d47-vk9qt" Oct 30 20:46:08.975381 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:46:08.975332 5378 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pgw8w\" (UniqueName: \"kubernetes.io/projected/7bdaf64e-4296-4533-890f-c664325df4af-kube-api-access-pgw8w\") pod \"kserve-controller-manager-776cc66d47-vk9qt\" (UID: \"7bdaf64e-4296-4533-890f-c664325df4af\") " pod="opendatahub/kserve-controller-manager-776cc66d47-vk9qt" Oct 30 20:46:09.076059 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:46:09.076026 5378 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pgw8w\" (UniqueName: \"kubernetes.io/projected/7bdaf64e-4296-4533-890f-c664325df4af-kube-api-access-pgw8w\") pod \"kserve-controller-manager-776cc66d47-vk9qt\" (UID: \"7bdaf64e-4296-4533-890f-c664325df4af\") " pod="opendatahub/kserve-controller-manager-776cc66d47-vk9qt" Oct 30 20:46:09.076183 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:46:09.076070 5378 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/7bdaf64e-4296-4533-890f-c664325df4af-cert\") pod \"kserve-controller-manager-776cc66d47-vk9qt\" (UID: \"7bdaf64e-4296-4533-890f-c664325df4af\") " pod="opendatahub/kserve-controller-manager-776cc66d47-vk9qt" Oct 30 20:46:09.078369 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:46:09.078346 5378 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/7bdaf64e-4296-4533-890f-c664325df4af-cert\") pod \"kserve-controller-manager-776cc66d47-vk9qt\" (UID: \"7bdaf64e-4296-4533-890f-c664325df4af\") " pod="opendatahub/kserve-controller-manager-776cc66d47-vk9qt" Oct 30 20:46:09.099875 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:46:09.099851 5378 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/model-serving-api-75bd89fdfc-gj4j5"] Oct 30 20:46:09.102993 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:46:09.102973 5378 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/model-serving-api-75bd89fdfc-gj4j5" Oct 30 20:46:09.119384 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:46:09.119366 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 20:46:09.134091 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:46:09.134073 5378 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pgw8w\" (UniqueName: \"kubernetes.io/projected/7bdaf64e-4296-4533-890f-c664325df4af-kube-api-access-pgw8w\") pod \"kserve-controller-manager-776cc66d47-vk9qt\" (UID: \"7bdaf64e-4296-4533-890f-c664325df4af\") " pod="opendatahub/kserve-controller-manager-776cc66d47-vk9qt" Oct 30 20:46:09.163384 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:46:09.163364 5378 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"model-serving-api-tls\"" Oct 30 20:46:09.163500 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:46:09.163390 5378 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"model-serving-api-dockercfg-m2722\"" Oct 30 20:46:09.164299 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:46:09.164278 5378 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/model-serving-api-75bd89fdfc-gj4j5"] Oct 30 20:46:09.168603 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:46:09.168558 5378 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-controller-manager-776cc66d47-vk9qt" Oct 30 20:46:09.176611 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:46:09.176585 5378 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e598f3d1-bf40-44c3-acdb-12dcdb3a7153-tls-certs\") pod \"model-serving-api-75bd89fdfc-gj4j5\" (UID: \"e598f3d1-bf40-44c3-acdb-12dcdb3a7153\") " pod="opendatahub/model-serving-api-75bd89fdfc-gj4j5" Oct 30 20:46:09.176720 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:46:09.176639 5378 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qqdbz\" (UniqueName: \"kubernetes.io/projected/e598f3d1-bf40-44c3-acdb-12dcdb3a7153-kube-api-access-qqdbz\") pod \"model-serving-api-75bd89fdfc-gj4j5\" (UID: \"e598f3d1-bf40-44c3-acdb-12dcdb3a7153\") " pod="opendatahub/model-serving-api-75bd89fdfc-gj4j5" Oct 30 20:46:09.194131 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:46:09.194106 5378 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/odh-model-controller-769875db56-66h86"] Oct 30 20:46:09.198400 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:46:09.198383 5378 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-model-controller-769875db56-66h86" Oct 30 20:46:09.255778 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:46:09.255748 5378 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/odh-model-controller-769875db56-66h86"] Oct 30 20:46:09.263502 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:46:09.263479 5378 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-metrics-tls\"" Oct 30 20:46:09.263644 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:46:09.263512 5378 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-webhook-cert\"" Oct 30 20:46:09.263644 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:46:09.263517 5378 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-dockercfg-pqvtk\"" Oct 30 20:46:09.277422 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:46:09.277394 5378 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qqdbz\" (UniqueName: \"kubernetes.io/projected/e598f3d1-bf40-44c3-acdb-12dcdb3a7153-kube-api-access-qqdbz\") pod \"model-serving-api-75bd89fdfc-gj4j5\" (UID: \"e598f3d1-bf40-44c3-acdb-12dcdb3a7153\") " pod="opendatahub/model-serving-api-75bd89fdfc-gj4j5" Oct 30 20:46:09.277532 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:46:09.277440 5378 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fqvkk\" (UniqueName: \"kubernetes.io/projected/cadcf041-ccba-4462-80a8-2d273479f50f-kube-api-access-fqvkk\") pod \"odh-model-controller-769875db56-66h86\" (UID: \"cadcf041-ccba-4462-80a8-2d273479f50f\") " pod="opendatahub/odh-model-controller-769875db56-66h86" Oct 30 20:46:09.277532 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:46:09.277520 5378 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e598f3d1-bf40-44c3-acdb-12dcdb3a7153-tls-certs\") pod \"model-serving-api-75bd89fdfc-gj4j5\" (UID: \"e598f3d1-bf40-44c3-acdb-12dcdb3a7153\") " pod="opendatahub/model-serving-api-75bd89fdfc-gj4j5" Oct 30 20:46:09.277657 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:46:09.277616 5378 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/cadcf041-ccba-4462-80a8-2d273479f50f-cert\") pod \"odh-model-controller-769875db56-66h86\" (UID: \"cadcf041-ccba-4462-80a8-2d273479f50f\") " pod="opendatahub/odh-model-controller-769875db56-66h86" Oct 30 20:46:09.277657 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:46:09.277649 5378 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cadcf041-ccba-4462-80a8-2d273479f50f-metrics-certs\") pod \"odh-model-controller-769875db56-66h86\" (UID: \"cadcf041-ccba-4462-80a8-2d273479f50f\") " pod="opendatahub/odh-model-controller-769875db56-66h86" Oct 30 20:46:09.279939 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:46:09.279917 5378 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e598f3d1-bf40-44c3-acdb-12dcdb3a7153-tls-certs\") pod \"model-serving-api-75bd89fdfc-gj4j5\" (UID: \"e598f3d1-bf40-44c3-acdb-12dcdb3a7153\") " pod="opendatahub/model-serving-api-75bd89fdfc-gj4j5" Oct 30 20:46:09.338234 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:46:09.338211 5378 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qqdbz\" (UniqueName: \"kubernetes.io/projected/e598f3d1-bf40-44c3-acdb-12dcdb3a7153-kube-api-access-qqdbz\") pod \"model-serving-api-75bd89fdfc-gj4j5\" (UID: \"e598f3d1-bf40-44c3-acdb-12dcdb3a7153\") " pod="opendatahub/model-serving-api-75bd89fdfc-gj4j5" Oct 30 20:46:09.378349 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:46:09.378321 5378 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/cadcf041-ccba-4462-80a8-2d273479f50f-cert\") pod \"odh-model-controller-769875db56-66h86\" (UID: \"cadcf041-ccba-4462-80a8-2d273479f50f\") " pod="opendatahub/odh-model-controller-769875db56-66h86" Oct 30 20:46:09.378458 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:46:09.378355 5378 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cadcf041-ccba-4462-80a8-2d273479f50f-metrics-certs\") pod \"odh-model-controller-769875db56-66h86\" (UID: \"cadcf041-ccba-4462-80a8-2d273479f50f\") " pod="opendatahub/odh-model-controller-769875db56-66h86" Oct 30 20:46:09.378458 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:46:09.378394 5378 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fqvkk\" (UniqueName: \"kubernetes.io/projected/cadcf041-ccba-4462-80a8-2d273479f50f-kube-api-access-fqvkk\") pod \"odh-model-controller-769875db56-66h86\" (UID: \"cadcf041-ccba-4462-80a8-2d273479f50f\") " pod="opendatahub/odh-model-controller-769875db56-66h86" Oct 30 20:46:09.380724 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:46:09.380704 5378 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cadcf041-ccba-4462-80a8-2d273479f50f-metrics-certs\") pod \"odh-model-controller-769875db56-66h86\" (UID: \"cadcf041-ccba-4462-80a8-2d273479f50f\") " pod="opendatahub/odh-model-controller-769875db56-66h86" Oct 30 20:46:09.380804 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:46:09.380743 5378 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/cadcf041-ccba-4462-80a8-2d273479f50f-cert\") pod \"odh-model-controller-769875db56-66h86\" (UID: \"cadcf041-ccba-4462-80a8-2d273479f50f\") " pod="opendatahub/odh-model-controller-769875db56-66h86" Oct 30 20:46:09.415217 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:46:09.415191 5378 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/model-serving-api-75bd89fdfc-gj4j5" Oct 30 20:46:09.436265 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:46:09.436244 5378 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fqvkk\" (UniqueName: \"kubernetes.io/projected/cadcf041-ccba-4462-80a8-2d273479f50f-kube-api-access-fqvkk\") pod \"odh-model-controller-769875db56-66h86\" (UID: \"cadcf041-ccba-4462-80a8-2d273479f50f\") " pod="opendatahub/odh-model-controller-769875db56-66h86" Oct 30 20:46:09.512727 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:46:09.512698 5378 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-model-controller-769875db56-66h86" Oct 30 20:46:09.537855 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:46:09.537832 5378 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-controller-manager-776cc66d47-vk9qt"] Oct 30 20:46:09.540276 ip-10-0-138-149 kubenswrapper[5378]: W1030 20:46:09.540250 5378 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7bdaf64e_4296_4533_890f_c664325df4af.slice/crio-13a171aec142621d9ab4623a6f6ab72d6aad3099fbec9cbcf5b52aa237874f58 WatchSource:0}: Error finding container 13a171aec142621d9ab4623a6f6ab72d6aad3099fbec9cbcf5b52aa237874f58: Status 404 returned error can't find the container with id 13a171aec142621d9ab4623a6f6ab72d6aad3099fbec9cbcf5b52aa237874f58 Oct 30 20:46:09.779928 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:46:09.779902 5378 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/model-serving-api-75bd89fdfc-gj4j5"] Oct 30 20:46:09.782249 ip-10-0-138-149 kubenswrapper[5378]: W1030 20:46:09.782217 5378 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode598f3d1_bf40_44c3_acdb_12dcdb3a7153.slice/crio-a5cb49b8d3435d33c2eb2afe682219aff08c2eb2a06f780b9306e80e3166c0b9 WatchSource:0}: Error finding container a5cb49b8d3435d33c2eb2afe682219aff08c2eb2a06f780b9306e80e3166c0b9: Status 404 returned error can't find the container with id a5cb49b8d3435d33c2eb2afe682219aff08c2eb2a06f780b9306e80e3166c0b9 Oct 30 20:46:09.883091 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:46:09.883066 5378 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/odh-model-controller-769875db56-66h86"] Oct 30 20:46:09.885426 ip-10-0-138-149 kubenswrapper[5378]: W1030 20:46:09.885402 5378 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcadcf041_ccba_4462_80a8_2d273479f50f.slice/crio-0d86518e91d57d01eb032614a14cd18f5b65440c48abb00d240f32d9445f51f4 WatchSource:0}: Error finding container 0d86518e91d57d01eb032614a14cd18f5b65440c48abb00d240f32d9445f51f4: Status 404 returned error can't find the container with id 0d86518e91d57d01eb032614a14cd18f5b65440c48abb00d240f32d9445f51f4 Oct 30 20:46:10.223140 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:46:10.223106 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 20:46:10.395798 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:46:10.395764 5378 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-769875db56-66h86" event={"ID":"cadcf041-ccba-4462-80a8-2d273479f50f","Type":"ContainerStarted","Data":"0d86518e91d57d01eb032614a14cd18f5b65440c48abb00d240f32d9445f51f4"} Oct 30 20:46:10.397333 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:46:10.397299 5378 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/model-serving-api-75bd89fdfc-gj4j5" event={"ID":"e598f3d1-bf40-44c3-acdb-12dcdb3a7153","Type":"ContainerStarted","Data":"a5cb49b8d3435d33c2eb2afe682219aff08c2eb2a06f780b9306e80e3166c0b9"} Oct 30 20:46:10.398610 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:46:10.398582 5378 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-controller-manager-776cc66d47-vk9qt" event={"ID":"7bdaf64e-4296-4533-890f-c664325df4af","Type":"ContainerStarted","Data":"13a171aec142621d9ab4623a6f6ab72d6aad3099fbec9cbcf5b52aa237874f58"} Oct 30 20:46:13.413756 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:46:13.413712 5378 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-controller-manager-776cc66d47-vk9qt" event={"ID":"7bdaf64e-4296-4533-890f-c664325df4af","Type":"ContainerStarted","Data":"adbae754115f04d910e899bbfaf5e7c993451dd3db0748e643662efa9c1752bb"} Oct 30 20:46:13.414237 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:46:13.413778 5378 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/kserve-controller-manager-776cc66d47-vk9qt" Oct 30 20:46:13.415383 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:46:13.415353 5378 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-769875db56-66h86" event={"ID":"cadcf041-ccba-4462-80a8-2d273479f50f","Type":"ContainerStarted","Data":"6bd54a03a36ef2e83261e06fa548aa150d7145d4e8c7f2a133af1473df5b09fe"} Oct 30 20:46:13.415518 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:46:13.415494 5378 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/odh-model-controller-769875db56-66h86" Oct 30 20:46:13.416881 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:46:13.416861 5378 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/model-serving-api-75bd89fdfc-gj4j5" event={"ID":"e598f3d1-bf40-44c3-acdb-12dcdb3a7153","Type":"ContainerStarted","Data":"19352bd5150193e9fd5fc9a3c6a46ed5a8438c2f9c10218652925109281a18ff"} Oct 30 20:46:13.416997 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:46:13.416981 5378 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/model-serving-api-75bd89fdfc-gj4j5" Oct 30 20:46:13.531038 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:46:13.530921 5378 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/kserve-controller-manager-776cc66d47-vk9qt" podStartSLOduration=2.580556675 podStartE2EDuration="5.530905943s" totalImagesPullingTime="2.950349268s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:46:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:46:13.530717995 +0000 UTC m=+463.958831915" watchObservedRunningTime="2026-10-30 20:46:13.530905943 +0000 UTC m=+463.959019864" Oct 30 20:46:13.646576 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:46:13.646523 5378 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/odh-model-controller-769875db56-66h86" podStartSLOduration=1.2873797759999999 podStartE2EDuration="4.646508583s" totalImagesPullingTime="3.359128807s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:46:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:46:13.646218535 +0000 UTC m=+464.074332456" watchObservedRunningTime="2026-10-30 20:46:13.646508583 +0000 UTC m=+464.074622496" Oct 30 20:46:13.762113 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:46:13.762049 5378 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/model-serving-api-75bd89fdfc-gj4j5" podStartSLOduration=1.610596366 podStartE2EDuration="4.762032305s" totalImagesPullingTime="3.151435939s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:46:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:46:13.761274321 +0000 UTC m=+464.189388235" watchObservedRunningTime="2026-10-30 20:46:13.762032305 +0000 UTC m=+464.190146225" Oct 30 20:46:15.308137 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:46:15.308108 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 20:46:15.442582 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:46:15.442532 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 20:46:15.586965 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:46:15.586887 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 20:46:15.734024 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:46:15.733978 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 20:46:15.885503 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:46:15.885475 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 20:46:16.058346 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:46:16.058306 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 20:46:16.216947 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:46:16.216875 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 20:46:16.469826 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:46:16.469748 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 20:46:16.670468 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:46:16.670434 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 20:46:18.622856 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:46:18.622824 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 20:46:18.836750 ip-10-0-138-149 kubenswrapper[5378]: E1030 20:46:18.836715 5378 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q6mx7" containerName="extractor" Oct 30 20:46:18.840098 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:46:18.840071 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 20:46:20.699909 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:46:20.699878 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 20:46:20.862785 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:46:20.862746 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 20:46:21.048715 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:46:21.048630 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 20:46:21.206251 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:46:21.206208 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 20:46:21.914707 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:46:21.914673 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 20:46:22.049141 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:46:22.049111 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 20:46:24.422727 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:46:24.422697 5378 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/odh-model-controller-769875db56-66h86" Oct 30 20:46:24.424917 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:46:24.424897 5378 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/model-serving-api-75bd89fdfc-gj4j5" Oct 30 20:46:27.273756 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:46:27.273723 5378 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-r9g59"] Oct 30 20:46:33.170381 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:46:33.170350 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 20:46:33.471397 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:46:33.471308 5378 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-6f6944bcdd-2khll"] Oct 30 20:46:33.478648 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:46:33.478623 5378 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6f6944bcdd-2khll" Oct 30 20:46:33.520433 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:46:33.520409 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 20:46:33.530605 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:46:33.530536 5378 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-6f6944bcdd-2khll"] Oct 30 20:46:33.575022 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:46:33.575000 5378 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pj76x\" (UniqueName: \"kubernetes.io/projected/5c44a13c-2436-4e61-a094-becb828e2580-kube-api-access-pj76x\") pod \"authorino-6f6944bcdd-2khll\" (UID: \"5c44a13c-2436-4e61-a094-becb828e2580\") " pod="kuadrant-system/authorino-6f6944bcdd-2khll" Oct 30 20:46:33.580228 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:46:33.580211 5378 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-qrtn9\"" Oct 30 20:46:33.676199 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:46:33.676166 5378 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pj76x\" (UniqueName: \"kubernetes.io/projected/5c44a13c-2436-4e61-a094-becb828e2580-kube-api-access-pj76x\") pod \"authorino-6f6944bcdd-2khll\" (UID: \"5c44a13c-2436-4e61-a094-becb828e2580\") " pod="kuadrant-system/authorino-6f6944bcdd-2khll" Oct 30 20:46:33.742453 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:46:33.742388 5378 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pj76x\" (UniqueName: \"kubernetes.io/projected/5c44a13c-2436-4e61-a094-becb828e2580-kube-api-access-pj76x\") pod \"authorino-6f6944bcdd-2khll\" (UID: \"5c44a13c-2436-4e61-a094-becb828e2580\") " pod="kuadrant-system/authorino-6f6944bcdd-2khll" Oct 30 20:46:33.789337 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:46:33.789309 5378 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6f6944bcdd-2khll" Oct 30 20:46:33.840141 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:46:33.840086 5378 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-6f6944bcdd-2khll"] Oct 30 20:46:33.936516 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:46:33.936483 5378 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-74988f67bf-j28zg"] Oct 30 20:46:33.942587 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:46:33.941480 5378 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-74988f67bf-j28zg" Oct 30 20:46:33.979016 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:46:33.978983 5378 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2sbrx\" (UniqueName: \"kubernetes.io/projected/81e1102b-bdfc-4515-9ece-fd20f82f0add-kube-api-access-2sbrx\") pod \"authorino-74988f67bf-j28zg\" (UID: \"81e1102b-bdfc-4515-9ece-fd20f82f0add\") " pod="kuadrant-system/authorino-74988f67bf-j28zg" Oct 30 20:46:34.007480 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:46:34.007416 5378 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-74988f67bf-j28zg"] Oct 30 20:46:34.079554 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:46:34.079523 5378 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2sbrx\" (UniqueName: \"kubernetes.io/projected/81e1102b-bdfc-4515-9ece-fd20f82f0add-kube-api-access-2sbrx\") pod \"authorino-74988f67bf-j28zg\" (UID: \"81e1102b-bdfc-4515-9ece-fd20f82f0add\") " pod="kuadrant-system/authorino-74988f67bf-j28zg" Oct 30 20:46:34.137816 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:46:34.137786 5378 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2sbrx\" (UniqueName: \"kubernetes.io/projected/81e1102b-bdfc-4515-9ece-fd20f82f0add-kube-api-access-2sbrx\") pod \"authorino-74988f67bf-j28zg\" (UID: \"81e1102b-bdfc-4515-9ece-fd20f82f0add\") " pod="kuadrant-system/authorino-74988f67bf-j28zg" Oct 30 20:46:34.156595 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:46:34.156536 5378 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-6f6944bcdd-2khll"] Oct 30 20:46:34.159382 ip-10-0-138-149 kubenswrapper[5378]: W1030 20:46:34.159358 5378 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod5c44a13c_2436_4e61_a094_becb828e2580.slice/crio-7041bc79dab99fd8d11a3f2c36acb634bbc0a57eae3789a39715825caf787a83 WatchSource:0}: Error finding container 7041bc79dab99fd8d11a3f2c36acb634bbc0a57eae3789a39715825caf787a83: Status 404 returned error can't find the container with id 7041bc79dab99fd8d11a3f2c36acb634bbc0a57eae3789a39715825caf787a83 Oct 30 20:46:34.253531 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:46:34.253505 5378 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-74988f67bf-j28zg" Oct 30 20:46:34.313680 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:46:34.313655 5378 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-74988f67bf-j28zg"] Oct 30 20:46:34.337779 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:46:34.337754 5378 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-7965b594c5-65jjp"] Oct 30 20:46:34.342106 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:46:34.342087 5378 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7965b594c5-65jjp" Oct 30 20:46:34.382424 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:46:34.382394 5378 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/75843e1d-767b-47b2-9dbb-b18a6025fe62-tls-cert\") pod \"authorino-7965b594c5-65jjp\" (UID: \"75843e1d-767b-47b2-9dbb-b18a6025fe62\") " pod="kuadrant-system/authorino-7965b594c5-65jjp" Oct 30 20:46:34.382529 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:46:34.382446 5378 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wv7nn\" (UniqueName: \"kubernetes.io/projected/75843e1d-767b-47b2-9dbb-b18a6025fe62-kube-api-access-wv7nn\") pod \"authorino-7965b594c5-65jjp\" (UID: \"75843e1d-767b-47b2-9dbb-b18a6025fe62\") " pod="kuadrant-system/authorino-7965b594c5-65jjp" Oct 30 20:46:34.395669 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:46:34.395642 5378 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-server-cert\"" Oct 30 20:46:34.396663 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:46:34.396639 5378 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-7965b594c5-65jjp"] Oct 30 20:46:34.482958 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:46:34.482933 5378 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wv7nn\" (UniqueName: \"kubernetes.io/projected/75843e1d-767b-47b2-9dbb-b18a6025fe62-kube-api-access-wv7nn\") pod \"authorino-7965b594c5-65jjp\" (UID: \"75843e1d-767b-47b2-9dbb-b18a6025fe62\") " pod="kuadrant-system/authorino-7965b594c5-65jjp" Oct 30 20:46:34.483090 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:46:34.482993 5378 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/75843e1d-767b-47b2-9dbb-b18a6025fe62-tls-cert\") pod \"authorino-7965b594c5-65jjp\" (UID: \"75843e1d-767b-47b2-9dbb-b18a6025fe62\") " pod="kuadrant-system/authorino-7965b594c5-65jjp" Oct 30 20:46:34.485322 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:46:34.485303 5378 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/75843e1d-767b-47b2-9dbb-b18a6025fe62-tls-cert\") pod \"authorino-7965b594c5-65jjp\" (UID: \"75843e1d-767b-47b2-9dbb-b18a6025fe62\") " pod="kuadrant-system/authorino-7965b594c5-65jjp" Oct 30 20:46:34.499427 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:46:34.499403 5378 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-6f6944bcdd-2khll" event={"ID":"5c44a13c-2436-4e61-a094-becb828e2580","Type":"ContainerStarted","Data":"7041bc79dab99fd8d11a3f2c36acb634bbc0a57eae3789a39715825caf787a83"} Oct 30 20:46:34.545315 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:46:34.545257 5378 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wv7nn\" (UniqueName: \"kubernetes.io/projected/75843e1d-767b-47b2-9dbb-b18a6025fe62-kube-api-access-wv7nn\") pod \"authorino-7965b594c5-65jjp\" (UID: \"75843e1d-767b-47b2-9dbb-b18a6025fe62\") " pod="kuadrant-system/authorino-7965b594c5-65jjp" Oct 30 20:46:34.629853 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:46:34.629801 5378 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-74988f67bf-j28zg"] Oct 30 20:46:34.632295 ip-10-0-138-149 kubenswrapper[5378]: W1030 20:46:34.632271 5378 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod81e1102b_bdfc_4515_9ece_fd20f82f0add.slice/crio-9e630f80ce8cb928dad1a97fdb134189dfc3d0c0f8fd22ee140ad6c6d17f6697 WatchSource:0}: Error finding container 9e630f80ce8cb928dad1a97fdb134189dfc3d0c0f8fd22ee140ad6c6d17f6697: Status 404 returned error can't find the container with id 9e630f80ce8cb928dad1a97fdb134189dfc3d0c0f8fd22ee140ad6c6d17f6697 Oct 30 20:46:34.654525 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:46:34.654504 5378 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7965b594c5-65jjp" Oct 30 20:46:35.019918 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:46:35.019878 5378 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-7965b594c5-65jjp"] Oct 30 20:46:35.022747 ip-10-0-138-149 kubenswrapper[5378]: W1030 20:46:35.022714 5378 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod75843e1d_767b_47b2_9dbb_b18a6025fe62.slice/crio-beae0667ec9b806d3592c8736c274dc0b3c572ba37421f58c93d9c219e8f05ea WatchSource:0}: Error finding container beae0667ec9b806d3592c8736c274dc0b3c572ba37421f58c93d9c219e8f05ea: Status 404 returned error can't find the container with id beae0667ec9b806d3592c8736c274dc0b3c572ba37421f58c93d9c219e8f05ea Oct 30 20:46:35.504607 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:46:35.504538 5378 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-74988f67bf-j28zg" event={"ID":"81e1102b-bdfc-4515-9ece-fd20f82f0add","Type":"ContainerStarted","Data":"9e630f80ce8cb928dad1a97fdb134189dfc3d0c0f8fd22ee140ad6c6d17f6697"} Oct 30 20:46:35.505744 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:46:35.505724 5378 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7965b594c5-65jjp" event={"ID":"75843e1d-767b-47b2-9dbb-b18a6025fe62","Type":"ContainerStarted","Data":"beae0667ec9b806d3592c8736c274dc0b3c572ba37421f58c93d9c219e8f05ea"} Oct 30 20:46:36.512102 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:46:36.512016 5378 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7965b594c5-65jjp" event={"ID":"75843e1d-767b-47b2-9dbb-b18a6025fe62","Type":"ContainerStarted","Data":"ea2a98804b892b97ca938fc88f74ef0ff7e900875f1594462a0dfe3efb5d2ca2"} Oct 30 20:46:36.513714 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:46:36.513688 5378 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-6f6944bcdd-2khll" event={"ID":"5c44a13c-2436-4e61-a094-becb828e2580","Type":"ContainerStarted","Data":"61534a63847b4b033681c5690290286bea99e77854fc32c30a55a6ab70f65507"} Oct 30 20:46:36.513835 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:46:36.513793 5378 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/authorino-6f6944bcdd-2khll" podUID="5c44a13c-2436-4e61-a094-becb828e2580" containerName="authorino" containerID="cri-o://61534a63847b4b033681c5690290286bea99e77854fc32c30a55a6ab70f65507" gracePeriod=30 Oct 30 20:46:36.515841 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:46:36.515813 5378 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-74988f67bf-j28zg" event={"ID":"81e1102b-bdfc-4515-9ece-fd20f82f0add","Type":"ContainerStarted","Data":"5934411661a038b0a8c59f2ee3478ec375c4f4ea00886627e6d7cff2497d4b8b"} Oct 30 20:46:36.515932 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:46:36.515890 5378 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/authorino-74988f67bf-j28zg" podUID="81e1102b-bdfc-4515-9ece-fd20f82f0add" containerName="authorino" containerID="cri-o://5934411661a038b0a8c59f2ee3478ec375c4f4ea00886627e6d7cff2497d4b8b" gracePeriod=30 Oct 30 20:46:36.627325 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:46:36.627274 5378 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/authorino-7965b594c5-65jjp" podStartSLOduration=1.42104422 podStartE2EDuration="2.627259965s" totalImagesPullingTime="1.206215745s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:46:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:46:36.626461112 +0000 UTC m=+487.054575032" watchObservedRunningTime="2026-10-30 20:46:36.627259965 +0000 UTC m=+487.055373917" Oct 30 20:46:36.784205 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:46:36.784179 5378 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-74988f67bf-j28zg" Oct 30 20:46:36.787539 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:46:36.787518 5378 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6f6944bcdd-2khll" Oct 30 20:46:36.802751 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:46:36.802725 5378 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/81e1102b-bdfc-4515-9ece-fd20f82f0add-kube-api-access-2sbrx\" (UniqueName: \"kubernetes.io/projected/81e1102b-bdfc-4515-9ece-fd20f82f0add-kube-api-access-2sbrx\") pod \"81e1102b-bdfc-4515-9ece-fd20f82f0add\" (UID: \"81e1102b-bdfc-4515-9ece-fd20f82f0add\") " Oct 30 20:46:36.802882 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:46:36.802780 5378 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5c44a13c-2436-4e61-a094-becb828e2580-kube-api-access-pj76x\" (UniqueName: \"kubernetes.io/projected/5c44a13c-2436-4e61-a094-becb828e2580-kube-api-access-pj76x\") pod \"5c44a13c-2436-4e61-a094-becb828e2580\" (UID: \"5c44a13c-2436-4e61-a094-becb828e2580\") " Oct 30 20:46:36.804874 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:46:36.804846 5378 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/81e1102b-bdfc-4515-9ece-fd20f82f0add-kube-api-access-2sbrx" pod "81e1102b-bdfc-4515-9ece-fd20f82f0add" (UID: "81e1102b-bdfc-4515-9ece-fd20f82f0add"). InnerVolumeSpecName "kube-api-access-2sbrx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:46:36.804994 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:46:36.804974 5378 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5c44a13c-2436-4e61-a094-becb828e2580-kube-api-access-pj76x" pod "5c44a13c-2436-4e61-a094-becb828e2580" (UID: "5c44a13c-2436-4e61-a094-becb828e2580"). InnerVolumeSpecName "kube-api-access-pj76x". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:46:36.857290 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:46:36.857246 5378 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/authorino-74988f67bf-j28zg" podStartSLOduration=2.659005334 podStartE2EDuration="3.85723298s" totalImagesPullingTime="1.198227646s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:46:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:46:36.74275507 +0000 UTC m=+487.170868992" watchObservedRunningTime="2026-10-30 20:46:36.85723298 +0000 UTC m=+487.285346908" Oct 30 20:46:36.857527 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:46:36.857508 5378 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/authorino-6f6944bcdd-2khll" podStartSLOduration=2.233742804 podStartE2EDuration="3.857503835s" totalImagesPullingTime="1.623761031s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:46:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:46:36.856068173 +0000 UTC m=+487.284182092" watchObservedRunningTime="2026-10-30 20:46:36.857503835 +0000 UTC m=+487.285617785" Oct 30 20:46:36.904172 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:46:36.904139 5378 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2sbrx\" (UniqueName: \"kubernetes.io/projected/81e1102b-bdfc-4515-9ece-fd20f82f0add-kube-api-access-2sbrx\") on node \"ip-10-0-138-149.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:46:36.904172 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:46:36.904171 5378 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pj76x\" (UniqueName: \"kubernetes.io/projected/5c44a13c-2436-4e61-a094-becb828e2580-kube-api-access-pj76x\") on node \"ip-10-0-138-149.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:46:37.522607 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:46:37.522555 5378 generic.go:356] "Generic (PLEG): container finished" podID="5c44a13c-2436-4e61-a094-becb828e2580" containerID="61534a63847b4b033681c5690290286bea99e77854fc32c30a55a6ab70f65507" exitCode=0 Oct 30 20:46:37.522607 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:46:37.522598 5378 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-6f6944bcdd-2khll" event={"ID":"5c44a13c-2436-4e61-a094-becb828e2580","Type":"ContainerDied","Data":"61534a63847b4b033681c5690290286bea99e77854fc32c30a55a6ab70f65507"} Oct 30 20:46:37.523107 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:46:37.522616 5378 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6f6944bcdd-2khll" Oct 30 20:46:37.523107 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:46:37.522642 5378 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-6f6944bcdd-2khll" event={"ID":"5c44a13c-2436-4e61-a094-becb828e2580","Type":"ContainerDied","Data":"7041bc79dab99fd8d11a3f2c36acb634bbc0a57eae3789a39715825caf787a83"} Oct 30 20:46:37.523107 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:46:37.522659 5378 scope.go:122] "RemoveContainer" containerID="61534a63847b4b033681c5690290286bea99e77854fc32c30a55a6ab70f65507" Oct 30 20:46:37.523945 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:46:37.523924 5378 generic.go:356] "Generic (PLEG): container finished" podID="81e1102b-bdfc-4515-9ece-fd20f82f0add" containerID="5934411661a038b0a8c59f2ee3478ec375c4f4ea00886627e6d7cff2497d4b8b" exitCode=0 Oct 30 20:46:37.524023 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:46:37.523968 5378 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-74988f67bf-j28zg" Oct 30 20:46:37.524023 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:46:37.523983 5378 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-74988f67bf-j28zg" event={"ID":"81e1102b-bdfc-4515-9ece-fd20f82f0add","Type":"ContainerDied","Data":"5934411661a038b0a8c59f2ee3478ec375c4f4ea00886627e6d7cff2497d4b8b"} Oct 30 20:46:37.524134 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:46:37.524029 5378 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-74988f67bf-j28zg" event={"ID":"81e1102b-bdfc-4515-9ece-fd20f82f0add","Type":"ContainerDied","Data":"9e630f80ce8cb928dad1a97fdb134189dfc3d0c0f8fd22ee140ad6c6d17f6697"} Oct 30 20:46:37.531729 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:46:37.531511 5378 scope.go:122] "RemoveContainer" containerID="61534a63847b4b033681c5690290286bea99e77854fc32c30a55a6ab70f65507" Oct 30 20:46:37.531946 ip-10-0-138-149 kubenswrapper[5378]: E1030 20:46:37.531922 5378 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"61534a63847b4b033681c5690290286bea99e77854fc32c30a55a6ab70f65507\": container with ID starting with 61534a63847b4b033681c5690290286bea99e77854fc32c30a55a6ab70f65507 not found: ID does not exist" containerID="61534a63847b4b033681c5690290286bea99e77854fc32c30a55a6ab70f65507" Oct 30 20:46:37.531993 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:46:37.531957 5378 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"61534a63847b4b033681c5690290286bea99e77854fc32c30a55a6ab70f65507"} err="failed to get container status \"61534a63847b4b033681c5690290286bea99e77854fc32c30a55a6ab70f65507\": rpc error: code = NotFound desc = could not find container \"61534a63847b4b033681c5690290286bea99e77854fc32c30a55a6ab70f65507\": container with ID starting with 61534a63847b4b033681c5690290286bea99e77854fc32c30a55a6ab70f65507 not found: ID does not exist" Oct 30 20:46:37.531993 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:46:37.531980 5378 scope.go:122] "RemoveContainer" containerID="5934411661a038b0a8c59f2ee3478ec375c4f4ea00886627e6d7cff2497d4b8b" Oct 30 20:46:37.539308 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:46:37.539286 5378 scope.go:122] "RemoveContainer" containerID="5934411661a038b0a8c59f2ee3478ec375c4f4ea00886627e6d7cff2497d4b8b" Oct 30 20:46:37.539539 ip-10-0-138-149 kubenswrapper[5378]: E1030 20:46:37.539518 5378 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5934411661a038b0a8c59f2ee3478ec375c4f4ea00886627e6d7cff2497d4b8b\": container with ID starting with 5934411661a038b0a8c59f2ee3478ec375c4f4ea00886627e6d7cff2497d4b8b not found: ID does not exist" containerID="5934411661a038b0a8c59f2ee3478ec375c4f4ea00886627e6d7cff2497d4b8b" Oct 30 20:46:37.539658 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:46:37.539544 5378 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5934411661a038b0a8c59f2ee3478ec375c4f4ea00886627e6d7cff2497d4b8b"} err="failed to get container status \"5934411661a038b0a8c59f2ee3478ec375c4f4ea00886627e6d7cff2497d4b8b\": rpc error: code = NotFound desc = could not find container \"5934411661a038b0a8c59f2ee3478ec375c4f4ea00886627e6d7cff2497d4b8b\": container with ID starting with 5934411661a038b0a8c59f2ee3478ec375c4f4ea00886627e6d7cff2497d4b8b not found: ID does not exist" Oct 30 20:46:37.685626 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:46:37.685591 5378 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-6f6944bcdd-2khll"] Oct 30 20:46:37.691331 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:46:37.690488 5378 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-6f6944bcdd-2khll"] Oct 30 20:46:37.739225 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:46:37.739194 5378 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5c44a13c-2436-4e61-a094-becb828e2580" path="/var/lib/kubelet/pods/5c44a13c-2436-4e61-a094-becb828e2580/volumes" Oct 30 20:46:37.799605 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:46:37.799516 5378 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-74988f67bf-j28zg"] Oct 30 20:46:37.801653 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:46:37.801630 5378 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-74988f67bf-j28zg"] Oct 30 20:46:38.433550 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:46:38.433505 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 20:46:39.738984 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:46:39.738949 5378 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="81e1102b-bdfc-4515-9ece-fd20f82f0add" path="/var/lib/kubelet/pods/81e1102b-bdfc-4515-9ece-fd20f82f0add/volumes" Oct 30 20:46:44.426587 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:46:44.426543 5378 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/kserve-controller-manager-776cc66d47-vk9qt" Oct 30 20:46:48.835807 ip-10-0-138-149 kubenswrapper[5378]: E1030 20:46:48.835769 5378 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q6mx7" containerName="extractor" Oct 30 20:47:18.835350 ip-10-0-138-149 kubenswrapper[5378]: E1030 20:47:18.835300 5378 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q6mx7" containerName="extractor" Oct 30 20:47:38.801035 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:47:38.800938 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 20:47:48.835258 ip-10-0-138-149 kubenswrapper[5378]: E1030 20:47:48.835221 5378 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q6mx7" containerName="extractor" Oct 30 20:47:51.944355 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:47:51.944306 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 20:47:52.090874 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:47:52.090800 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 20:48:13.208218 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:48:13.208181 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 20:48:16.487060 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:48:16.487025 5378 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-69b588897d-tkbxl"] Oct 30 20:48:16.487432 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:48:16.487338 5378 cpu_manager.go:414] "containerMap: removing container" podUID="5c44a13c-2436-4e61-a094-becb828e2580" containerName="authorino" Oct 30 20:48:16.487432 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:48:16.487349 5378 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5c44a13c-2436-4e61-a094-becb828e2580" containerName="authorino" Oct 30 20:48:16.487432 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:48:16.487361 5378 cpu_manager.go:414] "containerMap: removing container" podUID="81e1102b-bdfc-4515-9ece-fd20f82f0add" containerName="authorino" Oct 30 20:48:16.487432 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:48:16.487366 5378 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="81e1102b-bdfc-4515-9ece-fd20f82f0add" containerName="authorino" Oct 30 20:48:16.487432 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:48:16.487427 5378 memory_manager.go:365] "RemoveStaleState removing state" podUID="5c44a13c-2436-4e61-a094-becb828e2580" containerName="authorino" Oct 30 20:48:16.487432 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:48:16.487435 5378 memory_manager.go:365] "RemoveStaleState removing state" podUID="81e1102b-bdfc-4515-9ece-fd20f82f0add" containerName="authorino" Oct 30 20:48:16.489279 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:48:16.489262 5378 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-tkbxl" Oct 30 20:48:16.543141 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:48:16.543084 5378 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\"" Oct 30 20:48:16.578546 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:48:16.578514 5378 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/13d6b67c-8825-489e-91af-407eb360cae7-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-tkbxl\" (UID: \"13d6b67c-8825-489e-91af-407eb360cae7\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-tkbxl" Oct 30 20:48:16.578723 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:48:16.578598 5378 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/13d6b67c-8825-489e-91af-407eb360cae7-model-cache\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-tkbxl\" (UID: \"13d6b67c-8825-489e-91af-407eb360cae7\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-tkbxl" Oct 30 20:48:16.578723 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:48:16.578694 5378 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/13d6b67c-8825-489e-91af-407eb360cae7-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-tkbxl\" (UID: \"13d6b67c-8825-489e-91af-407eb360cae7\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-tkbxl" Oct 30 20:48:16.578815 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:48:16.578755 5378 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/13d6b67c-8825-489e-91af-407eb360cae7-home\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-tkbxl\" (UID: \"13d6b67c-8825-489e-91af-407eb360cae7\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-tkbxl" Oct 30 20:48:16.578856 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:48:16.578821 5378 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/13d6b67c-8825-489e-91af-407eb360cae7-dshm\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-tkbxl\" (UID: \"13d6b67c-8825-489e-91af-407eb360cae7\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-tkbxl" Oct 30 20:48:16.578899 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:48:16.578866 5378 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vtjcr\" (UniqueName: \"kubernetes.io/projected/13d6b67c-8825-489e-91af-407eb360cae7-kube-api-access-vtjcr\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-tkbxl\" (UID: \"13d6b67c-8825-489e-91af-407eb360cae7\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-tkbxl" Oct 30 20:48:16.591849 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:48:16.591824 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 20:48:16.592963 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:48:16.592940 5378 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\"" Oct 30 20:48:16.593072 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:48:16.593022 5378 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-trlp-test-simulated-kserve-self-signed-certs\"" Oct 30 20:48:16.593072 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:48:16.593022 5378 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-r2j5p\"" Oct 30 20:48:16.595184 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:48:16.595162 5378 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-69b588897d-tkbxl"] Oct 30 20:48:16.679844 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:48:16.679811 5378 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vtjcr\" (UniqueName: \"kubernetes.io/projected/13d6b67c-8825-489e-91af-407eb360cae7-kube-api-access-vtjcr\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-tkbxl\" (UID: \"13d6b67c-8825-489e-91af-407eb360cae7\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-tkbxl" Oct 30 20:48:16.680009 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:48:16.679869 5378 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/13d6b67c-8825-489e-91af-407eb360cae7-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-tkbxl\" (UID: \"13d6b67c-8825-489e-91af-407eb360cae7\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-tkbxl" Oct 30 20:48:16.680009 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:48:16.679910 5378 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/13d6b67c-8825-489e-91af-407eb360cae7-model-cache\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-tkbxl\" (UID: \"13d6b67c-8825-489e-91af-407eb360cae7\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-tkbxl" Oct 30 20:48:16.680009 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:48:16.679926 5378 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/13d6b67c-8825-489e-91af-407eb360cae7-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-tkbxl\" (UID: \"13d6b67c-8825-489e-91af-407eb360cae7\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-tkbxl" Oct 30 20:48:16.680009 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:48:16.679955 5378 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/13d6b67c-8825-489e-91af-407eb360cae7-home\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-tkbxl\" (UID: \"13d6b67c-8825-489e-91af-407eb360cae7\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-tkbxl" Oct 30 20:48:16.680009 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:48:16.679984 5378 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/13d6b67c-8825-489e-91af-407eb360cae7-dshm\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-tkbxl\" (UID: \"13d6b67c-8825-489e-91af-407eb360cae7\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-tkbxl" Oct 30 20:48:16.680399 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:48:16.680367 5378 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/13d6b67c-8825-489e-91af-407eb360cae7-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-tkbxl\" (UID: \"13d6b67c-8825-489e-91af-407eb360cae7\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-tkbxl" Oct 30 20:48:16.680399 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:48:16.680386 5378 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/13d6b67c-8825-489e-91af-407eb360cae7-model-cache\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-tkbxl\" (UID: \"13d6b67c-8825-489e-91af-407eb360cae7\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-tkbxl" Oct 30 20:48:16.680532 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:48:16.680430 5378 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/13d6b67c-8825-489e-91af-407eb360cae7-home\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-tkbxl\" (UID: \"13d6b67c-8825-489e-91af-407eb360cae7\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-tkbxl" Oct 30 20:48:16.682317 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:48:16.682297 5378 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/13d6b67c-8825-489e-91af-407eb360cae7-dshm\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-tkbxl\" (UID: \"13d6b67c-8825-489e-91af-407eb360cae7\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-tkbxl" Oct 30 20:48:16.682651 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:48:16.682632 5378 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/13d6b67c-8825-489e-91af-407eb360cae7-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-tkbxl\" (UID: \"13d6b67c-8825-489e-91af-407eb360cae7\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-tkbxl" Oct 30 20:48:16.686409 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:48:16.686385 5378 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-r9g59"] Oct 30 20:48:16.743467 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:48:16.743393 5378 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vtjcr\" (UniqueName: \"kubernetes.io/projected/13d6b67c-8825-489e-91af-407eb360cae7-kube-api-access-vtjcr\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-tkbxl\" (UID: \"13d6b67c-8825-489e-91af-407eb360cae7\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-tkbxl" Oct 30 20:48:16.801112 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:48:16.801077 5378 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-tkbxl" Oct 30 20:48:17.179259 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:48:17.179238 5378 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-69b588897d-tkbxl"] Oct 30 20:48:17.181364 ip-10-0-138-149 kubenswrapper[5378]: W1030 20:48:17.181334 5378 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod13d6b67c_8825_489e_91af_407eb360cae7.slice/crio-06ea2745eeb645003a2315f86c51918b38a4257b75ed632f69dc782a3190d0cd WatchSource:0}: Error finding container 06ea2745eeb645003a2315f86c51918b38a4257b75ed632f69dc782a3190d0cd: Status 404 returned error can't find the container with id 06ea2745eeb645003a2315f86c51918b38a4257b75ed632f69dc782a3190d0cd Oct 30 20:48:17.928753 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:48:17.928716 5378 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-tkbxl" event={"ID":"13d6b67c-8825-489e-91af-407eb360cae7","Type":"ContainerStarted","Data":"06ea2745eeb645003a2315f86c51918b38a4257b75ed632f69dc782a3190d0cd"} Oct 30 20:48:18.835496 ip-10-0-138-149 kubenswrapper[5378]: E1030 20:48:18.835462 5378 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q6mx7" containerName="extractor" Oct 30 20:48:18.934211 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:48:18.934136 5378 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-tkbxl" event={"ID":"13d6b67c-8825-489e-91af-407eb360cae7","Type":"ContainerStarted","Data":"d7479ce50179be338e0cb26f76046b0ecbc175b9ad1bb836e5d766afec4a2d10"} Oct 30 20:48:19.802123 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:48:19.802091 5378 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-tkbxl" Oct 30 20:48:19.926461 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:48:19.926428 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 20:48:23.479458 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:48:23.479406 5378 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-tkbxl" podStartSLOduration=6.048095638 podStartE2EDuration="7.479391401s" totalImagesPullingTime="1.431295763s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:48:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:48:19.064218483 +0000 UTC m=+589.492332399" watchObservedRunningTime="2026-10-30 20:48:23.479391401 +0000 UTC m=+593.907505357" Oct 30 20:48:23.479942 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:48:23.479606 5378 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/echo-simulator-kserve-64b6f8f96c-hjktp"] Oct 30 20:48:23.481794 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:48:23.481777 5378 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/echo-simulator-kserve-64b6f8f96c-hjktp" Oct 30 20:48:23.546754 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:48:23.546726 5378 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"echo-simulator-kserve-self-signed-certs\"" Oct 30 20:48:23.587064 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:48:23.587031 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 20:48:23.590917 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:48:23.590895 5378 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/echo-simulator-kserve-64b6f8f96c-hjktp"] Oct 30 20:48:23.644265 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:48:23.644222 5378 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/52a53c39-758b-41c5-be5d-729a097e58fc-dshm\") pod \"echo-simulator-kserve-64b6f8f96c-hjktp\" (UID: \"52a53c39-758b-41c5-be5d-729a097e58fc\") " pod="llm/echo-simulator-kserve-64b6f8f96c-hjktp" Oct 30 20:48:23.644444 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:48:23.644273 5378 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/52a53c39-758b-41c5-be5d-729a097e58fc-tmp-dir\") pod \"echo-simulator-kserve-64b6f8f96c-hjktp\" (UID: \"52a53c39-758b-41c5-be5d-729a097e58fc\") " pod="llm/echo-simulator-kserve-64b6f8f96c-hjktp" Oct 30 20:48:23.644444 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:48:23.644356 5378 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/52a53c39-758b-41c5-be5d-729a097e58fc-model-cache\") pod \"echo-simulator-kserve-64b6f8f96c-hjktp\" (UID: \"52a53c39-758b-41c5-be5d-729a097e58fc\") " pod="llm/echo-simulator-kserve-64b6f8f96c-hjktp" Oct 30 20:48:23.644528 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:48:23.644460 5378 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dlccb\" (UniqueName: \"kubernetes.io/projected/52a53c39-758b-41c5-be5d-729a097e58fc-kube-api-access-dlccb\") pod \"echo-simulator-kserve-64b6f8f96c-hjktp\" (UID: \"52a53c39-758b-41c5-be5d-729a097e58fc\") " pod="llm/echo-simulator-kserve-64b6f8f96c-hjktp" Oct 30 20:48:23.644528 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:48:23.644507 5378 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/52a53c39-758b-41c5-be5d-729a097e58fc-tls-certs\") pod \"echo-simulator-kserve-64b6f8f96c-hjktp\" (UID: \"52a53c39-758b-41c5-be5d-729a097e58fc\") " pod="llm/echo-simulator-kserve-64b6f8f96c-hjktp" Oct 30 20:48:23.644644 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:48:23.644534 5378 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/52a53c39-758b-41c5-be5d-729a097e58fc-home\") pod \"echo-simulator-kserve-64b6f8f96c-hjktp\" (UID: \"52a53c39-758b-41c5-be5d-729a097e58fc\") " pod="llm/echo-simulator-kserve-64b6f8f96c-hjktp" Oct 30 20:48:23.745820 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:48:23.745734 5378 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dlccb\" (UniqueName: \"kubernetes.io/projected/52a53c39-758b-41c5-be5d-729a097e58fc-kube-api-access-dlccb\") pod \"echo-simulator-kserve-64b6f8f96c-hjktp\" (UID: \"52a53c39-758b-41c5-be5d-729a097e58fc\") " pod="llm/echo-simulator-kserve-64b6f8f96c-hjktp" Oct 30 20:48:23.745820 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:48:23.745809 5378 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/52a53c39-758b-41c5-be5d-729a097e58fc-tls-certs\") pod \"echo-simulator-kserve-64b6f8f96c-hjktp\" (UID: \"52a53c39-758b-41c5-be5d-729a097e58fc\") " pod="llm/echo-simulator-kserve-64b6f8f96c-hjktp" Oct 30 20:48:23.746049 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:48:23.745851 5378 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/52a53c39-758b-41c5-be5d-729a097e58fc-home\") pod \"echo-simulator-kserve-64b6f8f96c-hjktp\" (UID: \"52a53c39-758b-41c5-be5d-729a097e58fc\") " pod="llm/echo-simulator-kserve-64b6f8f96c-hjktp" Oct 30 20:48:23.746049 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:48:23.745907 5378 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/52a53c39-758b-41c5-be5d-729a097e58fc-dshm\") pod \"echo-simulator-kserve-64b6f8f96c-hjktp\" (UID: \"52a53c39-758b-41c5-be5d-729a097e58fc\") " pod="llm/echo-simulator-kserve-64b6f8f96c-hjktp" Oct 30 20:48:23.746049 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:48:23.745932 5378 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/52a53c39-758b-41c5-be5d-729a097e58fc-tmp-dir\") pod \"echo-simulator-kserve-64b6f8f96c-hjktp\" (UID: \"52a53c39-758b-41c5-be5d-729a097e58fc\") " pod="llm/echo-simulator-kserve-64b6f8f96c-hjktp" Oct 30 20:48:23.746049 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:48:23.745957 5378 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/52a53c39-758b-41c5-be5d-729a097e58fc-model-cache\") pod \"echo-simulator-kserve-64b6f8f96c-hjktp\" (UID: \"52a53c39-758b-41c5-be5d-729a097e58fc\") " pod="llm/echo-simulator-kserve-64b6f8f96c-hjktp" Oct 30 20:48:23.746444 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:48:23.746393 5378 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/52a53c39-758b-41c5-be5d-729a097e58fc-home\") pod \"echo-simulator-kserve-64b6f8f96c-hjktp\" (UID: \"52a53c39-758b-41c5-be5d-729a097e58fc\") " pod="llm/echo-simulator-kserve-64b6f8f96c-hjktp" Oct 30 20:48:23.746576 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:48:23.746453 5378 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/52a53c39-758b-41c5-be5d-729a097e58fc-tmp-dir\") pod \"echo-simulator-kserve-64b6f8f96c-hjktp\" (UID: \"52a53c39-758b-41c5-be5d-729a097e58fc\") " pod="llm/echo-simulator-kserve-64b6f8f96c-hjktp" Oct 30 20:48:23.746576 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:48:23.746478 5378 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/52a53c39-758b-41c5-be5d-729a097e58fc-model-cache\") pod \"echo-simulator-kserve-64b6f8f96c-hjktp\" (UID: \"52a53c39-758b-41c5-be5d-729a097e58fc\") " pod="llm/echo-simulator-kserve-64b6f8f96c-hjktp" Oct 30 20:48:23.748443 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:48:23.748415 5378 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/52a53c39-758b-41c5-be5d-729a097e58fc-dshm\") pod \"echo-simulator-kserve-64b6f8f96c-hjktp\" (UID: \"52a53c39-758b-41c5-be5d-729a097e58fc\") " pod="llm/echo-simulator-kserve-64b6f8f96c-hjktp" Oct 30 20:48:23.748443 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:48:23.748429 5378 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/52a53c39-758b-41c5-be5d-729a097e58fc-tls-certs\") pod \"echo-simulator-kserve-64b6f8f96c-hjktp\" (UID: \"52a53c39-758b-41c5-be5d-729a097e58fc\") " pod="llm/echo-simulator-kserve-64b6f8f96c-hjktp" Oct 30 20:48:23.803718 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:48:23.803684 5378 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dlccb\" (UniqueName: \"kubernetes.io/projected/52a53c39-758b-41c5-be5d-729a097e58fc-kube-api-access-dlccb\") pod \"echo-simulator-kserve-64b6f8f96c-hjktp\" (UID: \"52a53c39-758b-41c5-be5d-729a097e58fc\") " pod="llm/echo-simulator-kserve-64b6f8f96c-hjktp" Oct 30 20:48:24.096199 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:48:24.096166 5378 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/echo-simulator-kserve-64b6f8f96c-hjktp" Oct 30 20:48:24.491227 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:48:24.491200 5378 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/echo-simulator-kserve-64b6f8f96c-hjktp"] Oct 30 20:48:24.493619 ip-10-0-138-149 kubenswrapper[5378]: W1030 20:48:24.493591 5378 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod52a53c39_758b_41c5_be5d_729a097e58fc.slice/crio-a58a40f8c06a73e759e69294b9fe357c14373c6e6543eb4c916ff2659cf87dc0 WatchSource:0}: Error finding container a58a40f8c06a73e759e69294b9fe357c14373c6e6543eb4c916ff2659cf87dc0: Status 404 returned error can't find the container with id a58a40f8c06a73e759e69294b9fe357c14373c6e6543eb4c916ff2659cf87dc0 Oct 30 20:48:24.958087 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:48:24.958051 5378 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/echo-simulator-kserve-64b6f8f96c-hjktp" event={"ID":"52a53c39-758b-41c5-be5d-729a097e58fc","Type":"ContainerStarted","Data":"a58a40f8c06a73e759e69294b9fe357c14373c6e6543eb4c916ff2659cf87dc0"} Oct 30 20:48:26.542374 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:48:26.542344 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 20:48:26.801327 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:48:26.801250 5378 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-tkbxl" Oct 30 20:48:26.815482 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:48:26.815457 5378 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-tkbxl" Oct 30 20:48:26.977719 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:48:26.977600 5378 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-tkbxl" Oct 30 20:48:27.971915 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:48:27.971874 5378 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/echo-simulator-kserve-64b6f8f96c-hjktp" event={"ID":"52a53c39-758b-41c5-be5d-729a097e58fc","Type":"ContainerStarted","Data":"182a393840d68028458978b84b65004453188cf64b4a3ff18ca7fe7988c8f47c"} Oct 30 20:48:28.097023 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:48:28.096981 5378 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/echo-simulator-kserve-64b6f8f96c-hjktp" Oct 30 20:48:28.142205 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:48:28.142145 5378 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/echo-simulator-kserve-64b6f8f96c-hjktp" podStartSLOduration=2.6583682680000003 podStartE2EDuration="5.142130122s" totalImagesPullingTime="2.483761854s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:48:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:48:28.105294387 +0000 UTC m=+598.533408306" watchObservedRunningTime="2026-10-30 20:48:28.142130122 +0000 UTC m=+598.570244034" Oct 30 20:48:29.625274 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:48:29.625241 5378 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-r9g59"] Oct 30 20:48:29.783683 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:48:29.783653 5378 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-7x9jg_c2ff359c-cf92-4a5b-8610-f6eb41acbe73/ovn-acl-logging/0.log" Oct 30 20:48:29.784311 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:48:29.784271 5378 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-7x9jg_c2ff359c-cf92-4a5b-8610-f6eb41acbe73/ovn-acl-logging/0.log" Oct 30 20:48:33.922659 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:48:33.922627 5378 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-cbd4f5586-xdmrj"] Oct 30 20:48:33.925957 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:48:33.925940 5378 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-xdmrj" Oct 30 20:48:34.025369 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:48:34.025341 5378 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-distinct-2-simulated-kserve-self-signed-certs\"" Oct 30 20:48:34.029739 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:48:34.029715 5378 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c9c625ce-f432-4bd9-8a6f-0a01335b886b-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-xdmrj\" (UID: \"c9c625ce-f432-4bd9-8a6f-0a01335b886b\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-xdmrj" Oct 30 20:48:34.029819 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:48:34.029787 5378 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c9c625ce-f432-4bd9-8a6f-0a01335b886b-dshm\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-xdmrj\" (UID: \"c9c625ce-f432-4bd9-8a6f-0a01335b886b\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-xdmrj" Oct 30 20:48:34.029883 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:48:34.029867 5378 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c9c625ce-f432-4bd9-8a6f-0a01335b886b-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-xdmrj\" (UID: \"c9c625ce-f432-4bd9-8a6f-0a01335b886b\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-xdmrj" Oct 30 20:48:34.029923 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:48:34.029898 5378 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c9c625ce-f432-4bd9-8a6f-0a01335b886b-model-cache\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-xdmrj\" (UID: \"c9c625ce-f432-4bd9-8a6f-0a01335b886b\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-xdmrj" Oct 30 20:48:34.030020 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:48:34.030000 5378 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fs5zv\" (UniqueName: \"kubernetes.io/projected/c9c625ce-f432-4bd9-8a6f-0a01335b886b-kube-api-access-fs5zv\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-xdmrj\" (UID: \"c9c625ce-f432-4bd9-8a6f-0a01335b886b\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-xdmrj" Oct 30 20:48:34.030061 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:48:34.030040 5378 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c9c625ce-f432-4bd9-8a6f-0a01335b886b-home\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-xdmrj\" (UID: \"c9c625ce-f432-4bd9-8a6f-0a01335b886b\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-xdmrj" Oct 30 20:48:34.030953 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:48:34.030935 5378 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-cbd4f5586-xdmrj"] Oct 30 20:48:34.078268 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:48:34.078242 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 20:48:34.097207 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:48:34.097181 5378 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/echo-simulator-kserve-64b6f8f96c-hjktp" Oct 30 20:48:34.109546 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:48:34.109526 5378 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/echo-simulator-kserve-64b6f8f96c-hjktp" Oct 30 20:48:34.130992 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:48:34.130968 5378 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fs5zv\" (UniqueName: \"kubernetes.io/projected/c9c625ce-f432-4bd9-8a6f-0a01335b886b-kube-api-access-fs5zv\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-xdmrj\" (UID: \"c9c625ce-f432-4bd9-8a6f-0a01335b886b\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-xdmrj" Oct 30 20:48:34.131094 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:48:34.130999 5378 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c9c625ce-f432-4bd9-8a6f-0a01335b886b-home\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-xdmrj\" (UID: \"c9c625ce-f432-4bd9-8a6f-0a01335b886b\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-xdmrj" Oct 30 20:48:34.131094 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:48:34.131033 5378 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c9c625ce-f432-4bd9-8a6f-0a01335b886b-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-xdmrj\" (UID: \"c9c625ce-f432-4bd9-8a6f-0a01335b886b\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-xdmrj" Oct 30 20:48:34.131094 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:48:34.131079 5378 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c9c625ce-f432-4bd9-8a6f-0a01335b886b-dshm\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-xdmrj\" (UID: \"c9c625ce-f432-4bd9-8a6f-0a01335b886b\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-xdmrj" Oct 30 20:48:34.131258 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:48:34.131115 5378 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c9c625ce-f432-4bd9-8a6f-0a01335b886b-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-xdmrj\" (UID: \"c9c625ce-f432-4bd9-8a6f-0a01335b886b\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-xdmrj" Oct 30 20:48:34.131258 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:48:34.131140 5378 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c9c625ce-f432-4bd9-8a6f-0a01335b886b-model-cache\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-xdmrj\" (UID: \"c9c625ce-f432-4bd9-8a6f-0a01335b886b\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-xdmrj" Oct 30 20:48:34.131453 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:48:34.131434 5378 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c9c625ce-f432-4bd9-8a6f-0a01335b886b-home\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-xdmrj\" (UID: \"c9c625ce-f432-4bd9-8a6f-0a01335b886b\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-xdmrj" Oct 30 20:48:34.131508 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:48:34.131458 5378 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c9c625ce-f432-4bd9-8a6f-0a01335b886b-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-xdmrj\" (UID: \"c9c625ce-f432-4bd9-8a6f-0a01335b886b\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-xdmrj" Oct 30 20:48:34.131544 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:48:34.131511 5378 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c9c625ce-f432-4bd9-8a6f-0a01335b886b-model-cache\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-xdmrj\" (UID: \"c9c625ce-f432-4bd9-8a6f-0a01335b886b\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-xdmrj" Oct 30 20:48:34.133439 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:48:34.133411 5378 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c9c625ce-f432-4bd9-8a6f-0a01335b886b-dshm\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-xdmrj\" (UID: \"c9c625ce-f432-4bd9-8a6f-0a01335b886b\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-xdmrj" Oct 30 20:48:34.133586 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:48:34.133547 5378 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c9c625ce-f432-4bd9-8a6f-0a01335b886b-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-xdmrj\" (UID: \"c9c625ce-f432-4bd9-8a6f-0a01335b886b\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-xdmrj" Oct 30 20:48:34.197279 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:48:34.197219 5378 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fs5zv\" (UniqueName: \"kubernetes.io/projected/c9c625ce-f432-4bd9-8a6f-0a01335b886b-kube-api-access-fs5zv\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-xdmrj\" (UID: \"c9c625ce-f432-4bd9-8a6f-0a01335b886b\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-xdmrj" Oct 30 20:48:34.237763 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:48:34.237740 5378 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-xdmrj" Oct 30 20:48:34.608138 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:48:34.608115 5378 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-cbd4f5586-xdmrj"] Oct 30 20:48:34.610158 ip-10-0-138-149 kubenswrapper[5378]: W1030 20:48:34.610131 5378 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc9c625ce_f432_4bd9_8a6f_0a01335b886b.slice/crio-390913bc5283937f2bdc39645cfa525ad32902ece9dd24f93917b8a9b77846c7 WatchSource:0}: Error finding container 390913bc5283937f2bdc39645cfa525ad32902ece9dd24f93917b8a9b77846c7: Status 404 returned error can't find the container with id 390913bc5283937f2bdc39645cfa525ad32902ece9dd24f93917b8a9b77846c7 Oct 30 20:48:35.004153 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:48:35.004075 5378 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-xdmrj" event={"ID":"c9c625ce-f432-4bd9-8a6f-0a01335b886b","Type":"ContainerStarted","Data":"390913bc5283937f2bdc39645cfa525ad32902ece9dd24f93917b8a9b77846c7"} Oct 30 20:48:35.015470 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:48:35.015451 5378 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/echo-simulator-kserve-64b6f8f96c-hjktp" Oct 30 20:48:36.009885 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:48:36.009846 5378 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-xdmrj" event={"ID":"c9c625ce-f432-4bd9-8a6f-0a01335b886b","Type":"ContainerStarted","Data":"c20483e4d61966c4d2c21158c2a7027b948f25311d615c56e18ac4842eb65635"} Oct 30 20:48:36.129581 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:48:36.129510 5378 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-xdmrj" podStartSLOduration=2.5304993319999998 podStartE2EDuration="3.129494981s" totalImagesPullingTime="598.995649ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:48:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:48:36.128767847 +0000 UTC m=+606.556881766" watchObservedRunningTime="2026-10-30 20:48:36.129494981 +0000 UTC m=+606.557608910" Oct 30 20:48:36.238410 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:48:36.238376 5378 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-xdmrj" Oct 30 20:48:37.325337 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:48:37.325305 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 20:48:39.142962 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:48:39.142928 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 20:48:40.907220 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:48:40.907188 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 20:48:44.238560 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:48:44.238521 5378 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-xdmrj" Oct 30 20:48:44.251262 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:48:44.251237 5378 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-xdmrj" Oct 30 20:48:44.794708 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:48:44.794669 5378 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-r9g59"] Oct 30 20:48:45.014728 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:48:45.014648 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 20:48:45.064048 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:48:45.063960 5378 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-xdmrj" Oct 30 20:48:48.835857 ip-10-0-138-149 kubenswrapper[5378]: E1030 20:48:48.835819 5378 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q6mx7" containerName="extractor" Oct 30 20:48:50.961306 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:48:50.961272 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 20:48:51.097769 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:48:51.097736 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 20:48:52.060808 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:48:52.060770 5378 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-769df7846c-8jkpf"] Oct 30 20:48:52.066434 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:48:52.066406 5378 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-8jkpf" Oct 30 20:48:52.204748 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:48:52.204712 5378 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0d0a80d9-4ec9-43cb-8368-15e6b0cb7b22-workload-socket\") pod \"maas-default-gateway-openshift-default-769df7846c-8jkpf\" (UID: \"0d0a80d9-4ec9-43cb-8368-15e6b0cb7b22\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-8jkpf" Oct 30 20:48:52.204905 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:48:52.204775 5378 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0d0a80d9-4ec9-43cb-8368-15e6b0cb7b22-workload-certs\") pod \"maas-default-gateway-openshift-default-769df7846c-8jkpf\" (UID: \"0d0a80d9-4ec9-43cb-8368-15e6b0cb7b22\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-8jkpf" Oct 30 20:48:52.204905 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:48:52.204854 5378 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0d0a80d9-4ec9-43cb-8368-15e6b0cb7b22-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-769df7846c-8jkpf\" (UID: \"0d0a80d9-4ec9-43cb-8368-15e6b0cb7b22\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-8jkpf" Oct 30 20:48:52.204905 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:48:52.204884 5378 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2dnhg\" (UniqueName: \"kubernetes.io/projected/0d0a80d9-4ec9-43cb-8368-15e6b0cb7b22-kube-api-access-2dnhg\") pod \"maas-default-gateway-openshift-default-769df7846c-8jkpf\" (UID: \"0d0a80d9-4ec9-43cb-8368-15e6b0cb7b22\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-8jkpf" Oct 30 20:48:52.205050 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:48:52.204921 5378 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0d0a80d9-4ec9-43cb-8368-15e6b0cb7b22-istio-envoy\") pod \"maas-default-gateway-openshift-default-769df7846c-8jkpf\" (UID: \"0d0a80d9-4ec9-43cb-8368-15e6b0cb7b22\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-8jkpf" Oct 30 20:48:52.205050 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:48:52.204967 5378 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0d0a80d9-4ec9-43cb-8368-15e6b0cb7b22-istio-token\") pod \"maas-default-gateway-openshift-default-769df7846c-8jkpf\" (UID: \"0d0a80d9-4ec9-43cb-8368-15e6b0cb7b22\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-8jkpf" Oct 30 20:48:52.205050 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:48:52.204991 5378 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0d0a80d9-4ec9-43cb-8368-15e6b0cb7b22-credential-socket\") pod \"maas-default-gateway-openshift-default-769df7846c-8jkpf\" (UID: \"0d0a80d9-4ec9-43cb-8368-15e6b0cb7b22\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-8jkpf" Oct 30 20:48:52.205205 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:48:52.205100 5378 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0d0a80d9-4ec9-43cb-8368-15e6b0cb7b22-istio-podinfo\") pod \"maas-default-gateway-openshift-default-769df7846c-8jkpf\" (UID: \"0d0a80d9-4ec9-43cb-8368-15e6b0cb7b22\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-8jkpf" Oct 30 20:48:52.205205 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:48:52.205131 5378 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0d0a80d9-4ec9-43cb-8368-15e6b0cb7b22-istio-data\") pod \"maas-default-gateway-openshift-default-769df7846c-8jkpf\" (UID: \"0d0a80d9-4ec9-43cb-8368-15e6b0cb7b22\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-8jkpf" Oct 30 20:48:52.239812 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:48:52.239787 5378 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-qbqvq\"" Oct 30 20:48:52.286518 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:48:52.286478 5378 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-769df7846c-8jkpf"] Oct 30 20:48:52.306602 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:48:52.306559 5378 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0d0a80d9-4ec9-43cb-8368-15e6b0cb7b22-istio-podinfo\") pod \"maas-default-gateway-openshift-default-769df7846c-8jkpf\" (UID: \"0d0a80d9-4ec9-43cb-8368-15e6b0cb7b22\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-8jkpf" Oct 30 20:48:52.306735 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:48:52.306613 5378 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0d0a80d9-4ec9-43cb-8368-15e6b0cb7b22-istio-data\") pod \"maas-default-gateway-openshift-default-769df7846c-8jkpf\" (UID: \"0d0a80d9-4ec9-43cb-8368-15e6b0cb7b22\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-8jkpf" Oct 30 20:48:52.306735 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:48:52.306644 5378 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0d0a80d9-4ec9-43cb-8368-15e6b0cb7b22-workload-socket\") pod \"maas-default-gateway-openshift-default-769df7846c-8jkpf\" (UID: \"0d0a80d9-4ec9-43cb-8368-15e6b0cb7b22\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-8jkpf" Oct 30 20:48:52.306735 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:48:52.306684 5378 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0d0a80d9-4ec9-43cb-8368-15e6b0cb7b22-workload-certs\") pod \"maas-default-gateway-openshift-default-769df7846c-8jkpf\" (UID: \"0d0a80d9-4ec9-43cb-8368-15e6b0cb7b22\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-8jkpf" Oct 30 20:48:52.306735 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:48:52.306703 5378 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0d0a80d9-4ec9-43cb-8368-15e6b0cb7b22-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-769df7846c-8jkpf\" (UID: \"0d0a80d9-4ec9-43cb-8368-15e6b0cb7b22\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-8jkpf" Oct 30 20:48:52.306941 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:48:52.306877 5378 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2dnhg\" (UniqueName: \"kubernetes.io/projected/0d0a80d9-4ec9-43cb-8368-15e6b0cb7b22-kube-api-access-2dnhg\") pod \"maas-default-gateway-openshift-default-769df7846c-8jkpf\" (UID: \"0d0a80d9-4ec9-43cb-8368-15e6b0cb7b22\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-8jkpf" Oct 30 20:48:52.306998 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:48:52.306938 5378 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0d0a80d9-4ec9-43cb-8368-15e6b0cb7b22-istio-envoy\") pod \"maas-default-gateway-openshift-default-769df7846c-8jkpf\" (UID: \"0d0a80d9-4ec9-43cb-8368-15e6b0cb7b22\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-8jkpf" Oct 30 20:48:52.306998 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:48:52.306990 5378 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0d0a80d9-4ec9-43cb-8368-15e6b0cb7b22-istio-token\") pod \"maas-default-gateway-openshift-default-769df7846c-8jkpf\" (UID: \"0d0a80d9-4ec9-43cb-8368-15e6b0cb7b22\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-8jkpf" Oct 30 20:48:52.307098 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:48:52.307031 5378 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0d0a80d9-4ec9-43cb-8368-15e6b0cb7b22-credential-socket\") pod \"maas-default-gateway-openshift-default-769df7846c-8jkpf\" (UID: \"0d0a80d9-4ec9-43cb-8368-15e6b0cb7b22\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-8jkpf" Oct 30 20:48:52.307098 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:48:52.307047 5378 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0d0a80d9-4ec9-43cb-8368-15e6b0cb7b22-istio-data\") pod \"maas-default-gateway-openshift-default-769df7846c-8jkpf\" (UID: \"0d0a80d9-4ec9-43cb-8368-15e6b0cb7b22\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-8jkpf" Oct 30 20:48:52.307195 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:48:52.307113 5378 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0d0a80d9-4ec9-43cb-8368-15e6b0cb7b22-workload-socket\") pod \"maas-default-gateway-openshift-default-769df7846c-8jkpf\" (UID: \"0d0a80d9-4ec9-43cb-8368-15e6b0cb7b22\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-8jkpf" Oct 30 20:48:52.307195 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:48:52.307166 5378 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0d0a80d9-4ec9-43cb-8368-15e6b0cb7b22-workload-certs\") pod \"maas-default-gateway-openshift-default-769df7846c-8jkpf\" (UID: \"0d0a80d9-4ec9-43cb-8368-15e6b0cb7b22\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-8jkpf" Oct 30 20:48:52.307519 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:48:52.307495 5378 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0d0a80d9-4ec9-43cb-8368-15e6b0cb7b22-credential-socket\") pod \"maas-default-gateway-openshift-default-769df7846c-8jkpf\" (UID: \"0d0a80d9-4ec9-43cb-8368-15e6b0cb7b22\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-8jkpf" Oct 30 20:48:52.307643 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:48:52.307593 5378 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0d0a80d9-4ec9-43cb-8368-15e6b0cb7b22-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-769df7846c-8jkpf\" (UID: \"0d0a80d9-4ec9-43cb-8368-15e6b0cb7b22\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-8jkpf" Oct 30 20:48:52.309084 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:48:52.309059 5378 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0d0a80d9-4ec9-43cb-8368-15e6b0cb7b22-istio-envoy\") pod \"maas-default-gateway-openshift-default-769df7846c-8jkpf\" (UID: \"0d0a80d9-4ec9-43cb-8368-15e6b0cb7b22\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-8jkpf" Oct 30 20:48:52.309183 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:48:52.309096 5378 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0d0a80d9-4ec9-43cb-8368-15e6b0cb7b22-istio-podinfo\") pod \"maas-default-gateway-openshift-default-769df7846c-8jkpf\" (UID: \"0d0a80d9-4ec9-43cb-8368-15e6b0cb7b22\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-8jkpf" Oct 30 20:48:52.367368 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:48:52.367336 5378 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2dnhg\" (UniqueName: \"kubernetes.io/projected/0d0a80d9-4ec9-43cb-8368-15e6b0cb7b22-kube-api-access-2dnhg\") pod \"maas-default-gateway-openshift-default-769df7846c-8jkpf\" (UID: \"0d0a80d9-4ec9-43cb-8368-15e6b0cb7b22\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-8jkpf" Oct 30 20:48:52.367480 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:48:52.367423 5378 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0d0a80d9-4ec9-43cb-8368-15e6b0cb7b22-istio-token\") pod \"maas-default-gateway-openshift-default-769df7846c-8jkpf\" (UID: \"0d0a80d9-4ec9-43cb-8368-15e6b0cb7b22\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-8jkpf" Oct 30 20:48:52.379867 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:48:52.379844 5378 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-8jkpf" Oct 30 20:48:52.804099 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:48:52.804016 5378 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-769df7846c-8jkpf"] Oct 30 20:48:52.806906 ip-10-0-138-149 kubenswrapper[5378]: W1030 20:48:52.806866 5378 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0d0a80d9_4ec9_43cb_8368_15e6b0cb7b22.slice/crio-0e076fc52f98d805515a39fcfcd0d341ef591022ddeddac40efe4cc3730a4c77 WatchSource:0}: Error finding container 0e076fc52f98d805515a39fcfcd0d341ef591022ddeddac40efe4cc3730a4c77: Status 404 returned error can't find the container with id 0e076fc52f98d805515a39fcfcd0d341ef591022ddeddac40efe4cc3730a4c77 Oct 30 20:48:53.081423 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:48:53.081338 5378 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-8jkpf" event={"ID":"0d0a80d9-4ec9-43cb-8368-15e6b0cb7b22","Type":"ContainerStarted","Data":"0e076fc52f98d805515a39fcfcd0d341ef591022ddeddac40efe4cc3730a4c77"} Oct 30 20:48:56.187779 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:48:56.187742 5378 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-6d699dfc88-qgrhg"] Oct 30 20:48:56.195755 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:48:56.195727 5378 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6d699dfc88-qgrhg" Oct 30 20:48:56.255370 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:48:56.255341 5378 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-6d699dfc88-qgrhg"] Oct 30 20:48:56.344131 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:48:56.344084 5378 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gbd8m\" (UniqueName: \"kubernetes.io/projected/7f07de12-d7f7-42db-a68f-d65bb9e84c48-kube-api-access-gbd8m\") pod \"authorino-6d699dfc88-qgrhg\" (UID: \"7f07de12-d7f7-42db-a68f-d65bb9e84c48\") " pod="kuadrant-system/authorino-6d699dfc88-qgrhg" Oct 30 20:48:56.344306 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:48:56.344168 5378 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/7f07de12-d7f7-42db-a68f-d65bb9e84c48-tls-cert\") pod \"authorino-6d699dfc88-qgrhg\" (UID: \"7f07de12-d7f7-42db-a68f-d65bb9e84c48\") " pod="kuadrant-system/authorino-6d699dfc88-qgrhg" Oct 30 20:48:56.402937 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:48:56.402897 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 20:48:56.416918 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:48:56.416883 5378 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/ai-gateway-controller-5f6dd4f69f-sff92"] Oct 30 20:48:56.423461 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:48:56.423438 5378 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/ai-gateway-controller-5f6dd4f69f-sff92" Oct 30 20:48:56.444868 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:48:56.444803 5378 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/7f07de12-d7f7-42db-a68f-d65bb9e84c48-tls-cert\") pod \"authorino-6d699dfc88-qgrhg\" (UID: \"7f07de12-d7f7-42db-a68f-d65bb9e84c48\") " pod="kuadrant-system/authorino-6d699dfc88-qgrhg" Oct 30 20:48:56.444982 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:48:56.444908 5378 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gbd8m\" (UniqueName: \"kubernetes.io/projected/7f07de12-d7f7-42db-a68f-d65bb9e84c48-kube-api-access-gbd8m\") pod \"authorino-6d699dfc88-qgrhg\" (UID: \"7f07de12-d7f7-42db-a68f-d65bb9e84c48\") " pod="kuadrant-system/authorino-6d699dfc88-qgrhg" Oct 30 20:48:56.447556 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:48:56.447523 5378 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/7f07de12-d7f7-42db-a68f-d65bb9e84c48-tls-cert\") pod \"authorino-6d699dfc88-qgrhg\" (UID: \"7f07de12-d7f7-42db-a68f-d65bb9e84c48\") " pod="kuadrant-system/authorino-6d699dfc88-qgrhg" Oct 30 20:48:56.481340 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:48:56.481316 5378 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"ai-gateway-controller-parameters\"" Oct 30 20:48:56.482745 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:48:56.482711 5378 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"ai-gateway-controller-metrics-tls\"" Oct 30 20:48:56.482925 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:48:56.482907 5378 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"ai-gateway-controller-dockercfg-46fbq\"" Oct 30 20:48:56.494346 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:48:56.494316 5378 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/ai-gateway-controller-5f6dd4f69f-sff92"] Oct 30 20:48:56.522011 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:48:56.521980 5378 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gbd8m\" (UniqueName: \"kubernetes.io/projected/7f07de12-d7f7-42db-a68f-d65bb9e84c48-kube-api-access-gbd8m\") pod \"authorino-6d699dfc88-qgrhg\" (UID: \"7f07de12-d7f7-42db-a68f-d65bb9e84c48\") " pod="kuadrant-system/authorino-6d699dfc88-qgrhg" Oct 30 20:48:56.545892 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:48:56.545861 5378 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f0bacb9b-8d64-41dc-98e9-faa1e4f83855-metrics-certs\") pod \"ai-gateway-controller-5f6dd4f69f-sff92\" (UID: \"f0bacb9b-8d64-41dc-98e9-faa1e4f83855\") " pod="opendatahub/ai-gateway-controller-5f6dd4f69f-sff92" Oct 30 20:48:56.546024 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:48:56.545894 5378 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dbcf7\" (UniqueName: \"kubernetes.io/projected/f0bacb9b-8d64-41dc-98e9-faa1e4f83855-kube-api-access-dbcf7\") pod \"ai-gateway-controller-5f6dd4f69f-sff92\" (UID: \"f0bacb9b-8d64-41dc-98e9-faa1e4f83855\") " pod="opendatahub/ai-gateway-controller-5f6dd4f69f-sff92" Oct 30 20:48:56.591316 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:48:56.591262 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 20:48:56.620675 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:48:56.620647 5378 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 20:48:56.620757 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:48:56.620729 5378 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 20:48:56.620794 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:48:56.620755 5378 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 20:48:56.646825 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:48:56.646797 5378 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f0bacb9b-8d64-41dc-98e9-faa1e4f83855-metrics-certs\") pod \"ai-gateway-controller-5f6dd4f69f-sff92\" (UID: \"f0bacb9b-8d64-41dc-98e9-faa1e4f83855\") " pod="opendatahub/ai-gateway-controller-5f6dd4f69f-sff92" Oct 30 20:48:56.646932 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:48:56.646832 5378 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dbcf7\" (UniqueName: \"kubernetes.io/projected/f0bacb9b-8d64-41dc-98e9-faa1e4f83855-kube-api-access-dbcf7\") pod \"ai-gateway-controller-5f6dd4f69f-sff92\" (UID: \"f0bacb9b-8d64-41dc-98e9-faa1e4f83855\") " pod="opendatahub/ai-gateway-controller-5f6dd4f69f-sff92" Oct 30 20:48:56.649139 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:48:56.649121 5378 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f0bacb9b-8d64-41dc-98e9-faa1e4f83855-metrics-certs\") pod \"ai-gateway-controller-5f6dd4f69f-sff92\" (UID: \"f0bacb9b-8d64-41dc-98e9-faa1e4f83855\") " pod="opendatahub/ai-gateway-controller-5f6dd4f69f-sff92" Oct 30 20:48:56.705384 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:48:56.705327 5378 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dbcf7\" (UniqueName: \"kubernetes.io/projected/f0bacb9b-8d64-41dc-98e9-faa1e4f83855-kube-api-access-dbcf7\") pod \"ai-gateway-controller-5f6dd4f69f-sff92\" (UID: \"f0bacb9b-8d64-41dc-98e9-faa1e4f83855\") " pod="opendatahub/ai-gateway-controller-5f6dd4f69f-sff92" Oct 30 20:48:56.738917 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:48:56.738885 5378 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/ai-gateway-controller-5f6dd4f69f-sff92" Oct 30 20:48:56.809412 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:48:56.809382 5378 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6d699dfc88-qgrhg" Oct 30 20:48:57.100234 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:48:57.100201 5378 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-8jkpf" event={"ID":"0d0a80d9-4ec9-43cb-8368-15e6b0cb7b22","Type":"ContainerStarted","Data":"683bd8651a4c13ca27c4d4126f9342bf1f02e18227d22e733dae96c651abc3d9"} Oct 30 20:48:57.115280 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:48:57.115255 5378 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/ai-gateway-controller-5f6dd4f69f-sff92"] Oct 30 20:48:57.117662 ip-10-0-138-149 kubenswrapper[5378]: W1030 20:48:57.117637 5378 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf0bacb9b_8d64_41dc_98e9_faa1e4f83855.slice/crio-d8706d765432a59e1f414c1c38169e4530496c3b725180bcc36f04f88413736a WatchSource:0}: Error finding container d8706d765432a59e1f414c1c38169e4530496c3b725180bcc36f04f88413736a: Status 404 returned error can't find the container with id d8706d765432a59e1f414c1c38169e4530496c3b725180bcc36f04f88413736a Oct 30 20:48:57.215684 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:48:57.215651 5378 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-6d699dfc88-qgrhg"] Oct 30 20:48:57.219130 ip-10-0-138-149 kubenswrapper[5378]: W1030 20:48:57.219102 5378 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod7f07de12_d7f7_42db_a68f_d65bb9e84c48.slice/crio-e9ba6e127a584d16f903b96dd809c4d69aefe953098b8e2e0e4d11e7b3f23d03 WatchSource:0}: Error finding container e9ba6e127a584d16f903b96dd809c4d69aefe953098b8e2e0e4d11e7b3f23d03: Status 404 returned error can't find the container with id e9ba6e127a584d16f903b96dd809c4d69aefe953098b8e2e0e4d11e7b3f23d03 Oct 30 20:48:57.244047 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:48:57.244007 5378 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-8jkpf" podStartSLOduration=1.433129721 podStartE2EDuration="5.243994548s" totalImagesPullingTime="3.810864827s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:48:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:48:57.24276986 +0000 UTC m=+627.670883782" watchObservedRunningTime="2026-10-30 20:48:57.243994548 +0000 UTC m=+627.672108467" Oct 30 20:48:57.380695 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:48:57.380606 5378 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-8jkpf" Oct 30 20:48:57.382110 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:48:57.382080 5378 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-769df7846c-8jkpf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.35:15021/healthz/ready\": dial tcp 10.133.0.35:15021: connect: connection refused" start-of-body= Oct 30 20:48:57.382248 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:48:57.382124 5378 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-8jkpf" podUID="0d0a80d9-4ec9-43cb-8368-15e6b0cb7b22" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.35:15021/healthz/ready\": dial tcp 10.133.0.35:15021: connect: connection refused" Oct 30 20:48:58.107173 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:48:58.107132 5378 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-5f6dd4f69f-sff92" event={"ID":"f0bacb9b-8d64-41dc-98e9-faa1e4f83855","Type":"ContainerStarted","Data":"d8706d765432a59e1f414c1c38169e4530496c3b725180bcc36f04f88413736a"} Oct 30 20:48:58.108526 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:48:58.108495 5378 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-6d699dfc88-qgrhg" event={"ID":"7f07de12-d7f7-42db-a68f-d65bb9e84c48","Type":"ContainerStarted","Data":"e9ba6e127a584d16f903b96dd809c4d69aefe953098b8e2e0e4d11e7b3f23d03"} Oct 30 20:48:58.381048 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:48:58.380515 5378 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-769df7846c-8jkpf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.35:15021/healthz/ready\": dial tcp 10.133.0.35:15021: connect: connection refused" start-of-body= Oct 30 20:48:58.381048 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:48:58.380607 5378 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-8jkpf" podUID="0d0a80d9-4ec9-43cb-8368-15e6b0cb7b22" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.35:15021/healthz/ready\": dial tcp 10.133.0.35:15021: connect: connection refused" Oct 30 20:48:59.115461 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:48:59.115425 5378 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-6d699dfc88-qgrhg" event={"ID":"7f07de12-d7f7-42db-a68f-d65bb9e84c48","Type":"ContainerStarted","Data":"d87d7778979e4cb99d0aadc370492207db02d09f9553f718e6ecb286c285cce9"} Oct 30 20:48:59.292705 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:48:59.292398 5378 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/authorino-6d699dfc88-qgrhg" podStartSLOduration=1.7820565350000002 podStartE2EDuration="3.29237698s" totalImagesPullingTime="1.510320445s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:48:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:48:59.242101488 +0000 UTC m=+629.670215407" watchObservedRunningTime="2026-10-30 20:48:59.29237698 +0000 UTC m=+629.720490912" Oct 30 20:48:59.293932 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:48:59.293907 5378 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-7965b594c5-65jjp"] Oct 30 20:48:59.294146 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:48:59.294110 5378 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/authorino-7965b594c5-65jjp" podUID="75843e1d-767b-47b2-9dbb-b18a6025fe62" containerName="authorino" containerID="cri-o://ea2a98804b892b97ca938fc88f74ef0ff7e900875f1594462a0dfe3efb5d2ca2" gracePeriod=30 Oct 30 20:48:59.381109 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:48:59.381026 5378 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-769df7846c-8jkpf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.35:15021/healthz/ready\": dial tcp 10.133.0.35:15021: connect: connection refused" start-of-body= Oct 30 20:48:59.381109 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:48:59.381081 5378 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-8jkpf" podUID="0d0a80d9-4ec9-43cb-8368-15e6b0cb7b22" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.35:15021/healthz/ready\": dial tcp 10.133.0.35:15021: connect: connection refused" Oct 30 20:48:59.591592 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:48:59.591553 5378 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7965b594c5-65jjp" Oct 30 20:48:59.690031 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:48:59.689951 5378 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/75843e1d-767b-47b2-9dbb-b18a6025fe62-kube-api-access-wv7nn\" (UniqueName: \"kubernetes.io/projected/75843e1d-767b-47b2-9dbb-b18a6025fe62-kube-api-access-wv7nn\") pod \"75843e1d-767b-47b2-9dbb-b18a6025fe62\" (UID: \"75843e1d-767b-47b2-9dbb-b18a6025fe62\") " Oct 30 20:48:59.690031 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:48:59.689993 5378 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/75843e1d-767b-47b2-9dbb-b18a6025fe62-tls-cert\" (UniqueName: \"kubernetes.io/secret/75843e1d-767b-47b2-9dbb-b18a6025fe62-tls-cert\") pod \"75843e1d-767b-47b2-9dbb-b18a6025fe62\" (UID: \"75843e1d-767b-47b2-9dbb-b18a6025fe62\") " Oct 30 20:48:59.692108 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:48:59.692073 5378 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/75843e1d-767b-47b2-9dbb-b18a6025fe62-kube-api-access-wv7nn" pod "75843e1d-767b-47b2-9dbb-b18a6025fe62" (UID: "75843e1d-767b-47b2-9dbb-b18a6025fe62"). InnerVolumeSpecName "kube-api-access-wv7nn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:48:59.700404 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:48:59.700377 5378 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/75843e1d-767b-47b2-9dbb-b18a6025fe62-tls-cert" pod "75843e1d-767b-47b2-9dbb-b18a6025fe62" (UID: "75843e1d-767b-47b2-9dbb-b18a6025fe62"). InnerVolumeSpecName "tls-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 20:48:59.791635 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:48:59.791608 5378 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wv7nn\" (UniqueName: \"kubernetes.io/projected/75843e1d-767b-47b2-9dbb-b18a6025fe62-kube-api-access-wv7nn\") on node \"ip-10-0-138-149.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:48:59.791635 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:48:59.791630 5378 reconciler_common.go:299] "Volume detached for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/75843e1d-767b-47b2-9dbb-b18a6025fe62-tls-cert\") on node \"ip-10-0-138-149.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:49:00.121409 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:49:00.121369 5378 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-5f6dd4f69f-sff92" event={"ID":"f0bacb9b-8d64-41dc-98e9-faa1e4f83855","Type":"ContainerStarted","Data":"a7595b0a164c95ef7e327ad6f12ed3b48ef7f7aef94be59fe5c75c0073aea0a0"} Oct 30 20:49:00.121616 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:49:00.121461 5378 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/ai-gateway-controller-5f6dd4f69f-sff92" Oct 30 20:49:00.122636 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:49:00.122604 5378 generic.go:356] "Generic (PLEG): container finished" podID="75843e1d-767b-47b2-9dbb-b18a6025fe62" containerID="ea2a98804b892b97ca938fc88f74ef0ff7e900875f1594462a0dfe3efb5d2ca2" exitCode=0 Oct 30 20:49:00.122736 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:49:00.122652 5378 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7965b594c5-65jjp" Oct 30 20:49:00.122736 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:49:00.122688 5378 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7965b594c5-65jjp" event={"ID":"75843e1d-767b-47b2-9dbb-b18a6025fe62","Type":"ContainerDied","Data":"ea2a98804b892b97ca938fc88f74ef0ff7e900875f1594462a0dfe3efb5d2ca2"} Oct 30 20:49:00.122736 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:49:00.122731 5378 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7965b594c5-65jjp" event={"ID":"75843e1d-767b-47b2-9dbb-b18a6025fe62","Type":"ContainerDied","Data":"beae0667ec9b806d3592c8736c274dc0b3c572ba37421f58c93d9c219e8f05ea"} Oct 30 20:49:00.122850 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:49:00.122755 5378 scope.go:122] "RemoveContainer" containerID="ea2a98804b892b97ca938fc88f74ef0ff7e900875f1594462a0dfe3efb5d2ca2" Oct 30 20:49:00.130990 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:49:00.130970 5378 scope.go:122] "RemoveContainer" containerID="ea2a98804b892b97ca938fc88f74ef0ff7e900875f1594462a0dfe3efb5d2ca2" Oct 30 20:49:00.131258 ip-10-0-138-149 kubenswrapper[5378]: E1030 20:49:00.131236 5378 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ea2a98804b892b97ca938fc88f74ef0ff7e900875f1594462a0dfe3efb5d2ca2\": container with ID starting with ea2a98804b892b97ca938fc88f74ef0ff7e900875f1594462a0dfe3efb5d2ca2 not found: ID does not exist" containerID="ea2a98804b892b97ca938fc88f74ef0ff7e900875f1594462a0dfe3efb5d2ca2" Oct 30 20:49:00.131337 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:49:00.131267 5378 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ea2a98804b892b97ca938fc88f74ef0ff7e900875f1594462a0dfe3efb5d2ca2"} err="failed to get container status \"ea2a98804b892b97ca938fc88f74ef0ff7e900875f1594462a0dfe3efb5d2ca2\": rpc error: code = NotFound desc = could not find container \"ea2a98804b892b97ca938fc88f74ef0ff7e900875f1594462a0dfe3efb5d2ca2\": container with ID starting with ea2a98804b892b97ca938fc88f74ef0ff7e900875f1594462a0dfe3efb5d2ca2 not found: ID does not exist" Oct 30 20:49:00.242818 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:49:00.242748 5378 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/ai-gateway-controller-5f6dd4f69f-sff92" podStartSLOduration=2.16669948 podStartE2EDuration="4.242729096s" totalImagesPullingTime="2.076029616s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:48:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:49:00.241459939 +0000 UTC m=+630.669573859" watchObservedRunningTime="2026-10-30 20:49:00.242729096 +0000 UTC m=+630.670843015" Oct 30 20:49:00.374639 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:49:00.360905 5378 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-7965b594c5-65jjp"] Oct 30 20:49:00.374639 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:49:00.363343 5378 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-7965b594c5-65jjp"] Oct 30 20:49:00.381956 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:49:00.381920 5378 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-769df7846c-8jkpf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.35:15021/healthz/ready\": dial tcp 10.133.0.35:15021: connect: connection refused" start-of-body= Oct 30 20:49:00.382400 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:49:00.381980 5378 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-8jkpf" podUID="0d0a80d9-4ec9-43cb-8368-15e6b0cb7b22" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.35:15021/healthz/ready\": dial tcp 10.133.0.35:15021: connect: connection refused" Oct 30 20:49:01.384022 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:49:01.383981 5378 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-769df7846c-8jkpf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.35:15021/healthz/ready\": dial tcp 10.133.0.35:15021: connect: connection refused" start-of-body= Oct 30 20:49:01.384449 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:49:01.384053 5378 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-8jkpf" podUID="0d0a80d9-4ec9-43cb-8368-15e6b0cb7b22" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.35:15021/healthz/ready\": dial tcp 10.133.0.35:15021: connect: connection refused" Oct 30 20:49:01.740117 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:49:01.740034 5378 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="75843e1d-767b-47b2-9dbb-b18a6025fe62" path="/var/lib/kubelet/pods/75843e1d-767b-47b2-9dbb-b18a6025fe62/volumes" Oct 30 20:49:02.380717 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:49:02.380688 5378 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-769df7846c-8jkpf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.35:15021/healthz/ready\": dial tcp 10.133.0.35:15021: connect: connection refused" start-of-body= Oct 30 20:49:02.380947 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:49:02.380926 5378 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-8jkpf" podUID="0d0a80d9-4ec9-43cb-8368-15e6b0cb7b22" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.35:15021/healthz/ready\": dial tcp 10.133.0.35:15021: connect: connection refused" Oct 30 20:49:03.332990 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:49:03.332929 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 20:49:03.380998 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:49:03.380961 5378 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-769df7846c-8jkpf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.35:15021/healthz/ready\": dial tcp 10.133.0.35:15021: connect: connection refused" start-of-body= Oct 30 20:49:03.381178 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:49:03.381026 5378 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-8jkpf" podUID="0d0a80d9-4ec9-43cb-8368-15e6b0cb7b22" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.35:15021/healthz/ready\": dial tcp 10.133.0.35:15021: connect: connection refused" Oct 30 20:49:03.514699 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:49:03.514666 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 20:49:03.760230 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:49:03.760200 5378 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-7b98bb678c-7l6vk"] Oct 30 20:49:03.760541 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:49:03.760530 5378 cpu_manager.go:414] "containerMap: removing container" podUID="75843e1d-767b-47b2-9dbb-b18a6025fe62" containerName="authorino" Oct 30 20:49:03.760609 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:49:03.760542 5378 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="75843e1d-767b-47b2-9dbb-b18a6025fe62" containerName="authorino" Oct 30 20:49:03.760663 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:49:03.760653 5378 memory_manager.go:365] "RemoveStaleState removing state" podUID="75843e1d-767b-47b2-9dbb-b18a6025fe62" containerName="authorino" Oct 30 20:49:03.764943 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:49:03.764921 5378 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-7b98bb678c-7l6vk" Oct 30 20:49:03.818042 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:49:03.818005 5378 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins\"" Oct 30 20:49:03.837050 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:49:03.837006 5378 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rrlkp\" (UniqueName: \"kubernetes.io/projected/4c52be8c-3199-4691-a6ea-4f98842987c4-kube-api-access-rrlkp\") pod \"payload-pre-processing-7b98bb678c-7l6vk\" (UID: \"4c52be8c-3199-4691-a6ea-4f98842987c4\") " pod="openshift-ingress/payload-pre-processing-7b98bb678c-7l6vk" Oct 30 20:49:03.837262 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:49:03.837082 5378 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4c52be8c-3199-4691-a6ea-4f98842987c4-plugins-config-volume\") pod \"payload-pre-processing-7b98bb678c-7l6vk\" (UID: \"4c52be8c-3199-4691-a6ea-4f98842987c4\") " pod="openshift-ingress/payload-pre-processing-7b98bb678c-7l6vk" Oct 30 20:49:03.868546 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:49:03.868509 5378 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-dockercfg-lztv5\"" Oct 30 20:49:03.869684 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:49:03.869665 5378 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-7b98bb678c-7l6vk"] Oct 30 20:49:03.920215 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:49:03.920185 5378 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-5f78687d5b-2r7nt"] Oct 30 20:49:03.923514 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:49:03.923491 5378 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-5f78687d5b-2r7nt" Oct 30 20:49:03.938133 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:49:03.938110 5378 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rrlkp\" (UniqueName: \"kubernetes.io/projected/4c52be8c-3199-4691-a6ea-4f98842987c4-kube-api-access-rrlkp\") pod \"payload-pre-processing-7b98bb678c-7l6vk\" (UID: \"4c52be8c-3199-4691-a6ea-4f98842987c4\") " pod="openshift-ingress/payload-pre-processing-7b98bb678c-7l6vk" Oct 30 20:49:03.938264 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:49:03.938140 5378 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4c52be8c-3199-4691-a6ea-4f98842987c4-plugins-config-volume\") pod \"payload-pre-processing-7b98bb678c-7l6vk\" (UID: \"4c52be8c-3199-4691-a6ea-4f98842987c4\") " pod="openshift-ingress/payload-pre-processing-7b98bb678c-7l6vk" Oct 30 20:49:03.938840 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:49:03.938817 5378 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4c52be8c-3199-4691-a6ea-4f98842987c4-plugins-config-volume\") pod \"payload-pre-processing-7b98bb678c-7l6vk\" (UID: \"4c52be8c-3199-4691-a6ea-4f98842987c4\") " pod="openshift-ingress/payload-pre-processing-7b98bb678c-7l6vk" Oct 30 20:49:03.958461 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:49:03.958427 5378 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-5f78687d5b-2r7nt"] Oct 30 20:49:03.999712 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:49:03.999676 5378 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rrlkp\" (UniqueName: \"kubernetes.io/projected/4c52be8c-3199-4691-a6ea-4f98842987c4-kube-api-access-rrlkp\") pod \"payload-pre-processing-7b98bb678c-7l6vk\" (UID: \"4c52be8c-3199-4691-a6ea-4f98842987c4\") " pod="openshift-ingress/payload-pre-processing-7b98bb678c-7l6vk" Oct 30 20:49:04.038956 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:49:04.038869 5378 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tb6c2\" (UniqueName: \"kubernetes.io/projected/273afb91-cad1-42d5-89c7-20d2521e082e-kube-api-access-tb6c2\") pod \"payload-processing-5f78687d5b-2r7nt\" (UID: \"273afb91-cad1-42d5-89c7-20d2521e082e\") " pod="openshift-ingress/payload-processing-5f78687d5b-2r7nt" Oct 30 20:49:04.038956 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:49:04.038913 5378 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/273afb91-cad1-42d5-89c7-20d2521e082e-plugins-config-volume\") pod \"payload-processing-5f78687d5b-2r7nt\" (UID: \"273afb91-cad1-42d5-89c7-20d2521e082e\") " pod="openshift-ingress/payload-processing-5f78687d5b-2r7nt" Oct 30 20:49:04.077090 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:49:04.077060 5378 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-7b98bb678c-7l6vk" Oct 30 20:49:04.143408 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:49:04.142199 5378 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tb6c2\" (UniqueName: \"kubernetes.io/projected/273afb91-cad1-42d5-89c7-20d2521e082e-kube-api-access-tb6c2\") pod \"payload-processing-5f78687d5b-2r7nt\" (UID: \"273afb91-cad1-42d5-89c7-20d2521e082e\") " pod="openshift-ingress/payload-processing-5f78687d5b-2r7nt" Oct 30 20:49:04.143408 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:49:04.142263 5378 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/273afb91-cad1-42d5-89c7-20d2521e082e-plugins-config-volume\") pod \"payload-processing-5f78687d5b-2r7nt\" (UID: \"273afb91-cad1-42d5-89c7-20d2521e082e\") " pod="openshift-ingress/payload-processing-5f78687d5b-2r7nt" Oct 30 20:49:04.143408 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:49:04.143196 5378 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/273afb91-cad1-42d5-89c7-20d2521e082e-plugins-config-volume\") pod \"payload-processing-5f78687d5b-2r7nt\" (UID: \"273afb91-cad1-42d5-89c7-20d2521e082e\") " pod="openshift-ingress/payload-processing-5f78687d5b-2r7nt" Oct 30 20:49:04.206478 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:49:04.206443 5378 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tb6c2\" (UniqueName: \"kubernetes.io/projected/273afb91-cad1-42d5-89c7-20d2521e082e-kube-api-access-tb6c2\") pod \"payload-processing-5f78687d5b-2r7nt\" (UID: \"273afb91-cad1-42d5-89c7-20d2521e082e\") " pod="openshift-ingress/payload-processing-5f78687d5b-2r7nt" Oct 30 20:49:04.237236 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:49:04.237202 5378 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-5f78687d5b-2r7nt" Oct 30 20:49:04.458218 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:49:04.458193 5378 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-7b98bb678c-7l6vk"] Oct 30 20:49:04.461848 ip-10-0-138-149 kubenswrapper[5378]: W1030 20:49:04.461823 5378 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4c52be8c_3199_4691_a6ea_4f98842987c4.slice/crio-002db3c53fb030c17170b2c5cfabe95787340175c0ec81720211601dfe7787d6 WatchSource:0}: Error finding container 002db3c53fb030c17170b2c5cfabe95787340175c0ec81720211601dfe7787d6: Status 404 returned error can't find the container with id 002db3c53fb030c17170b2c5cfabe95787340175c0ec81720211601dfe7787d6 Oct 30 20:49:04.629512 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:49:04.629485 5378 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-5f78687d5b-2r7nt"] Oct 30 20:49:04.632278 ip-10-0-138-149 kubenswrapper[5378]: W1030 20:49:04.632247 5378 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod273afb91_cad1_42d5_89c7_20d2521e082e.slice/crio-cce2077abf9463ffc10053d391bc31532776d422b2e3bbfc73f81037a344a3ad WatchSource:0}: Error finding container cce2077abf9463ffc10053d391bc31532776d422b2e3bbfc73f81037a344a3ad: Status 404 returned error can't find the container with id cce2077abf9463ffc10053d391bc31532776d422b2e3bbfc73f81037a344a3ad Oct 30 20:49:05.120901 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:49:05.120855 5378 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-8jkpf" Oct 30 20:49:05.121269 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:49:05.121241 5378 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-8jkpf" Oct 30 20:49:05.122150 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:49:05.122127 5378 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-8jkpf" Oct 30 20:49:05.167737 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:49:05.167701 5378 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-7b98bb678c-7l6vk" event={"ID":"4c52be8c-3199-4691-a6ea-4f98842987c4","Type":"ContainerStarted","Data":"002db3c53fb030c17170b2c5cfabe95787340175c0ec81720211601dfe7787d6"} Oct 30 20:49:05.168970 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:49:05.168949 5378 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-5f78687d5b-2r7nt" event={"ID":"273afb91-cad1-42d5-89c7-20d2521e082e","Type":"ContainerStarted","Data":"cce2077abf9463ffc10053d391bc31532776d422b2e3bbfc73f81037a344a3ad"} Oct 30 20:49:07.176590 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:49:07.157251 5378 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-769df7846c-xghmf"] Oct 30 20:49:07.176590 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:49:07.169833 5378 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-xghmf" Oct 30 20:49:07.276587 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:49:07.274181 5378 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f93de8dc-026f-4868-94fb-d1b2006dfb63-istio-podinfo\") pod \"maas-default-gateway-openshift-default-769df7846c-xghmf\" (UID: \"f93de8dc-026f-4868-94fb-d1b2006dfb63\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-xghmf" Oct 30 20:49:07.276587 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:49:07.274237 5378 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f93de8dc-026f-4868-94fb-d1b2006dfb63-credential-socket\") pod \"maas-default-gateway-openshift-default-769df7846c-xghmf\" (UID: \"f93de8dc-026f-4868-94fb-d1b2006dfb63\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-xghmf" Oct 30 20:49:07.276587 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:49:07.274368 5378 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f93de8dc-026f-4868-94fb-d1b2006dfb63-workload-certs\") pod \"maas-default-gateway-openshift-default-769df7846c-xghmf\" (UID: \"f93de8dc-026f-4868-94fb-d1b2006dfb63\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-xghmf" Oct 30 20:49:07.276587 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:49:07.274429 5378 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-df5h6\" (UniqueName: \"kubernetes.io/projected/f93de8dc-026f-4868-94fb-d1b2006dfb63-kube-api-access-df5h6\") pod \"maas-default-gateway-openshift-default-769df7846c-xghmf\" (UID: \"f93de8dc-026f-4868-94fb-d1b2006dfb63\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-xghmf" Oct 30 20:49:07.276587 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:49:07.274486 5378 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f93de8dc-026f-4868-94fb-d1b2006dfb63-istio-envoy\") pod \"maas-default-gateway-openshift-default-769df7846c-xghmf\" (UID: \"f93de8dc-026f-4868-94fb-d1b2006dfb63\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-xghmf" Oct 30 20:49:07.276587 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:49:07.274516 5378 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f93de8dc-026f-4868-94fb-d1b2006dfb63-istio-token\") pod \"maas-default-gateway-openshift-default-769df7846c-xghmf\" (UID: \"f93de8dc-026f-4868-94fb-d1b2006dfb63\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-xghmf" Oct 30 20:49:07.276587 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:49:07.274539 5378 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f93de8dc-026f-4868-94fb-d1b2006dfb63-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-769df7846c-xghmf\" (UID: \"f93de8dc-026f-4868-94fb-d1b2006dfb63\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-xghmf" Oct 30 20:49:07.276587 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:49:07.274628 5378 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f93de8dc-026f-4868-94fb-d1b2006dfb63-workload-socket\") pod \"maas-default-gateway-openshift-default-769df7846c-xghmf\" (UID: \"f93de8dc-026f-4868-94fb-d1b2006dfb63\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-xghmf" Oct 30 20:49:07.276587 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:49:07.274676 5378 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f93de8dc-026f-4868-94fb-d1b2006dfb63-istio-data\") pod \"maas-default-gateway-openshift-default-769df7846c-xghmf\" (UID: \"f93de8dc-026f-4868-94fb-d1b2006dfb63\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-xghmf" Oct 30 20:49:07.276587 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:49:07.276164 5378 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-769df7846c-xghmf"] Oct 30 20:49:07.386642 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:49:07.377007 5378 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f93de8dc-026f-4868-94fb-d1b2006dfb63-istio-data\") pod \"maas-default-gateway-openshift-default-769df7846c-xghmf\" (UID: \"f93de8dc-026f-4868-94fb-d1b2006dfb63\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-xghmf" Oct 30 20:49:07.386642 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:49:07.377062 5378 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f93de8dc-026f-4868-94fb-d1b2006dfb63-istio-podinfo\") pod \"maas-default-gateway-openshift-default-769df7846c-xghmf\" (UID: \"f93de8dc-026f-4868-94fb-d1b2006dfb63\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-xghmf" Oct 30 20:49:07.386642 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:49:07.377096 5378 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f93de8dc-026f-4868-94fb-d1b2006dfb63-credential-socket\") pod \"maas-default-gateway-openshift-default-769df7846c-xghmf\" (UID: \"f93de8dc-026f-4868-94fb-d1b2006dfb63\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-xghmf" Oct 30 20:49:07.386642 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:49:07.377140 5378 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f93de8dc-026f-4868-94fb-d1b2006dfb63-workload-certs\") pod \"maas-default-gateway-openshift-default-769df7846c-xghmf\" (UID: \"f93de8dc-026f-4868-94fb-d1b2006dfb63\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-xghmf" Oct 30 20:49:07.386642 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:49:07.377181 5378 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-df5h6\" (UniqueName: \"kubernetes.io/projected/f93de8dc-026f-4868-94fb-d1b2006dfb63-kube-api-access-df5h6\") pod \"maas-default-gateway-openshift-default-769df7846c-xghmf\" (UID: \"f93de8dc-026f-4868-94fb-d1b2006dfb63\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-xghmf" Oct 30 20:49:07.386642 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:49:07.377242 5378 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f93de8dc-026f-4868-94fb-d1b2006dfb63-istio-envoy\") pod \"maas-default-gateway-openshift-default-769df7846c-xghmf\" (UID: \"f93de8dc-026f-4868-94fb-d1b2006dfb63\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-xghmf" Oct 30 20:49:07.386642 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:49:07.377273 5378 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f93de8dc-026f-4868-94fb-d1b2006dfb63-istio-token\") pod \"maas-default-gateway-openshift-default-769df7846c-xghmf\" (UID: \"f93de8dc-026f-4868-94fb-d1b2006dfb63\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-xghmf" Oct 30 20:49:07.386642 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:49:07.377297 5378 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f93de8dc-026f-4868-94fb-d1b2006dfb63-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-769df7846c-xghmf\" (UID: \"f93de8dc-026f-4868-94fb-d1b2006dfb63\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-xghmf" Oct 30 20:49:07.386642 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:49:07.377374 5378 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f93de8dc-026f-4868-94fb-d1b2006dfb63-workload-socket\") pod \"maas-default-gateway-openshift-default-769df7846c-xghmf\" (UID: \"f93de8dc-026f-4868-94fb-d1b2006dfb63\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-xghmf" Oct 30 20:49:07.386642 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:49:07.377839 5378 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f93de8dc-026f-4868-94fb-d1b2006dfb63-workload-socket\") pod \"maas-default-gateway-openshift-default-769df7846c-xghmf\" (UID: \"f93de8dc-026f-4868-94fb-d1b2006dfb63\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-xghmf" Oct 30 20:49:07.386642 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:49:07.377927 5378 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f93de8dc-026f-4868-94fb-d1b2006dfb63-workload-certs\") pod \"maas-default-gateway-openshift-default-769df7846c-xghmf\" (UID: \"f93de8dc-026f-4868-94fb-d1b2006dfb63\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-xghmf" Oct 30 20:49:07.386642 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:49:07.378212 5378 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f93de8dc-026f-4868-94fb-d1b2006dfb63-istio-data\") pod \"maas-default-gateway-openshift-default-769df7846c-xghmf\" (UID: \"f93de8dc-026f-4868-94fb-d1b2006dfb63\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-xghmf" Oct 30 20:49:07.386642 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:49:07.379099 5378 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f93de8dc-026f-4868-94fb-d1b2006dfb63-credential-socket\") pod \"maas-default-gateway-openshift-default-769df7846c-xghmf\" (UID: \"f93de8dc-026f-4868-94fb-d1b2006dfb63\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-xghmf" Oct 30 20:49:07.386642 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:49:07.379755 5378 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f93de8dc-026f-4868-94fb-d1b2006dfb63-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-769df7846c-xghmf\" (UID: \"f93de8dc-026f-4868-94fb-d1b2006dfb63\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-xghmf" Oct 30 20:49:07.386642 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:49:07.383969 5378 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f93de8dc-026f-4868-94fb-d1b2006dfb63-istio-envoy\") pod \"maas-default-gateway-openshift-default-769df7846c-xghmf\" (UID: \"f93de8dc-026f-4868-94fb-d1b2006dfb63\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-xghmf" Oct 30 20:49:07.386642 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:49:07.385910 5378 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f93de8dc-026f-4868-94fb-d1b2006dfb63-istio-podinfo\") pod \"maas-default-gateway-openshift-default-769df7846c-xghmf\" (UID: \"f93de8dc-026f-4868-94fb-d1b2006dfb63\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-xghmf" Oct 30 20:49:07.437597 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:49:07.437501 5378 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-df5h6\" (UniqueName: \"kubernetes.io/projected/f93de8dc-026f-4868-94fb-d1b2006dfb63-kube-api-access-df5h6\") pod \"maas-default-gateway-openshift-default-769df7846c-xghmf\" (UID: \"f93de8dc-026f-4868-94fb-d1b2006dfb63\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-xghmf" Oct 30 20:49:07.438266 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:49:07.438244 5378 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f93de8dc-026f-4868-94fb-d1b2006dfb63-istio-token\") pod \"maas-default-gateway-openshift-default-769df7846c-xghmf\" (UID: \"f93de8dc-026f-4868-94fb-d1b2006dfb63\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-xghmf" Oct 30 20:49:07.493530 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:49:07.493318 5378 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-xghmf" Oct 30 20:49:08.002509 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:49:08.002480 5378 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-769df7846c-xghmf"] Oct 30 20:49:08.026556 ip-10-0-138-149 kubenswrapper[5378]: W1030 20:49:08.026529 5378 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf93de8dc_026f_4868_94fb_d1b2006dfb63.slice/crio-a4db54218679f32c2cf80d2a21e9beb706fa211fd2b68b5ac8ede4d9b156c25e WatchSource:0}: Error finding container a4db54218679f32c2cf80d2a21e9beb706fa211fd2b68b5ac8ede4d9b156c25e: Status 404 returned error can't find the container with id a4db54218679f32c2cf80d2a21e9beb706fa211fd2b68b5ac8ede4d9b156c25e Oct 30 20:49:08.031890 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:49:08.031851 5378 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 20:49:08.031992 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:49:08.031936 5378 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 20:49:08.031992 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:49:08.031983 5378 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 20:49:08.199424 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:49:08.199390 5378 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-5f78687d5b-2r7nt" event={"ID":"273afb91-cad1-42d5-89c7-20d2521e082e","Type":"ContainerStarted","Data":"9b3ea892edaab19ca4278d07eabdcd24910f9a399cff7f5c374100f11510b810"} Oct 30 20:49:08.199818 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:49:08.199760 5378 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-5f78687d5b-2r7nt" Oct 30 20:49:08.201872 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:49:08.201839 5378 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-xghmf" event={"ID":"f93de8dc-026f-4868-94fb-d1b2006dfb63","Type":"ContainerStarted","Data":"a03c490fb106ddd77b0b5b17727dbb2f14aca6ddef347ab0984b35fe70931ea1"} Oct 30 20:49:08.201990 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:49:08.201905 5378 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-xghmf" event={"ID":"f93de8dc-026f-4868-94fb-d1b2006dfb63","Type":"ContainerStarted","Data":"a4db54218679f32c2cf80d2a21e9beb706fa211fd2b68b5ac8ede4d9b156c25e"} Oct 30 20:49:08.322428 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:49:08.322315 5378 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-5f78687d5b-2r7nt" podStartSLOduration=1.917766765 podStartE2EDuration="5.322298613s" totalImagesPullingTime="3.404531848s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:49:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:49:08.321800045 +0000 UTC m=+638.749913965" watchObservedRunningTime="2026-10-30 20:49:08.322298613 +0000 UTC m=+638.750412533" Oct 30 20:49:08.494339 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:49:08.494258 5378 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-xghmf" Oct 30 20:49:09.207594 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:49:09.207527 5378 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-7b98bb678c-7l6vk" event={"ID":"4c52be8c-3199-4691-a6ea-4f98842987c4","Type":"ContainerStarted","Data":"a52b95f5dee92f58655c0929756dd8b454cb5c140c31d714698318e3c886bc95"} Oct 30 20:49:09.208062 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:49:09.207962 5378 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-7b98bb678c-7l6vk" Oct 30 20:49:09.342268 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:49:09.342208 5378 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-7b98bb678c-7l6vk" podStartSLOduration=2.658165262 podStartE2EDuration="6.342191516s" totalImagesPullingTime="3.684026254s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:49:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:49:09.342118322 +0000 UTC m=+639.770232241" watchObservedRunningTime="2026-10-30 20:49:09.342191516 +0000 UTC m=+639.770305454" Oct 30 20:49:09.342477 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:49:09.342438 5378 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-xghmf" podStartSLOduration=2.342427335 podStartE2EDuration="2.342427335s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:49:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:49:08.446473043 +0000 UTC m=+638.874586962" watchObservedRunningTime="2026-10-30 20:49:09.342427335 +0000 UTC m=+639.770541256" Oct 30 20:49:09.495484 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:49:09.495409 5378 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-769df7846c-xghmf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.40:15021/healthz/ready\": dial tcp 10.133.0.40:15021: connect: connection refused" start-of-body= Oct 30 20:49:09.495484 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:49:09.495464 5378 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-xghmf" podUID="f93de8dc-026f-4868-94fb-d1b2006dfb63" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.40:15021/healthz/ready\": dial tcp 10.133.0.40:15021: connect: connection refused" Oct 30 20:49:10.493854 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:49:10.493819 5378 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-769df7846c-xghmf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.40:15021/healthz/ready\": dial tcp 10.133.0.40:15021: connect: connection refused" start-of-body= Oct 30 20:49:10.494338 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:49:10.493881 5378 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-xghmf" podUID="f93de8dc-026f-4868-94fb-d1b2006dfb63" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.40:15021/healthz/ready\": dial tcp 10.133.0.40:15021: connect: connection refused" Oct 30 20:49:11.130423 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:49:11.130395 5378 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/ai-gateway-controller-5f6dd4f69f-sff92" Oct 30 20:49:11.481063 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:49:11.480983 5378 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/ai-gateway-controller-6c6c76f5b9-7zvnk"] Oct 30 20:49:11.484910 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:49:11.484889 5378 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/ai-gateway-controller-6c6c76f5b9-7zvnk" Oct 30 20:49:11.494492 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:49:11.494469 5378 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-769df7846c-xghmf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.40:15021/healthz/ready\": dial tcp 10.133.0.40:15021: connect: connection refused" start-of-body= Oct 30 20:49:11.494897 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:49:11.494540 5378 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-xghmf" podUID="f93de8dc-026f-4868-94fb-d1b2006dfb63" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.40:15021/healthz/ready\": dial tcp 10.133.0.40:15021: connect: connection refused" Oct 30 20:49:11.541986 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:49:11.541959 5378 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/ai-gateway-controller-6c6c76f5b9-7zvnk"] Oct 30 20:49:11.617449 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:49:11.617416 5378 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cg7td\" (UniqueName: \"kubernetes.io/projected/a1568576-80d5-42ec-af7b-99d0dd79fa20-kube-api-access-cg7td\") pod \"ai-gateway-controller-6c6c76f5b9-7zvnk\" (UID: \"a1568576-80d5-42ec-af7b-99d0dd79fa20\") " pod="opendatahub/ai-gateway-controller-6c6c76f5b9-7zvnk" Oct 30 20:49:11.617626 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:49:11.617466 5378 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a1568576-80d5-42ec-af7b-99d0dd79fa20-metrics-certs\") pod \"ai-gateway-controller-6c6c76f5b9-7zvnk\" (UID: \"a1568576-80d5-42ec-af7b-99d0dd79fa20\") " pod="opendatahub/ai-gateway-controller-6c6c76f5b9-7zvnk" Oct 30 20:49:11.718257 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:49:11.718218 5378 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cg7td\" (UniqueName: \"kubernetes.io/projected/a1568576-80d5-42ec-af7b-99d0dd79fa20-kube-api-access-cg7td\") pod \"ai-gateway-controller-6c6c76f5b9-7zvnk\" (UID: \"a1568576-80d5-42ec-af7b-99d0dd79fa20\") " pod="opendatahub/ai-gateway-controller-6c6c76f5b9-7zvnk" Oct 30 20:49:11.718434 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:49:11.718278 5378 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a1568576-80d5-42ec-af7b-99d0dd79fa20-metrics-certs\") pod \"ai-gateway-controller-6c6c76f5b9-7zvnk\" (UID: \"a1568576-80d5-42ec-af7b-99d0dd79fa20\") " pod="opendatahub/ai-gateway-controller-6c6c76f5b9-7zvnk" Oct 30 20:49:11.720730 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:49:11.720702 5378 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a1568576-80d5-42ec-af7b-99d0dd79fa20-metrics-certs\") pod \"ai-gateway-controller-6c6c76f5b9-7zvnk\" (UID: \"a1568576-80d5-42ec-af7b-99d0dd79fa20\") " pod="opendatahub/ai-gateway-controller-6c6c76f5b9-7zvnk" Oct 30 20:49:11.778502 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:49:11.778421 5378 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cg7td\" (UniqueName: \"kubernetes.io/projected/a1568576-80d5-42ec-af7b-99d0dd79fa20-kube-api-access-cg7td\") pod \"ai-gateway-controller-6c6c76f5b9-7zvnk\" (UID: \"a1568576-80d5-42ec-af7b-99d0dd79fa20\") " pod="opendatahub/ai-gateway-controller-6c6c76f5b9-7zvnk" Oct 30 20:49:11.798325 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:49:11.798298 5378 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/ai-gateway-controller-6c6c76f5b9-7zvnk" Oct 30 20:49:12.174589 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:49:12.168968 5378 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/ai-gateway-controller-6c6c76f5b9-7zvnk"] Oct 30 20:49:12.181571 ip-10-0-138-149 kubenswrapper[5378]: W1030 20:49:12.181527 5378 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda1568576_80d5_42ec_af7b_99d0dd79fa20.slice/crio-27b5d5cc6b767c036acaf80a3aae873d502dd8fc822a31649f0667ba63070377 WatchSource:0}: Error finding container 27b5d5cc6b767c036acaf80a3aae873d502dd8fc822a31649f0667ba63070377: Status 404 returned error can't find the container with id 27b5d5cc6b767c036acaf80a3aae873d502dd8fc822a31649f0667ba63070377 Oct 30 20:49:12.182987 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:49:12.182971 5378 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 30 20:49:12.220968 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:49:12.220940 5378 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-6c6c76f5b9-7zvnk" event={"ID":"a1568576-80d5-42ec-af7b-99d0dd79fa20","Type":"ContainerStarted","Data":"27b5d5cc6b767c036acaf80a3aae873d502dd8fc822a31649f0667ba63070377"} Oct 30 20:49:12.493918 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:49:12.493834 5378 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-769df7846c-xghmf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.40:15021/healthz/ready\": dial tcp 10.133.0.40:15021: connect: connection refused" start-of-body= Oct 30 20:49:12.493918 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:49:12.493887 5378 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-xghmf" podUID="f93de8dc-026f-4868-94fb-d1b2006dfb63" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.40:15021/healthz/ready\": dial tcp 10.133.0.40:15021: connect: connection refused" Oct 30 20:49:13.227010 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:49:13.226975 5378 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-6c6c76f5b9-7zvnk" event={"ID":"a1568576-80d5-42ec-af7b-99d0dd79fa20","Type":"ContainerStarted","Data":"ad324f1624b5b50923a47c644bdd2e28ce8f60f5992e9aaafb8cd74bfe001d94"} Oct 30 20:49:13.227385 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:49:13.227046 5378 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/ai-gateway-controller-6c6c76f5b9-7zvnk" Oct 30 20:49:13.344092 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:49:13.343992 5378 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/ai-gateway-controller-6c6c76f5b9-7zvnk" podStartSLOduration=1.503619049 podStartE2EDuration="2.343977442s" totalImagesPullingTime="840.358393ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:49:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:49:13.343338826 +0000 UTC m=+643.771452746" watchObservedRunningTime="2026-10-30 20:49:13.343977442 +0000 UTC m=+643.772091398" Oct 30 20:49:13.494578 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:49:13.494540 5378 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-769df7846c-xghmf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.40:15021/healthz/ready\": dial tcp 10.133.0.40:15021: connect: connection refused" start-of-body= Oct 30 20:49:13.494767 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:49:13.494642 5378 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-xghmf" podUID="f93de8dc-026f-4868-94fb-d1b2006dfb63" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.40:15021/healthz/ready\": dial tcp 10.133.0.40:15021: connect: connection refused" Oct 30 20:49:14.493845 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:49:14.493805 5378 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-769df7846c-xghmf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.40:15021/healthz/ready\": dial tcp 10.133.0.40:15021: connect: connection refused" start-of-body= Oct 30 20:49:14.494280 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:49:14.493872 5378 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-xghmf" podUID="f93de8dc-026f-4868-94fb-d1b2006dfb63" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.40:15021/healthz/ready\": dial tcp 10.133.0.40:15021: connect: connection refused" Oct 30 20:49:16.282807 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:49:16.282615 5378 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-xghmf" Oct 30 20:49:16.283735 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:49:16.283714 5378 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-xghmf" Oct 30 20:49:16.285019 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:49:16.285001 5378 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-xghmf" Oct 30 20:49:18.836625 ip-10-0-138-149 kubenswrapper[5378]: E1030 20:49:18.836592 5378 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q6mx7" containerName="extractor" Oct 30 20:49:19.210369 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:49:19.210344 5378 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-5f78687d5b-2r7nt" Oct 30 20:49:21.218876 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:49:21.218842 5378 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-7b98bb678c-7l6vk" Oct 30 20:49:24.234985 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:49:24.234954 5378 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/ai-gateway-controller-6c6c76f5b9-7zvnk" Oct 30 20:49:24.405298 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:49:24.405266 5378 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/ai-gateway-controller-5f6dd4f69f-sff92"] Oct 30 20:49:24.405526 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:49:24.405501 5378 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/ai-gateway-controller-5f6dd4f69f-sff92" podUID="f0bacb9b-8d64-41dc-98e9-faa1e4f83855" containerName="manager" containerID="cri-o://a7595b0a164c95ef7e327ad6f12ed3b48ef7f7aef94be59fe5c75c0073aea0a0" gracePeriod=30 Oct 30 20:49:24.666011 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:49:24.665986 5378 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/ai-gateway-controller-5f6dd4f69f-sff92" Oct 30 20:49:24.728118 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:49:24.728092 5378 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f0bacb9b-8d64-41dc-98e9-faa1e4f83855-metrics-certs\" (UniqueName: \"kubernetes.io/secret/f0bacb9b-8d64-41dc-98e9-faa1e4f83855-metrics-certs\") pod \"f0bacb9b-8d64-41dc-98e9-faa1e4f83855\" (UID: \"f0bacb9b-8d64-41dc-98e9-faa1e4f83855\") " Oct 30 20:49:24.728286 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:49:24.728149 5378 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f0bacb9b-8d64-41dc-98e9-faa1e4f83855-kube-api-access-dbcf7\" (UniqueName: \"kubernetes.io/projected/f0bacb9b-8d64-41dc-98e9-faa1e4f83855-kube-api-access-dbcf7\") pod \"f0bacb9b-8d64-41dc-98e9-faa1e4f83855\" (UID: \"f0bacb9b-8d64-41dc-98e9-faa1e4f83855\") " Oct 30 20:49:24.730137 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:49:24.730112 5378 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f0bacb9b-8d64-41dc-98e9-faa1e4f83855-kube-api-access-dbcf7" pod "f0bacb9b-8d64-41dc-98e9-faa1e4f83855" (UID: "f0bacb9b-8d64-41dc-98e9-faa1e4f83855"). InnerVolumeSpecName "kube-api-access-dbcf7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:49:24.730257 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:49:24.730138 5378 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f0bacb9b-8d64-41dc-98e9-faa1e4f83855-metrics-certs" pod "f0bacb9b-8d64-41dc-98e9-faa1e4f83855" (UID: "f0bacb9b-8d64-41dc-98e9-faa1e4f83855"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 20:49:24.829554 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:49:24.829479 5378 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f0bacb9b-8d64-41dc-98e9-faa1e4f83855-metrics-certs\") on node \"ip-10-0-138-149.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:49:24.829554 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:49:24.829507 5378 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dbcf7\" (UniqueName: \"kubernetes.io/projected/f0bacb9b-8d64-41dc-98e9-faa1e4f83855-kube-api-access-dbcf7\") on node \"ip-10-0-138-149.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:49:25.282767 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:49:25.282732 5378 generic.go:356] "Generic (PLEG): container finished" podID="f0bacb9b-8d64-41dc-98e9-faa1e4f83855" containerID="a7595b0a164c95ef7e327ad6f12ed3b48ef7f7aef94be59fe5c75c0073aea0a0" exitCode=0 Oct 30 20:49:25.283267 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:49:25.282794 5378 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/ai-gateway-controller-5f6dd4f69f-sff92" Oct 30 20:49:25.283267 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:49:25.282813 5378 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-5f6dd4f69f-sff92" event={"ID":"f0bacb9b-8d64-41dc-98e9-faa1e4f83855","Type":"ContainerDied","Data":"a7595b0a164c95ef7e327ad6f12ed3b48ef7f7aef94be59fe5c75c0073aea0a0"} Oct 30 20:49:25.283267 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:49:25.282865 5378 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-5f6dd4f69f-sff92" event={"ID":"f0bacb9b-8d64-41dc-98e9-faa1e4f83855","Type":"ContainerDied","Data":"d8706d765432a59e1f414c1c38169e4530496c3b725180bcc36f04f88413736a"} Oct 30 20:49:25.283267 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:49:25.282890 5378 scope.go:122] "RemoveContainer" containerID="a7595b0a164c95ef7e327ad6f12ed3b48ef7f7aef94be59fe5c75c0073aea0a0" Oct 30 20:49:25.292246 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:49:25.292197 5378 scope.go:122] "RemoveContainer" containerID="a7595b0a164c95ef7e327ad6f12ed3b48ef7f7aef94be59fe5c75c0073aea0a0" Oct 30 20:49:25.292490 ip-10-0-138-149 kubenswrapper[5378]: E1030 20:49:25.292471 5378 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a7595b0a164c95ef7e327ad6f12ed3b48ef7f7aef94be59fe5c75c0073aea0a0\": container with ID starting with a7595b0a164c95ef7e327ad6f12ed3b48ef7f7aef94be59fe5c75c0073aea0a0 not found: ID does not exist" containerID="a7595b0a164c95ef7e327ad6f12ed3b48ef7f7aef94be59fe5c75c0073aea0a0" Oct 30 20:49:25.292556 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:49:25.292497 5378 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a7595b0a164c95ef7e327ad6f12ed3b48ef7f7aef94be59fe5c75c0073aea0a0"} err="failed to get container status \"a7595b0a164c95ef7e327ad6f12ed3b48ef7f7aef94be59fe5c75c0073aea0a0\": rpc error: code = NotFound desc = could not find container \"a7595b0a164c95ef7e327ad6f12ed3b48ef7f7aef94be59fe5c75c0073aea0a0\": container with ID starting with a7595b0a164c95ef7e327ad6f12ed3b48ef7f7aef94be59fe5c75c0073aea0a0 not found: ID does not exist" Oct 30 20:49:25.459512 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:49:25.459477 5378 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/ai-gateway-controller-5f6dd4f69f-sff92"] Oct 30 20:49:25.508862 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:49:25.508837 5378 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/ai-gateway-controller-5f6dd4f69f-sff92"] Oct 30 20:49:25.739670 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:49:25.739633 5378 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f0bacb9b-8d64-41dc-98e9-faa1e4f83855" path="/var/lib/kubelet/pods/f0bacb9b-8d64-41dc-98e9-faa1e4f83855/volumes" Oct 30 20:49:26.055887 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:49:26.055796 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 20:49:26.569638 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:49:26.569609 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 20:49:28.220674 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:49:28.220643 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 20:49:28.372333 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:49:28.372272 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 20:49:29.968317 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:49:29.968289 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 20:49:30.120349 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:49:30.120308 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 20:49:39.451244 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:49:39.451185 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 20:49:48.835583 ip-10-0-138-149 kubenswrapper[5378]: E1030 20:49:48.835528 5378 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q6mx7" containerName="extractor" Oct 30 20:49:55.713987 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:49:55.713947 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 20:49:55.854302 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:49:55.854269 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 20:49:56.288909 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:49:56.288875 5378 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-f9f277-openshift-default-869d58fb77-p2wtm"] Oct 30 20:49:56.289253 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:49:56.289242 5378 cpu_manager.go:414] "containerMap: removing container" podUID="f0bacb9b-8d64-41dc-98e9-faa1e4f83855" containerName="manager" Oct 30 20:49:56.289299 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:49:56.289254 5378 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f0bacb9b-8d64-41dc-98e9-faa1e4f83855" containerName="manager" Oct 30 20:49:56.289334 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:49:56.289329 5378 memory_manager.go:365] "RemoveStaleState removing state" podUID="f0bacb9b-8d64-41dc-98e9-faa1e4f83855" containerName="manager" Oct 30 20:49:56.292409 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:49:56.292390 5378 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-f9f277-openshift-default-869d58fb77-p2wtm" Oct 30 20:49:56.311696 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:49:56.311664 5378 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/261c5b5a-84f8-4527-9b51-567d6dbf17ca-istio-token\") pod \"e2e-worker-w1-f9f277-openshift-default-869d58fb77-p2wtm\" (UID: \"261c5b5a-84f8-4527-9b51-567d6dbf17ca\") " pod="openshift-ingress/e2e-worker-w1-f9f277-openshift-default-869d58fb77-p2wtm" Oct 30 20:49:56.311849 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:49:56.311821 5378 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/261c5b5a-84f8-4527-9b51-567d6dbf17ca-workload-socket\") pod \"e2e-worker-w1-f9f277-openshift-default-869d58fb77-p2wtm\" (UID: \"261c5b5a-84f8-4527-9b51-567d6dbf17ca\") " pod="openshift-ingress/e2e-worker-w1-f9f277-openshift-default-869d58fb77-p2wtm" Oct 30 20:49:56.311959 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:49:56.311934 5378 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/261c5b5a-84f8-4527-9b51-567d6dbf17ca-workload-certs\") pod \"e2e-worker-w1-f9f277-openshift-default-869d58fb77-p2wtm\" (UID: \"261c5b5a-84f8-4527-9b51-567d6dbf17ca\") " pod="openshift-ingress/e2e-worker-w1-f9f277-openshift-default-869d58fb77-p2wtm" Oct 30 20:49:56.312016 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:49:56.311981 5378 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/261c5b5a-84f8-4527-9b51-567d6dbf17ca-istiod-ca-cert\") pod \"e2e-worker-w1-f9f277-openshift-default-869d58fb77-p2wtm\" (UID: \"261c5b5a-84f8-4527-9b51-567d6dbf17ca\") " pod="openshift-ingress/e2e-worker-w1-f9f277-openshift-default-869d58fb77-p2wtm" Oct 30 20:49:56.312071 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:49:56.312014 5378 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/261c5b5a-84f8-4527-9b51-567d6dbf17ca-credential-socket\") pod \"e2e-worker-w1-f9f277-openshift-default-869d58fb77-p2wtm\" (UID: \"261c5b5a-84f8-4527-9b51-567d6dbf17ca\") " pod="openshift-ingress/e2e-worker-w1-f9f277-openshift-default-869d58fb77-p2wtm" Oct 30 20:49:56.312171 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:49:56.312146 5378 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/261c5b5a-84f8-4527-9b51-567d6dbf17ca-istio-envoy\") pod \"e2e-worker-w1-f9f277-openshift-default-869d58fb77-p2wtm\" (UID: \"261c5b5a-84f8-4527-9b51-567d6dbf17ca\") " pod="openshift-ingress/e2e-worker-w1-f9f277-openshift-default-869d58fb77-p2wtm" Oct 30 20:49:56.312216 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:49:56.312185 5378 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/261c5b5a-84f8-4527-9b51-567d6dbf17ca-istio-podinfo\") pod \"e2e-worker-w1-f9f277-openshift-default-869d58fb77-p2wtm\" (UID: \"261c5b5a-84f8-4527-9b51-567d6dbf17ca\") " pod="openshift-ingress/e2e-worker-w1-f9f277-openshift-default-869d58fb77-p2wtm" Oct 30 20:49:56.312267 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:49:56.312212 5378 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2hdv5\" (UniqueName: \"kubernetes.io/projected/261c5b5a-84f8-4527-9b51-567d6dbf17ca-kube-api-access-2hdv5\") pod \"e2e-worker-w1-f9f277-openshift-default-869d58fb77-p2wtm\" (UID: \"261c5b5a-84f8-4527-9b51-567d6dbf17ca\") " pod="openshift-ingress/e2e-worker-w1-f9f277-openshift-default-869d58fb77-p2wtm" Oct 30 20:49:56.312340 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:49:56.312316 5378 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/261c5b5a-84f8-4527-9b51-567d6dbf17ca-istio-data\") pod \"e2e-worker-w1-f9f277-openshift-default-869d58fb77-p2wtm\" (UID: \"261c5b5a-84f8-4527-9b51-567d6dbf17ca\") " pod="openshift-ingress/e2e-worker-w1-f9f277-openshift-default-869d58fb77-p2wtm" Oct 30 20:49:56.351771 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:49:56.351736 5378 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w1-f9f277-openshift-default-dockercfg-5xmn8\"" Oct 30 20:49:56.399590 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:49:56.399535 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 20:49:56.401838 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:49:56.401816 5378 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-f9f277-openshift-default-869d58fb77-p2wtm"] Oct 30 20:49:56.412780 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:49:56.412752 5378 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/261c5b5a-84f8-4527-9b51-567d6dbf17ca-istiod-ca-cert\") pod \"e2e-worker-w1-f9f277-openshift-default-869d58fb77-p2wtm\" (UID: \"261c5b5a-84f8-4527-9b51-567d6dbf17ca\") " pod="openshift-ingress/e2e-worker-w1-f9f277-openshift-default-869d58fb77-p2wtm" Oct 30 20:49:56.412906 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:49:56.412786 5378 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/261c5b5a-84f8-4527-9b51-567d6dbf17ca-credential-socket\") pod \"e2e-worker-w1-f9f277-openshift-default-869d58fb77-p2wtm\" (UID: \"261c5b5a-84f8-4527-9b51-567d6dbf17ca\") " pod="openshift-ingress/e2e-worker-w1-f9f277-openshift-default-869d58fb77-p2wtm" Oct 30 20:49:56.412906 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:49:56.412817 5378 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/261c5b5a-84f8-4527-9b51-567d6dbf17ca-istio-envoy\") pod \"e2e-worker-w1-f9f277-openshift-default-869d58fb77-p2wtm\" (UID: \"261c5b5a-84f8-4527-9b51-567d6dbf17ca\") " pod="openshift-ingress/e2e-worker-w1-f9f277-openshift-default-869d58fb77-p2wtm" Oct 30 20:49:56.412906 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:49:56.412840 5378 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/261c5b5a-84f8-4527-9b51-567d6dbf17ca-istio-podinfo\") pod \"e2e-worker-w1-f9f277-openshift-default-869d58fb77-p2wtm\" (UID: \"261c5b5a-84f8-4527-9b51-567d6dbf17ca\") " pod="openshift-ingress/e2e-worker-w1-f9f277-openshift-default-869d58fb77-p2wtm" Oct 30 20:49:56.412906 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:49:56.412864 5378 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2hdv5\" (UniqueName: \"kubernetes.io/projected/261c5b5a-84f8-4527-9b51-567d6dbf17ca-kube-api-access-2hdv5\") pod \"e2e-worker-w1-f9f277-openshift-default-869d58fb77-p2wtm\" (UID: \"261c5b5a-84f8-4527-9b51-567d6dbf17ca\") " pod="openshift-ingress/e2e-worker-w1-f9f277-openshift-default-869d58fb77-p2wtm" Oct 30 20:49:56.412906 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:49:56.412893 5378 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/261c5b5a-84f8-4527-9b51-567d6dbf17ca-istio-data\") pod \"e2e-worker-w1-f9f277-openshift-default-869d58fb77-p2wtm\" (UID: \"261c5b5a-84f8-4527-9b51-567d6dbf17ca\") " pod="openshift-ingress/e2e-worker-w1-f9f277-openshift-default-869d58fb77-p2wtm" Oct 30 20:49:56.413168 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:49:56.412932 5378 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/261c5b5a-84f8-4527-9b51-567d6dbf17ca-istio-token\") pod \"e2e-worker-w1-f9f277-openshift-default-869d58fb77-p2wtm\" (UID: \"261c5b5a-84f8-4527-9b51-567d6dbf17ca\") " pod="openshift-ingress/e2e-worker-w1-f9f277-openshift-default-869d58fb77-p2wtm" Oct 30 20:49:56.413168 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:49:56.413038 5378 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/261c5b5a-84f8-4527-9b51-567d6dbf17ca-workload-socket\") pod \"e2e-worker-w1-f9f277-openshift-default-869d58fb77-p2wtm\" (UID: \"261c5b5a-84f8-4527-9b51-567d6dbf17ca\") " pod="openshift-ingress/e2e-worker-w1-f9f277-openshift-default-869d58fb77-p2wtm" Oct 30 20:49:56.413168 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:49:56.413100 5378 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/261c5b5a-84f8-4527-9b51-567d6dbf17ca-workload-certs\") pod \"e2e-worker-w1-f9f277-openshift-default-869d58fb77-p2wtm\" (UID: \"261c5b5a-84f8-4527-9b51-567d6dbf17ca\") " pod="openshift-ingress/e2e-worker-w1-f9f277-openshift-default-869d58fb77-p2wtm" Oct 30 20:49:56.413321 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:49:56.413216 5378 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/261c5b5a-84f8-4527-9b51-567d6dbf17ca-credential-socket\") pod \"e2e-worker-w1-f9f277-openshift-default-869d58fb77-p2wtm\" (UID: \"261c5b5a-84f8-4527-9b51-567d6dbf17ca\") " pod="openshift-ingress/e2e-worker-w1-f9f277-openshift-default-869d58fb77-p2wtm" Oct 30 20:49:56.413383 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:49:56.413334 5378 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/261c5b5a-84f8-4527-9b51-567d6dbf17ca-istio-data\") pod \"e2e-worker-w1-f9f277-openshift-default-869d58fb77-p2wtm\" (UID: \"261c5b5a-84f8-4527-9b51-567d6dbf17ca\") " pod="openshift-ingress/e2e-worker-w1-f9f277-openshift-default-869d58fb77-p2wtm" Oct 30 20:49:56.413441 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:49:56.413422 5378 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/261c5b5a-84f8-4527-9b51-567d6dbf17ca-workload-certs\") pod \"e2e-worker-w1-f9f277-openshift-default-869d58fb77-p2wtm\" (UID: \"261c5b5a-84f8-4527-9b51-567d6dbf17ca\") " pod="openshift-ingress/e2e-worker-w1-f9f277-openshift-default-869d58fb77-p2wtm" Oct 30 20:49:56.413636 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:49:56.413612 5378 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/261c5b5a-84f8-4527-9b51-567d6dbf17ca-workload-socket\") pod \"e2e-worker-w1-f9f277-openshift-default-869d58fb77-p2wtm\" (UID: \"261c5b5a-84f8-4527-9b51-567d6dbf17ca\") " pod="openshift-ingress/e2e-worker-w1-f9f277-openshift-default-869d58fb77-p2wtm" Oct 30 20:49:56.413699 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:49:56.413620 5378 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/261c5b5a-84f8-4527-9b51-567d6dbf17ca-istiod-ca-cert\") pod \"e2e-worker-w1-f9f277-openshift-default-869d58fb77-p2wtm\" (UID: \"261c5b5a-84f8-4527-9b51-567d6dbf17ca\") " pod="openshift-ingress/e2e-worker-w1-f9f277-openshift-default-869d58fb77-p2wtm" Oct 30 20:49:56.415288 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:49:56.415263 5378 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/261c5b5a-84f8-4527-9b51-567d6dbf17ca-istio-envoy\") pod \"e2e-worker-w1-f9f277-openshift-default-869d58fb77-p2wtm\" (UID: \"261c5b5a-84f8-4527-9b51-567d6dbf17ca\") " pod="openshift-ingress/e2e-worker-w1-f9f277-openshift-default-869d58fb77-p2wtm" Oct 30 20:49:56.415623 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:49:56.415604 5378 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/261c5b5a-84f8-4527-9b51-567d6dbf17ca-istio-podinfo\") pod \"e2e-worker-w1-f9f277-openshift-default-869d58fb77-p2wtm\" (UID: \"261c5b5a-84f8-4527-9b51-567d6dbf17ca\") " pod="openshift-ingress/e2e-worker-w1-f9f277-openshift-default-869d58fb77-p2wtm" Oct 30 20:49:56.472501 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:49:56.472449 5378 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2hdv5\" (UniqueName: \"kubernetes.io/projected/261c5b5a-84f8-4527-9b51-567d6dbf17ca-kube-api-access-2hdv5\") pod \"e2e-worker-w1-f9f277-openshift-default-869d58fb77-p2wtm\" (UID: \"261c5b5a-84f8-4527-9b51-567d6dbf17ca\") " pod="openshift-ingress/e2e-worker-w1-f9f277-openshift-default-869d58fb77-p2wtm" Oct 30 20:49:56.472848 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:49:56.472825 5378 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/261c5b5a-84f8-4527-9b51-567d6dbf17ca-istio-token\") pod \"e2e-worker-w1-f9f277-openshift-default-869d58fb77-p2wtm\" (UID: \"261c5b5a-84f8-4527-9b51-567d6dbf17ca\") " pod="openshift-ingress/e2e-worker-w1-f9f277-openshift-default-869d58fb77-p2wtm" Oct 30 20:49:56.604812 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:49:56.604781 5378 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-f9f277-openshift-default-869d58fb77-p2wtm" Oct 30 20:49:56.736683 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:49:56.736619 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 20:49:56.902334 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:49:56.902234 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 20:49:56.980510 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:49:56.980480 5378 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-682999-openshift-default-54dcb797b5-r8dcv"] Oct 30 20:49:56.984949 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:49:56.984932 5378 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-682999-openshift-default-54dcb797b5-r8dcv" Oct 30 20:49:57.020288 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:49:57.020256 5378 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9b76fbb2-f5a8-4fea-bdfd-30f40bee7008-istio-token\") pod \"e2e-worker-w3-682999-openshift-default-54dcb797b5-r8dcv\" (UID: \"9b76fbb2-f5a8-4fea-bdfd-30f40bee7008\") " pod="openshift-ingress/e2e-worker-w3-682999-openshift-default-54dcb797b5-r8dcv" Oct 30 20:49:57.020434 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:49:57.020341 5378 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9b76fbb2-f5a8-4fea-bdfd-30f40bee7008-workload-certs\") pod \"e2e-worker-w3-682999-openshift-default-54dcb797b5-r8dcv\" (UID: \"9b76fbb2-f5a8-4fea-bdfd-30f40bee7008\") " pod="openshift-ingress/e2e-worker-w3-682999-openshift-default-54dcb797b5-r8dcv" Oct 30 20:49:57.020434 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:49:57.020366 5378 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gxkn6\" (UniqueName: \"kubernetes.io/projected/9b76fbb2-f5a8-4fea-bdfd-30f40bee7008-kube-api-access-gxkn6\") pod \"e2e-worker-w3-682999-openshift-default-54dcb797b5-r8dcv\" (UID: \"9b76fbb2-f5a8-4fea-bdfd-30f40bee7008\") " pod="openshift-ingress/e2e-worker-w3-682999-openshift-default-54dcb797b5-r8dcv" Oct 30 20:49:57.020434 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:49:57.020405 5378 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9b76fbb2-f5a8-4fea-bdfd-30f40bee7008-istio-envoy\") pod \"e2e-worker-w3-682999-openshift-default-54dcb797b5-r8dcv\" (UID: \"9b76fbb2-f5a8-4fea-bdfd-30f40bee7008\") " pod="openshift-ingress/e2e-worker-w3-682999-openshift-default-54dcb797b5-r8dcv" Oct 30 20:49:57.020647 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:49:57.020492 5378 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9b76fbb2-f5a8-4fea-bdfd-30f40bee7008-istio-podinfo\") pod \"e2e-worker-w3-682999-openshift-default-54dcb797b5-r8dcv\" (UID: \"9b76fbb2-f5a8-4fea-bdfd-30f40bee7008\") " pod="openshift-ingress/e2e-worker-w3-682999-openshift-default-54dcb797b5-r8dcv" Oct 30 20:49:57.020647 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:49:57.020599 5378 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9b76fbb2-f5a8-4fea-bdfd-30f40bee7008-istio-data\") pod \"e2e-worker-w3-682999-openshift-default-54dcb797b5-r8dcv\" (UID: \"9b76fbb2-f5a8-4fea-bdfd-30f40bee7008\") " pod="openshift-ingress/e2e-worker-w3-682999-openshift-default-54dcb797b5-r8dcv" Oct 30 20:49:57.020758 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:49:57.020652 5378 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9b76fbb2-f5a8-4fea-bdfd-30f40bee7008-workload-socket\") pod \"e2e-worker-w3-682999-openshift-default-54dcb797b5-r8dcv\" (UID: \"9b76fbb2-f5a8-4fea-bdfd-30f40bee7008\") " pod="openshift-ingress/e2e-worker-w3-682999-openshift-default-54dcb797b5-r8dcv" Oct 30 20:49:57.020758 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:49:57.020676 5378 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9b76fbb2-f5a8-4fea-bdfd-30f40bee7008-credential-socket\") pod \"e2e-worker-w3-682999-openshift-default-54dcb797b5-r8dcv\" (UID: \"9b76fbb2-f5a8-4fea-bdfd-30f40bee7008\") " pod="openshift-ingress/e2e-worker-w3-682999-openshift-default-54dcb797b5-r8dcv" Oct 30 20:49:57.020864 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:49:57.020761 5378 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9b76fbb2-f5a8-4fea-bdfd-30f40bee7008-istiod-ca-cert\") pod \"e2e-worker-w3-682999-openshift-default-54dcb797b5-r8dcv\" (UID: \"9b76fbb2-f5a8-4fea-bdfd-30f40bee7008\") " pod="openshift-ingress/e2e-worker-w3-682999-openshift-default-54dcb797b5-r8dcv" Oct 30 20:49:57.044703 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:49:57.044679 5378 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-f9f277-openshift-default-869d58fb77-p2wtm"] Oct 30 20:49:57.046990 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:49:57.046963 5378 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w3-682999-openshift-default-dockercfg-g27tl\"" Oct 30 20:49:57.047307 ip-10-0-138-149 kubenswrapper[5378]: W1030 20:49:57.047283 5378 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod261c5b5a_84f8_4527_9b51_567d6dbf17ca.slice/crio-88f7ac03b84fddec1234adb4a47a5f1d4f9cd7ececfa41bd277c72d5720ed713 WatchSource:0}: Error finding container 88f7ac03b84fddec1234adb4a47a5f1d4f9cd7ececfa41bd277c72d5720ed713: Status 404 returned error can't find the container with id 88f7ac03b84fddec1234adb4a47a5f1d4f9cd7ececfa41bd277c72d5720ed713 Oct 30 20:49:57.049535 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:49:57.049501 5378 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 20:49:57.049645 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:49:57.049590 5378 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 20:49:57.049645 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:49:57.049636 5378 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 20:49:57.077101 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:49:57.077078 5378 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-682999-openshift-default-54dcb797b5-r8dcv"] Oct 30 20:49:57.121428 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:49:57.121406 5378 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9b76fbb2-f5a8-4fea-bdfd-30f40bee7008-workload-certs\") pod \"e2e-worker-w3-682999-openshift-default-54dcb797b5-r8dcv\" (UID: \"9b76fbb2-f5a8-4fea-bdfd-30f40bee7008\") " pod="openshift-ingress/e2e-worker-w3-682999-openshift-default-54dcb797b5-r8dcv" Oct 30 20:49:57.121516 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:49:57.121434 5378 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gxkn6\" (UniqueName: \"kubernetes.io/projected/9b76fbb2-f5a8-4fea-bdfd-30f40bee7008-kube-api-access-gxkn6\") pod \"e2e-worker-w3-682999-openshift-default-54dcb797b5-r8dcv\" (UID: \"9b76fbb2-f5a8-4fea-bdfd-30f40bee7008\") " pod="openshift-ingress/e2e-worker-w3-682999-openshift-default-54dcb797b5-r8dcv" Oct 30 20:49:57.121516 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:49:57.121458 5378 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9b76fbb2-f5a8-4fea-bdfd-30f40bee7008-istio-envoy\") pod \"e2e-worker-w3-682999-openshift-default-54dcb797b5-r8dcv\" (UID: \"9b76fbb2-f5a8-4fea-bdfd-30f40bee7008\") " pod="openshift-ingress/e2e-worker-w3-682999-openshift-default-54dcb797b5-r8dcv" Oct 30 20:49:57.121516 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:49:57.121477 5378 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9b76fbb2-f5a8-4fea-bdfd-30f40bee7008-istio-podinfo\") pod \"e2e-worker-w3-682999-openshift-default-54dcb797b5-r8dcv\" (UID: \"9b76fbb2-f5a8-4fea-bdfd-30f40bee7008\") " pod="openshift-ingress/e2e-worker-w3-682999-openshift-default-54dcb797b5-r8dcv" Oct 30 20:49:57.121516 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:49:57.121503 5378 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9b76fbb2-f5a8-4fea-bdfd-30f40bee7008-istio-data\") pod \"e2e-worker-w3-682999-openshift-default-54dcb797b5-r8dcv\" (UID: \"9b76fbb2-f5a8-4fea-bdfd-30f40bee7008\") " pod="openshift-ingress/e2e-worker-w3-682999-openshift-default-54dcb797b5-r8dcv" Oct 30 20:49:57.121698 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:49:57.121533 5378 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9b76fbb2-f5a8-4fea-bdfd-30f40bee7008-workload-socket\") pod \"e2e-worker-w3-682999-openshift-default-54dcb797b5-r8dcv\" (UID: \"9b76fbb2-f5a8-4fea-bdfd-30f40bee7008\") " pod="openshift-ingress/e2e-worker-w3-682999-openshift-default-54dcb797b5-r8dcv" Oct 30 20:49:57.121698 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:49:57.121549 5378 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9b76fbb2-f5a8-4fea-bdfd-30f40bee7008-credential-socket\") pod \"e2e-worker-w3-682999-openshift-default-54dcb797b5-r8dcv\" (UID: \"9b76fbb2-f5a8-4fea-bdfd-30f40bee7008\") " pod="openshift-ingress/e2e-worker-w3-682999-openshift-default-54dcb797b5-r8dcv" Oct 30 20:49:57.121698 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:49:57.121643 5378 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9b76fbb2-f5a8-4fea-bdfd-30f40bee7008-istiod-ca-cert\") pod \"e2e-worker-w3-682999-openshift-default-54dcb797b5-r8dcv\" (UID: \"9b76fbb2-f5a8-4fea-bdfd-30f40bee7008\") " pod="openshift-ingress/e2e-worker-w3-682999-openshift-default-54dcb797b5-r8dcv" Oct 30 20:49:57.121698 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:49:57.121671 5378 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9b76fbb2-f5a8-4fea-bdfd-30f40bee7008-istio-token\") pod \"e2e-worker-w3-682999-openshift-default-54dcb797b5-r8dcv\" (UID: \"9b76fbb2-f5a8-4fea-bdfd-30f40bee7008\") " pod="openshift-ingress/e2e-worker-w3-682999-openshift-default-54dcb797b5-r8dcv" Oct 30 20:49:57.122110 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:49:57.122081 5378 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9b76fbb2-f5a8-4fea-bdfd-30f40bee7008-workload-certs\") pod \"e2e-worker-w3-682999-openshift-default-54dcb797b5-r8dcv\" (UID: \"9b76fbb2-f5a8-4fea-bdfd-30f40bee7008\") " pod="openshift-ingress/e2e-worker-w3-682999-openshift-default-54dcb797b5-r8dcv" Oct 30 20:49:57.122217 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:49:57.122153 5378 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9b76fbb2-f5a8-4fea-bdfd-30f40bee7008-istio-data\") pod \"e2e-worker-w3-682999-openshift-default-54dcb797b5-r8dcv\" (UID: \"9b76fbb2-f5a8-4fea-bdfd-30f40bee7008\") " pod="openshift-ingress/e2e-worker-w3-682999-openshift-default-54dcb797b5-r8dcv" Oct 30 20:49:57.122302 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:49:57.122229 5378 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9b76fbb2-f5a8-4fea-bdfd-30f40bee7008-workload-socket\") pod \"e2e-worker-w3-682999-openshift-default-54dcb797b5-r8dcv\" (UID: \"9b76fbb2-f5a8-4fea-bdfd-30f40bee7008\") " pod="openshift-ingress/e2e-worker-w3-682999-openshift-default-54dcb797b5-r8dcv" Oct 30 20:49:57.122456 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:49:57.122433 5378 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9b76fbb2-f5a8-4fea-bdfd-30f40bee7008-credential-socket\") pod \"e2e-worker-w3-682999-openshift-default-54dcb797b5-r8dcv\" (UID: \"9b76fbb2-f5a8-4fea-bdfd-30f40bee7008\") " pod="openshift-ingress/e2e-worker-w3-682999-openshift-default-54dcb797b5-r8dcv" Oct 30 20:49:57.122515 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:49:57.122439 5378 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9b76fbb2-f5a8-4fea-bdfd-30f40bee7008-istiod-ca-cert\") pod \"e2e-worker-w3-682999-openshift-default-54dcb797b5-r8dcv\" (UID: \"9b76fbb2-f5a8-4fea-bdfd-30f40bee7008\") " pod="openshift-ingress/e2e-worker-w3-682999-openshift-default-54dcb797b5-r8dcv" Oct 30 20:49:57.123646 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:49:57.123626 5378 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9b76fbb2-f5a8-4fea-bdfd-30f40bee7008-istio-envoy\") pod \"e2e-worker-w3-682999-openshift-default-54dcb797b5-r8dcv\" (UID: \"9b76fbb2-f5a8-4fea-bdfd-30f40bee7008\") " pod="openshift-ingress/e2e-worker-w3-682999-openshift-default-54dcb797b5-r8dcv" Oct 30 20:49:57.124084 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:49:57.124065 5378 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9b76fbb2-f5a8-4fea-bdfd-30f40bee7008-istio-podinfo\") pod \"e2e-worker-w3-682999-openshift-default-54dcb797b5-r8dcv\" (UID: \"9b76fbb2-f5a8-4fea-bdfd-30f40bee7008\") " pod="openshift-ingress/e2e-worker-w3-682999-openshift-default-54dcb797b5-r8dcv" Oct 30 20:49:57.193308 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:49:57.193140 5378 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gxkn6\" (UniqueName: \"kubernetes.io/projected/9b76fbb2-f5a8-4fea-bdfd-30f40bee7008-kube-api-access-gxkn6\") pod \"e2e-worker-w3-682999-openshift-default-54dcb797b5-r8dcv\" (UID: \"9b76fbb2-f5a8-4fea-bdfd-30f40bee7008\") " pod="openshift-ingress/e2e-worker-w3-682999-openshift-default-54dcb797b5-r8dcv" Oct 30 20:49:57.196318 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:49:57.196253 5378 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9b76fbb2-f5a8-4fea-bdfd-30f40bee7008-istio-token\") pod \"e2e-worker-w3-682999-openshift-default-54dcb797b5-r8dcv\" (UID: \"9b76fbb2-f5a8-4fea-bdfd-30f40bee7008\") " pod="openshift-ingress/e2e-worker-w3-682999-openshift-default-54dcb797b5-r8dcv" Oct 30 20:49:57.268692 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:49:57.268656 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 20:49:57.298742 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:49:57.298706 5378 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-682999-openshift-default-54dcb797b5-r8dcv" Oct 30 20:49:57.434260 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:49:57.434191 5378 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-f9f277-openshift-default-869d58fb77-p2wtm" event={"ID":"261c5b5a-84f8-4527-9b51-567d6dbf17ca","Type":"ContainerStarted","Data":"1ae019c888ef9d4c5b0a8b8d7e7b434bbad1b7c253644c927a9f6aa6276aa201"} Oct 30 20:49:57.434416 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:49:57.434292 5378 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-f9f277-openshift-default-869d58fb77-p2wtm" event={"ID":"261c5b5a-84f8-4527-9b51-567d6dbf17ca","Type":"ContainerStarted","Data":"88f7ac03b84fddec1234adb4a47a5f1d4f9cd7ececfa41bd277c72d5720ed713"} Oct 30 20:49:57.445030 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:49:57.444968 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 20:49:57.562383 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:49:57.562334 5378 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-f9f277-openshift-default-869d58fb77-p2wtm" podStartSLOduration=1.5623192719999999 podStartE2EDuration="1.562319272s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:49:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:49:57.561241295 +0000 UTC m=+687.989355213" watchObservedRunningTime="2026-10-30 20:49:57.562319272 +0000 UTC m=+687.990433190" Oct 30 20:49:57.605756 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:49:57.605721 5378 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-f9f277-openshift-default-869d58fb77-p2wtm" Oct 30 20:49:57.606744 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:49:57.606716 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 20:49:57.701436 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:49:57.701362 5378 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-1fdaa7-openshift-default-84586db7d5-sbpk8"] Oct 30 20:49:57.706116 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:49:57.706089 5378 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-1fdaa7-openshift-default-84586db7d5-sbpk8" Oct 30 20:49:57.729906 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:49:57.729252 5378 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6fq99\" (UniqueName: \"kubernetes.io/projected/2391a719-0c9b-4c56-b229-8f286a9aac96-kube-api-access-6fq99\") pod \"e2e-worker-w4-1fdaa7-openshift-default-84586db7d5-sbpk8\" (UID: \"2391a719-0c9b-4c56-b229-8f286a9aac96\") " pod="openshift-ingress/e2e-worker-w4-1fdaa7-openshift-default-84586db7d5-sbpk8" Oct 30 20:49:57.729906 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:49:57.729307 5378 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2391a719-0c9b-4c56-b229-8f286a9aac96-istio-data\") pod \"e2e-worker-w4-1fdaa7-openshift-default-84586db7d5-sbpk8\" (UID: \"2391a719-0c9b-4c56-b229-8f286a9aac96\") " pod="openshift-ingress/e2e-worker-w4-1fdaa7-openshift-default-84586db7d5-sbpk8" Oct 30 20:49:57.729906 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:49:57.729374 5378 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2391a719-0c9b-4c56-b229-8f286a9aac96-istiod-ca-cert\") pod \"e2e-worker-w4-1fdaa7-openshift-default-84586db7d5-sbpk8\" (UID: \"2391a719-0c9b-4c56-b229-8f286a9aac96\") " pod="openshift-ingress/e2e-worker-w4-1fdaa7-openshift-default-84586db7d5-sbpk8" Oct 30 20:49:57.729906 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:49:57.729406 5378 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2391a719-0c9b-4c56-b229-8f286a9aac96-istio-token\") pod \"e2e-worker-w4-1fdaa7-openshift-default-84586db7d5-sbpk8\" (UID: \"2391a719-0c9b-4c56-b229-8f286a9aac96\") " pod="openshift-ingress/e2e-worker-w4-1fdaa7-openshift-default-84586db7d5-sbpk8" Oct 30 20:49:57.729906 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:49:57.729437 5378 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2391a719-0c9b-4c56-b229-8f286a9aac96-istio-podinfo\") pod \"e2e-worker-w4-1fdaa7-openshift-default-84586db7d5-sbpk8\" (UID: \"2391a719-0c9b-4c56-b229-8f286a9aac96\") " pod="openshift-ingress/e2e-worker-w4-1fdaa7-openshift-default-84586db7d5-sbpk8" Oct 30 20:49:57.729906 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:49:57.729476 5378 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2391a719-0c9b-4c56-b229-8f286a9aac96-istio-envoy\") pod \"e2e-worker-w4-1fdaa7-openshift-default-84586db7d5-sbpk8\" (UID: \"2391a719-0c9b-4c56-b229-8f286a9aac96\") " pod="openshift-ingress/e2e-worker-w4-1fdaa7-openshift-default-84586db7d5-sbpk8" Oct 30 20:49:57.729906 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:49:57.729536 5378 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2391a719-0c9b-4c56-b229-8f286a9aac96-credential-socket\") pod \"e2e-worker-w4-1fdaa7-openshift-default-84586db7d5-sbpk8\" (UID: \"2391a719-0c9b-4c56-b229-8f286a9aac96\") " pod="openshift-ingress/e2e-worker-w4-1fdaa7-openshift-default-84586db7d5-sbpk8" Oct 30 20:49:57.729906 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:49:57.729599 5378 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2391a719-0c9b-4c56-b229-8f286a9aac96-workload-certs\") pod \"e2e-worker-w4-1fdaa7-openshift-default-84586db7d5-sbpk8\" (UID: \"2391a719-0c9b-4c56-b229-8f286a9aac96\") " pod="openshift-ingress/e2e-worker-w4-1fdaa7-openshift-default-84586db7d5-sbpk8" Oct 30 20:49:57.729906 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:49:57.729684 5378 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2391a719-0c9b-4c56-b229-8f286a9aac96-workload-socket\") pod \"e2e-worker-w4-1fdaa7-openshift-default-84586db7d5-sbpk8\" (UID: \"2391a719-0c9b-4c56-b229-8f286a9aac96\") " pod="openshift-ingress/e2e-worker-w4-1fdaa7-openshift-default-84586db7d5-sbpk8" Oct 30 20:49:57.744572 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:49:57.744535 5378 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-682999-openshift-default-54dcb797b5-r8dcv"] Oct 30 20:49:57.749159 ip-10-0-138-149 kubenswrapper[5378]: W1030 20:49:57.749128 5378 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9b76fbb2_f5a8_4fea_bdfd_30f40bee7008.slice/crio-8bc12fb5ff424f5a3e57cd4a0f2736b0b620bb112edb9633333d6ec65b2afd7b WatchSource:0}: Error finding container 8bc12fb5ff424f5a3e57cd4a0f2736b0b620bb112edb9633333d6ec65b2afd7b: Status 404 returned error can't find the container with id 8bc12fb5ff424f5a3e57cd4a0f2736b0b620bb112edb9633333d6ec65b2afd7b Oct 30 20:49:57.752015 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:49:57.751973 5378 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 20:49:57.752124 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:49:57.752062 5378 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 20:49:57.752124 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:49:57.752109 5378 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 20:49:57.760855 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:49:57.760831 5378 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w4-1fdaa7-openshift-default-dockercfg-57h8w\"" Oct 30 20:49:57.772035 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:49:57.772010 5378 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-1fdaa7-openshift-default-84586db7d5-sbpk8"] Oct 30 20:49:57.801168 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:49:57.801120 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 20:49:57.830137 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:49:57.830114 5378 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2391a719-0c9b-4c56-b229-8f286a9aac96-istio-envoy\") pod \"e2e-worker-w4-1fdaa7-openshift-default-84586db7d5-sbpk8\" (UID: \"2391a719-0c9b-4c56-b229-8f286a9aac96\") " pod="openshift-ingress/e2e-worker-w4-1fdaa7-openshift-default-84586db7d5-sbpk8" Oct 30 20:49:57.830293 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:49:57.830158 5378 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2391a719-0c9b-4c56-b229-8f286a9aac96-credential-socket\") pod \"e2e-worker-w4-1fdaa7-openshift-default-84586db7d5-sbpk8\" (UID: \"2391a719-0c9b-4c56-b229-8f286a9aac96\") " pod="openshift-ingress/e2e-worker-w4-1fdaa7-openshift-default-84586db7d5-sbpk8" Oct 30 20:49:57.830293 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:49:57.830193 5378 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2391a719-0c9b-4c56-b229-8f286a9aac96-workload-certs\") pod \"e2e-worker-w4-1fdaa7-openshift-default-84586db7d5-sbpk8\" (UID: \"2391a719-0c9b-4c56-b229-8f286a9aac96\") " pod="openshift-ingress/e2e-worker-w4-1fdaa7-openshift-default-84586db7d5-sbpk8" Oct 30 20:49:57.830293 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:49:57.830248 5378 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2391a719-0c9b-4c56-b229-8f286a9aac96-workload-socket\") pod \"e2e-worker-w4-1fdaa7-openshift-default-84586db7d5-sbpk8\" (UID: \"2391a719-0c9b-4c56-b229-8f286a9aac96\") " pod="openshift-ingress/e2e-worker-w4-1fdaa7-openshift-default-84586db7d5-sbpk8" Oct 30 20:49:57.830293 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:49:57.830278 5378 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6fq99\" (UniqueName: \"kubernetes.io/projected/2391a719-0c9b-4c56-b229-8f286a9aac96-kube-api-access-6fq99\") pod \"e2e-worker-w4-1fdaa7-openshift-default-84586db7d5-sbpk8\" (UID: \"2391a719-0c9b-4c56-b229-8f286a9aac96\") " pod="openshift-ingress/e2e-worker-w4-1fdaa7-openshift-default-84586db7d5-sbpk8" Oct 30 20:49:57.830531 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:49:57.830320 5378 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2391a719-0c9b-4c56-b229-8f286a9aac96-istio-data\") pod \"e2e-worker-w4-1fdaa7-openshift-default-84586db7d5-sbpk8\" (UID: \"2391a719-0c9b-4c56-b229-8f286a9aac96\") " pod="openshift-ingress/e2e-worker-w4-1fdaa7-openshift-default-84586db7d5-sbpk8" Oct 30 20:49:57.830531 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:49:57.830470 5378 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2391a719-0c9b-4c56-b229-8f286a9aac96-istiod-ca-cert\") pod \"e2e-worker-w4-1fdaa7-openshift-default-84586db7d5-sbpk8\" (UID: \"2391a719-0c9b-4c56-b229-8f286a9aac96\") " pod="openshift-ingress/e2e-worker-w4-1fdaa7-openshift-default-84586db7d5-sbpk8" Oct 30 20:49:57.830679 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:49:57.830588 5378 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2391a719-0c9b-4c56-b229-8f286a9aac96-istio-token\") pod \"e2e-worker-w4-1fdaa7-openshift-default-84586db7d5-sbpk8\" (UID: \"2391a719-0c9b-4c56-b229-8f286a9aac96\") " pod="openshift-ingress/e2e-worker-w4-1fdaa7-openshift-default-84586db7d5-sbpk8" Oct 30 20:49:57.830679 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:49:57.830621 5378 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2391a719-0c9b-4c56-b229-8f286a9aac96-credential-socket\") pod \"e2e-worker-w4-1fdaa7-openshift-default-84586db7d5-sbpk8\" (UID: \"2391a719-0c9b-4c56-b229-8f286a9aac96\") " pod="openshift-ingress/e2e-worker-w4-1fdaa7-openshift-default-84586db7d5-sbpk8" Oct 30 20:49:57.830679 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:49:57.830628 5378 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2391a719-0c9b-4c56-b229-8f286a9aac96-istio-podinfo\") pod \"e2e-worker-w4-1fdaa7-openshift-default-84586db7d5-sbpk8\" (UID: \"2391a719-0c9b-4c56-b229-8f286a9aac96\") " pod="openshift-ingress/e2e-worker-w4-1fdaa7-openshift-default-84586db7d5-sbpk8" Oct 30 20:49:57.831697 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:49:57.830743 5378 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2391a719-0c9b-4c56-b229-8f286a9aac96-workload-certs\") pod \"e2e-worker-w4-1fdaa7-openshift-default-84586db7d5-sbpk8\" (UID: \"2391a719-0c9b-4c56-b229-8f286a9aac96\") " pod="openshift-ingress/e2e-worker-w4-1fdaa7-openshift-default-84586db7d5-sbpk8" Oct 30 20:49:57.831697 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:49:57.830795 5378 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2391a719-0c9b-4c56-b229-8f286a9aac96-istio-data\") pod \"e2e-worker-w4-1fdaa7-openshift-default-84586db7d5-sbpk8\" (UID: \"2391a719-0c9b-4c56-b229-8f286a9aac96\") " pod="openshift-ingress/e2e-worker-w4-1fdaa7-openshift-default-84586db7d5-sbpk8" Oct 30 20:49:57.831697 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:49:57.830911 5378 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2391a719-0c9b-4c56-b229-8f286a9aac96-workload-socket\") pod \"e2e-worker-w4-1fdaa7-openshift-default-84586db7d5-sbpk8\" (UID: \"2391a719-0c9b-4c56-b229-8f286a9aac96\") " pod="openshift-ingress/e2e-worker-w4-1fdaa7-openshift-default-84586db7d5-sbpk8" Oct 30 20:49:57.831697 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:49:57.831347 5378 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2391a719-0c9b-4c56-b229-8f286a9aac96-istiod-ca-cert\") pod \"e2e-worker-w4-1fdaa7-openshift-default-84586db7d5-sbpk8\" (UID: \"2391a719-0c9b-4c56-b229-8f286a9aac96\") " pod="openshift-ingress/e2e-worker-w4-1fdaa7-openshift-default-84586db7d5-sbpk8" Oct 30 20:49:57.834601 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:49:57.833186 5378 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2391a719-0c9b-4c56-b229-8f286a9aac96-istio-envoy\") pod \"e2e-worker-w4-1fdaa7-openshift-default-84586db7d5-sbpk8\" (UID: \"2391a719-0c9b-4c56-b229-8f286a9aac96\") " pod="openshift-ingress/e2e-worker-w4-1fdaa7-openshift-default-84586db7d5-sbpk8" Oct 30 20:49:57.839360 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:49:57.839326 5378 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2391a719-0c9b-4c56-b229-8f286a9aac96-istio-podinfo\") pod \"e2e-worker-w4-1fdaa7-openshift-default-84586db7d5-sbpk8\" (UID: \"2391a719-0c9b-4c56-b229-8f286a9aac96\") " pod="openshift-ingress/e2e-worker-w4-1fdaa7-openshift-default-84586db7d5-sbpk8" Oct 30 20:49:57.892247 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:49:57.892214 5378 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6fq99\" (UniqueName: \"kubernetes.io/projected/2391a719-0c9b-4c56-b229-8f286a9aac96-kube-api-access-6fq99\") pod \"e2e-worker-w4-1fdaa7-openshift-default-84586db7d5-sbpk8\" (UID: \"2391a719-0c9b-4c56-b229-8f286a9aac96\") " pod="openshift-ingress/e2e-worker-w4-1fdaa7-openshift-default-84586db7d5-sbpk8" Oct 30 20:49:57.897580 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:49:57.897540 5378 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2391a719-0c9b-4c56-b229-8f286a9aac96-istio-token\") pod \"e2e-worker-w4-1fdaa7-openshift-default-84586db7d5-sbpk8\" (UID: \"2391a719-0c9b-4c56-b229-8f286a9aac96\") " pod="openshift-ingress/e2e-worker-w4-1fdaa7-openshift-default-84586db7d5-sbpk8" Oct 30 20:49:57.984266 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:49:57.984199 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 20:49:58.042901 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:49:58.042870 5378 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-1fdaa7-openshift-default-84586db7d5-sbpk8" Oct 30 20:49:58.165665 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:49:58.165624 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 20:49:58.378373 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:49:58.378345 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 20:49:58.442669 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:49:58.442638 5378 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-682999-openshift-default-54dcb797b5-r8dcv" event={"ID":"9b76fbb2-f5a8-4fea-bdfd-30f40bee7008","Type":"ContainerStarted","Data":"f371283ad7cbbc8f6a41616e270dfc8087188b60af99301046ead6a2cc9070d9"} Oct 30 20:49:58.442892 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:49:58.442868 5378 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-682999-openshift-default-54dcb797b5-r8dcv" event={"ID":"9b76fbb2-f5a8-4fea-bdfd-30f40bee7008","Type":"ContainerStarted","Data":"8bc12fb5ff424f5a3e57cd4a0f2736b0b620bb112edb9633333d6ec65b2afd7b"} Oct 30 20:49:58.491521 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:49:58.491475 5378 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-1fdaa7-openshift-default-84586db7d5-sbpk8"] Oct 30 20:49:58.493588 ip-10-0-138-149 kubenswrapper[5378]: W1030 20:49:58.493541 5378 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2391a719_0c9b_4c56_b229_8f286a9aac96.slice/crio-ff9f5edb579cceae20821e3cd210956f935a877232b79d3769f6712512de9b7e WatchSource:0}: Error finding container ff9f5edb579cceae20821e3cd210956f935a877232b79d3769f6712512de9b7e: Status 404 returned error can't find the container with id ff9f5edb579cceae20821e3cd210956f935a877232b79d3769f6712512de9b7e Oct 30 20:49:58.496002 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:49:58.495971 5378 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 20:49:58.496119 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:49:58.496068 5378 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 20:49:58.496187 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:49:58.496118 5378 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 20:49:58.544073 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:49:58.544038 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 20:49:58.568291 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:49:58.568240 5378 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-682999-openshift-default-54dcb797b5-r8dcv" podStartSLOduration=2.568227896 podStartE2EDuration="2.568227896s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:49:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:49:58.566691529 +0000 UTC m=+688.994805451" watchObservedRunningTime="2026-10-30 20:49:58.568227896 +0000 UTC m=+688.996341814" Oct 30 20:49:58.610989 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:49:58.610950 5378 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-f9f277-openshift-default-869d58fb77-p2wtm" Oct 30 20:49:58.943511 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:49:58.943478 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 20:49:59.299591 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:49:59.299491 5378 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-682999-openshift-default-54dcb797b5-r8dcv" Oct 30 20:49:59.304427 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:49:59.304404 5378 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-682999-openshift-default-54dcb797b5-r8dcv" Oct 30 20:49:59.448335 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:49:59.448297 5378 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-1fdaa7-openshift-default-84586db7d5-sbpk8" event={"ID":"2391a719-0c9b-4c56-b229-8f286a9aac96","Type":"ContainerStarted","Data":"4c4e053ee467b0b8deb3acae00ce4858327a6294f588d162c000012246a5364b"} Oct 30 20:49:59.448529 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:49:59.448359 5378 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-1fdaa7-openshift-default-84586db7d5-sbpk8" event={"ID":"2391a719-0c9b-4c56-b229-8f286a9aac96","Type":"ContainerStarted","Data":"ff9f5edb579cceae20821e3cd210956f935a877232b79d3769f6712512de9b7e"} Oct 30 20:49:59.448763 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:49:59.448737 5378 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-f9f277-openshift-default-869d58fb77-p2wtm" Oct 30 20:49:59.448945 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:49:59.448923 5378 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-682999-openshift-default-54dcb797b5-r8dcv" Oct 30 20:49:59.449670 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:49:59.449646 5378 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-682999-openshift-default-54dcb797b5-r8dcv" Oct 30 20:49:59.449785 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:49:59.449724 5378 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-f9f277-openshift-default-869d58fb77-p2wtm" Oct 30 20:49:59.588726 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:49:59.588677 5378 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-1fdaa7-openshift-default-84586db7d5-sbpk8" podStartSLOduration=2.5886656009999998 podStartE2EDuration="2.588665601s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:49:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:49:59.587108534 +0000 UTC m=+690.015222478" watchObservedRunningTime="2026-10-30 20:49:59.588665601 +0000 UTC m=+690.016779519" Oct 30 20:50:00.044086 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:50:00.044048 5378 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-1fdaa7-openshift-default-84586db7d5-sbpk8" Oct 30 20:50:00.049254 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:50:00.049230 5378 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-1fdaa7-openshift-default-84586db7d5-sbpk8" Oct 30 20:50:00.453634 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:50:00.453599 5378 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-1fdaa7-openshift-default-84586db7d5-sbpk8" Oct 30 20:50:00.454374 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:50:00.454352 5378 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-1fdaa7-openshift-default-84586db7d5-sbpk8" Oct 30 20:50:00.813249 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:50:00.813178 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 20:50:01.037083 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:50:01.037050 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 20:50:01.211779 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:50:01.211746 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 20:50:01.988425 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:50:01.988396 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 20:50:02.230446 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:50:02.230407 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 20:50:02.454478 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:50:02.454423 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 20:50:02.638882 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:50:02.638849 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 20:50:02.811778 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:50:02.811687 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 20:50:02.983307 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:50:02.983278 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 20:50:04.568442 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:50:04.568408 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 20:50:04.830812 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:50:04.830741 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 20:50:05.122830 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:50:05.122796 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 20:50:05.277165 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:50:05.277131 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 20:50:05.471915 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:50:05.471831 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 20:50:05.579181 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:50:05.579150 5378 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w1-f9f277-68cfd5cbbc-bz22r"] Oct 30 20:50:05.582819 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:50:05.582803 5378 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-f9f277-68cfd5cbbc-bz22r" Oct 30 20:50:05.623471 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:50:05.623444 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 20:50:05.639924 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:50:05.639902 5378 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w1-f9f277\"" Oct 30 20:50:05.695962 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:50:05.695911 5378 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w1-f9f277-dockercfg-x5pq7\"" Oct 30 20:50:05.698364 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:50:05.698332 5378 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w1-f9f277-68cfd5cbbc-bz22r"] Oct 30 20:50:05.714386 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:50:05.714100 5378 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/76bab521-3612-401b-a99a-8e8be187baef-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w1-f9f277-68cfd5cbbc-bz22r\" (UID: \"76bab521-3612-401b-a99a-8e8be187baef\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-f9f277-68cfd5cbbc-bz22r" Oct 30 20:50:05.714386 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:50:05.714235 5378 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2k45l\" (UniqueName: \"kubernetes.io/projected/76bab521-3612-401b-a99a-8e8be187baef-kube-api-access-2k45l\") pod \"payload-pre-processing-e2e-worker-w1-f9f277-68cfd5cbbc-bz22r\" (UID: \"76bab521-3612-401b-a99a-8e8be187baef\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-f9f277-68cfd5cbbc-bz22r" Oct 30 20:50:05.730388 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:50:05.730323 5378 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w1-f9f277-cf5c8b8bb-st9wf"] Oct 30 20:50:05.734159 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:50:05.734139 5378 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w1-f9f277-cf5c8b8bb-st9wf" Oct 30 20:50:05.791820 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:50:05.791793 5378 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w1-f9f277-cf5c8b8bb-st9wf"] Oct 30 20:50:05.814981 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:50:05.814948 5378 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fs2fl\" (UniqueName: \"kubernetes.io/projected/c360b269-e417-49b5-9482-34e06924da36-kube-api-access-fs2fl\") pod \"payload-processing-e2e-worker-w1-f9f277-cf5c8b8bb-st9wf\" (UID: \"c360b269-e417-49b5-9482-34e06924da36\") " pod="openshift-ingress/payload-processing-e2e-worker-w1-f9f277-cf5c8b8bb-st9wf" Oct 30 20:50:05.815100 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:50:05.814991 5378 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/76bab521-3612-401b-a99a-8e8be187baef-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w1-f9f277-68cfd5cbbc-bz22r\" (UID: \"76bab521-3612-401b-a99a-8e8be187baef\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-f9f277-68cfd5cbbc-bz22r" Oct 30 20:50:05.815143 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:50:05.815090 5378 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c360b269-e417-49b5-9482-34e06924da36-plugins-config-volume\") pod \"payload-processing-e2e-worker-w1-f9f277-cf5c8b8bb-st9wf\" (UID: \"c360b269-e417-49b5-9482-34e06924da36\") " pod="openshift-ingress/payload-processing-e2e-worker-w1-f9f277-cf5c8b8bb-st9wf" Oct 30 20:50:05.815195 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:50:05.815149 5378 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2k45l\" (UniqueName: \"kubernetes.io/projected/76bab521-3612-401b-a99a-8e8be187baef-kube-api-access-2k45l\") pod \"payload-pre-processing-e2e-worker-w1-f9f277-68cfd5cbbc-bz22r\" (UID: \"76bab521-3612-401b-a99a-8e8be187baef\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-f9f277-68cfd5cbbc-bz22r" Oct 30 20:50:05.815701 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:50:05.815682 5378 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/76bab521-3612-401b-a99a-8e8be187baef-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w1-f9f277-68cfd5cbbc-bz22r\" (UID: \"76bab521-3612-401b-a99a-8e8be187baef\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-f9f277-68cfd5cbbc-bz22r" Oct 30 20:50:05.873936 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:50:05.873907 5378 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2k45l\" (UniqueName: \"kubernetes.io/projected/76bab521-3612-401b-a99a-8e8be187baef-kube-api-access-2k45l\") pod \"payload-pre-processing-e2e-worker-w1-f9f277-68cfd5cbbc-bz22r\" (UID: \"76bab521-3612-401b-a99a-8e8be187baef\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-f9f277-68cfd5cbbc-bz22r" Oct 30 20:50:05.895109 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:50:05.895081 5378 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-f9f277-68cfd5cbbc-bz22r" Oct 30 20:50:05.906510 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:50:05.906485 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 20:50:05.916691 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:50:05.916666 5378 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c360b269-e417-49b5-9482-34e06924da36-plugins-config-volume\") pod \"payload-processing-e2e-worker-w1-f9f277-cf5c8b8bb-st9wf\" (UID: \"c360b269-e417-49b5-9482-34e06924da36\") " pod="openshift-ingress/payload-processing-e2e-worker-w1-f9f277-cf5c8b8bb-st9wf" Oct 30 20:50:05.916819 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:50:05.916799 5378 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fs2fl\" (UniqueName: \"kubernetes.io/projected/c360b269-e417-49b5-9482-34e06924da36-kube-api-access-fs2fl\") pod \"payload-processing-e2e-worker-w1-f9f277-cf5c8b8bb-st9wf\" (UID: \"c360b269-e417-49b5-9482-34e06924da36\") " pod="openshift-ingress/payload-processing-e2e-worker-w1-f9f277-cf5c8b8bb-st9wf" Oct 30 20:50:05.917719 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:50:05.917697 5378 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c360b269-e417-49b5-9482-34e06924da36-plugins-config-volume\") pod \"payload-processing-e2e-worker-w1-f9f277-cf5c8b8bb-st9wf\" (UID: \"c360b269-e417-49b5-9482-34e06924da36\") " pod="openshift-ingress/payload-processing-e2e-worker-w1-f9f277-cf5c8b8bb-st9wf" Oct 30 20:50:05.978254 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:50:05.978210 5378 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fs2fl\" (UniqueName: \"kubernetes.io/projected/c360b269-e417-49b5-9482-34e06924da36-kube-api-access-fs2fl\") pod \"payload-processing-e2e-worker-w1-f9f277-cf5c8b8bb-st9wf\" (UID: \"c360b269-e417-49b5-9482-34e06924da36\") " pod="openshift-ingress/payload-processing-e2e-worker-w1-f9f277-cf5c8b8bb-st9wf" Oct 30 20:50:06.047511 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:50:06.047445 5378 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w1-f9f277-cf5c8b8bb-st9wf" Oct 30 20:50:06.118028 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:50:06.118002 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 20:50:06.267348 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:50:06.267301 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 20:50:06.274906 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:50:06.274872 5378 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w1-f9f277-68cfd5cbbc-bz22r"] Oct 30 20:50:06.277318 ip-10-0-138-149 kubenswrapper[5378]: W1030 20:50:06.277288 5378 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod76bab521_3612_401b_a99a_8e8be187baef.slice/crio-12abefd12686e7ca0b2ff5e26f696f601eef2a0c4d7930f22720a4c6f88f7cc7 WatchSource:0}: Error finding container 12abefd12686e7ca0b2ff5e26f696f601eef2a0c4d7930f22720a4c6f88f7cc7: Status 404 returned error can't find the container with id 12abefd12686e7ca0b2ff5e26f696f601eef2a0c4d7930f22720a4c6f88f7cc7 Oct 30 20:50:06.419810 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:50:06.419786 5378 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w1-f9f277-cf5c8b8bb-st9wf"] Oct 30 20:50:06.421900 ip-10-0-138-149 kubenswrapper[5378]: W1030 20:50:06.421876 5378 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc360b269_e417_49b5_9482_34e06924da36.slice/crio-3952316435df6a8cf30b4e17a86b1216ac2389cfc6216f21059889374f939d17 WatchSource:0}: Error finding container 3952316435df6a8cf30b4e17a86b1216ac2389cfc6216f21059889374f939d17: Status 404 returned error can't find the container with id 3952316435df6a8cf30b4e17a86b1216ac2389cfc6216f21059889374f939d17 Oct 30 20:50:06.486639 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:50:06.486608 5378 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-f9f277-68cfd5cbbc-bz22r" event={"ID":"76bab521-3612-401b-a99a-8e8be187baef","Type":"ContainerStarted","Data":"8768a90ec84008cf853ee0a21042fb377db792be4f738b5293ea1577e3bf0022"} Oct 30 20:50:06.486639 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:50:06.486672 5378 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-f9f277-68cfd5cbbc-bz22r" event={"ID":"76bab521-3612-401b-a99a-8e8be187baef","Type":"ContainerStarted","Data":"12abefd12686e7ca0b2ff5e26f696f601eef2a0c4d7930f22720a4c6f88f7cc7"} Oct 30 20:50:06.487071 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:50:06.486822 5378 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-f9f277-68cfd5cbbc-bz22r" Oct 30 20:50:06.488834 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:50:06.488812 5378 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w1-f9f277-cf5c8b8bb-st9wf" event={"ID":"c360b269-e417-49b5-9482-34e06924da36","Type":"ContainerStarted","Data":"1daf7b461607444f74486b20584ab7362cb4557e5de900fcb167398d1572f567"} Oct 30 20:50:06.488923 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:50:06.488861 5378 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w1-f9f277-cf5c8b8bb-st9wf" Oct 30 20:50:06.488923 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:50:06.488894 5378 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w1-f9f277-cf5c8b8bb-st9wf" event={"ID":"c360b269-e417-49b5-9482-34e06924da36","Type":"ContainerStarted","Data":"3952316435df6a8cf30b4e17a86b1216ac2389cfc6216f21059889374f939d17"} Oct 30 20:50:06.608938 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:50:06.608889 5378 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-f9f277-68cfd5cbbc-bz22r" podStartSLOduration=1.608875531 podStartE2EDuration="1.608875531s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:50:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:50:06.607851372 +0000 UTC m=+697.035965418" watchObservedRunningTime="2026-10-30 20:50:06.608875531 +0000 UTC m=+697.036989449" Oct 30 20:50:06.735597 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:50:06.735508 5378 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w1-f9f277-cf5c8b8bb-st9wf" podStartSLOduration=1.735485942 podStartE2EDuration="1.735485942s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:50:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:50:06.73119649 +0000 UTC m=+697.159310412" watchObservedRunningTime="2026-10-30 20:50:06.735485942 +0000 UTC m=+697.163599862" Oct 30 20:50:07.951414 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:50:07.951366 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 20:50:08.094505 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:50:08.094435 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 20:50:09.518922 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:50:09.518890 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 20:50:09.737698 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:50:09.737671 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 20:50:09.899103 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:50:09.899067 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 20:50:10.277876 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:50:10.277772 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 20:50:10.460453 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:50:10.460423 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 20:50:10.811429 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:50:10.811392 5378 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w3-682999-69c9476f4b-qwh92"] Oct 30 20:50:10.815490 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:50:10.815471 5378 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w3-682999-69c9476f4b-qwh92" Oct 30 20:50:10.867935 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:50:10.867907 5378 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w3-682999-69c9476f4b-qwh92"] Oct 30 20:50:10.916625 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:50:10.916595 5378 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w3-682999-dockercfg-lphff\"" Oct 30 20:50:10.916749 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:50:10.916637 5378 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w3-682999\"" Oct 30 20:50:10.974444 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:50:10.974414 5378 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6351b682-cca8-4a28-bd5f-9cb3fd38085b-plugins-config-volume\") pod \"payload-processing-e2e-worker-w3-682999-69c9476f4b-qwh92\" (UID: \"6351b682-cca8-4a28-bd5f-9cb3fd38085b\") " pod="openshift-ingress/payload-processing-e2e-worker-w3-682999-69c9476f4b-qwh92" Oct 30 20:50:10.974603 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:50:10.974470 5378 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-984sn\" (UniqueName: \"kubernetes.io/projected/6351b682-cca8-4a28-bd5f-9cb3fd38085b-kube-api-access-984sn\") pod \"payload-processing-e2e-worker-w3-682999-69c9476f4b-qwh92\" (UID: \"6351b682-cca8-4a28-bd5f-9cb3fd38085b\") " pod="openshift-ingress/payload-processing-e2e-worker-w3-682999-69c9476f4b-qwh92" Oct 30 20:50:11.075293 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:50:11.075210 5378 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6351b682-cca8-4a28-bd5f-9cb3fd38085b-plugins-config-volume\") pod \"payload-processing-e2e-worker-w3-682999-69c9476f4b-qwh92\" (UID: \"6351b682-cca8-4a28-bd5f-9cb3fd38085b\") " pod="openshift-ingress/payload-processing-e2e-worker-w3-682999-69c9476f4b-qwh92" Oct 30 20:50:11.075293 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:50:11.075256 5378 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-984sn\" (UniqueName: \"kubernetes.io/projected/6351b682-cca8-4a28-bd5f-9cb3fd38085b-kube-api-access-984sn\") pod \"payload-processing-e2e-worker-w3-682999-69c9476f4b-qwh92\" (UID: \"6351b682-cca8-4a28-bd5f-9cb3fd38085b\") " pod="openshift-ingress/payload-processing-e2e-worker-w3-682999-69c9476f4b-qwh92" Oct 30 20:50:11.075952 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:50:11.075931 5378 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6351b682-cca8-4a28-bd5f-9cb3fd38085b-plugins-config-volume\") pod \"payload-processing-e2e-worker-w3-682999-69c9476f4b-qwh92\" (UID: \"6351b682-cca8-4a28-bd5f-9cb3fd38085b\") " pod="openshift-ingress/payload-processing-e2e-worker-w3-682999-69c9476f4b-qwh92" Oct 30 20:50:11.134345 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:50:11.134310 5378 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-984sn\" (UniqueName: \"kubernetes.io/projected/6351b682-cca8-4a28-bd5f-9cb3fd38085b-kube-api-access-984sn\") pod \"payload-processing-e2e-worker-w3-682999-69c9476f4b-qwh92\" (UID: \"6351b682-cca8-4a28-bd5f-9cb3fd38085b\") " pod="openshift-ingress/payload-processing-e2e-worker-w3-682999-69c9476f4b-qwh92" Oct 30 20:50:11.429487 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:50:11.429448 5378 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w3-682999-69c9476f4b-qwh92" Oct 30 20:50:11.800841 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:50:11.800818 5378 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w3-682999-69c9476f4b-qwh92"] Oct 30 20:50:11.802529 ip-10-0-138-149 kubenswrapper[5378]: W1030 20:50:11.802503 5378 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6351b682_cca8_4a28_bd5f_9cb3fd38085b.slice/crio-ce1aa027617321004034438c2f0b0e52ca3dc0e2b6e598bedd61bc654ff4df02 WatchSource:0}: Error finding container ce1aa027617321004034438c2f0b0e52ca3dc0e2b6e598bedd61bc654ff4df02: Status 404 returned error can't find the container with id ce1aa027617321004034438c2f0b0e52ca3dc0e2b6e598bedd61bc654ff4df02 Oct 30 20:50:12.521792 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:50:12.521757 5378 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w3-682999-69c9476f4b-qwh92" event={"ID":"6351b682-cca8-4a28-bd5f-9cb3fd38085b","Type":"ContainerStarted","Data":"90865cd40bf7485e78d246a2c67dff0b0be5544b98fb6da93b2f2f0bb08c7727"} Oct 30 20:50:12.522200 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:50:12.521822 5378 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w3-682999-69c9476f4b-qwh92" Oct 30 20:50:12.522200 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:50:12.521843 5378 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w3-682999-69c9476f4b-qwh92" event={"ID":"6351b682-cca8-4a28-bd5f-9cb3fd38085b","Type":"ContainerStarted","Data":"ce1aa027617321004034438c2f0b0e52ca3dc0e2b6e598bedd61bc654ff4df02"} Oct 30 20:50:12.840248 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:50:12.840165 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 20:50:12.993266 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:50:12.993236 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 20:50:13.227799 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:50:13.227723 5378 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w3-682999-69c9476f4b-qwh92" podStartSLOduration=3.227700244 podStartE2EDuration="3.227700244s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:50:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:50:12.670653664 +0000 UTC m=+703.098767583" watchObservedRunningTime="2026-10-30 20:50:13.227700244 +0000 UTC m=+703.655814164" Oct 30 20:50:13.231837 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:50:13.229445 5378 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w5-2b425d-59c9dc557f-sd78q"] Oct 30 20:50:13.235431 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:50:13.235408 5378 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-2b425d-59c9dc557f-sd78q" Oct 30 20:50:13.288484 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:50:13.288454 5378 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w5-2b425d-dockercfg-frp9b\"" Oct 30 20:50:13.288793 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:50:13.288774 5378 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w5-2b425d\"" Oct 30 20:50:13.324687 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:50:13.321199 5378 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vcs7j\" (UniqueName: \"kubernetes.io/projected/7715f68b-9673-4119-a43d-05372a096116-kube-api-access-vcs7j\") pod \"payload-pre-processing-e2e-worker-w5-2b425d-59c9dc557f-sd78q\" (UID: \"7715f68b-9673-4119-a43d-05372a096116\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-2b425d-59c9dc557f-sd78q" Oct 30 20:50:13.324687 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:50:13.321314 5378 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7715f68b-9673-4119-a43d-05372a096116-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w5-2b425d-59c9dc557f-sd78q\" (UID: \"7715f68b-9673-4119-a43d-05372a096116\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-2b425d-59c9dc557f-sd78q" Oct 30 20:50:13.343759 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:50:13.343721 5378 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w5-2b425d-59c9dc557f-sd78q"] Oct 30 20:50:13.420805 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:50:13.420776 5378 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w5-2b425d-74d89cbc4b-rdcqc"] Oct 30 20:50:13.422551 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:50:13.422524 5378 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vcs7j\" (UniqueName: \"kubernetes.io/projected/7715f68b-9673-4119-a43d-05372a096116-kube-api-access-vcs7j\") pod \"payload-pre-processing-e2e-worker-w5-2b425d-59c9dc557f-sd78q\" (UID: \"7715f68b-9673-4119-a43d-05372a096116\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-2b425d-59c9dc557f-sd78q" Oct 30 20:50:13.422691 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:50:13.422611 5378 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7715f68b-9673-4119-a43d-05372a096116-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w5-2b425d-59c9dc557f-sd78q\" (UID: \"7715f68b-9673-4119-a43d-05372a096116\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-2b425d-59c9dc557f-sd78q" Oct 30 20:50:13.423274 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:50:13.423250 5378 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7715f68b-9673-4119-a43d-05372a096116-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w5-2b425d-59c9dc557f-sd78q\" (UID: \"7715f68b-9673-4119-a43d-05372a096116\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-2b425d-59c9dc557f-sd78q" Oct 30 20:50:13.424528 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:50:13.424512 5378 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w5-2b425d-74d89cbc4b-rdcqc" Oct 30 20:50:13.453071 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:50:13.453034 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 20:50:13.482974 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:50:13.482919 5378 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w5-2b425d-74d89cbc4b-rdcqc"] Oct 30 20:50:13.485682 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:50:13.485660 5378 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vcs7j\" (UniqueName: \"kubernetes.io/projected/7715f68b-9673-4119-a43d-05372a096116-kube-api-access-vcs7j\") pod \"payload-pre-processing-e2e-worker-w5-2b425d-59c9dc557f-sd78q\" (UID: \"7715f68b-9673-4119-a43d-05372a096116\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-2b425d-59c9dc557f-sd78q" Oct 30 20:50:13.523226 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:50:13.523200 5378 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2d52d866-9725-4d46-9aa5-adcc43e87bbd-plugins-config-volume\") pod \"payload-processing-e2e-worker-w5-2b425d-74d89cbc4b-rdcqc\" (UID: \"2d52d866-9725-4d46-9aa5-adcc43e87bbd\") " pod="openshift-ingress/payload-processing-e2e-worker-w5-2b425d-74d89cbc4b-rdcqc" Oct 30 20:50:13.523560 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:50:13.523252 5378 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9lrmh\" (UniqueName: \"kubernetes.io/projected/2d52d866-9725-4d46-9aa5-adcc43e87bbd-kube-api-access-9lrmh\") pod \"payload-processing-e2e-worker-w5-2b425d-74d89cbc4b-rdcqc\" (UID: \"2d52d866-9725-4d46-9aa5-adcc43e87bbd\") " pod="openshift-ingress/payload-processing-e2e-worker-w5-2b425d-74d89cbc4b-rdcqc" Oct 30 20:50:13.549504 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:50:13.549480 5378 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-2b425d-59c9dc557f-sd78q" Oct 30 20:50:13.605836 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:50:13.605059 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 20:50:13.624243 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:50:13.624217 5378 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9lrmh\" (UniqueName: \"kubernetes.io/projected/2d52d866-9725-4d46-9aa5-adcc43e87bbd-kube-api-access-9lrmh\") pod \"payload-processing-e2e-worker-w5-2b425d-74d89cbc4b-rdcqc\" (UID: \"2d52d866-9725-4d46-9aa5-adcc43e87bbd\") " pod="openshift-ingress/payload-processing-e2e-worker-w5-2b425d-74d89cbc4b-rdcqc" Oct 30 20:50:13.624415 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:50:13.624390 5378 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2d52d866-9725-4d46-9aa5-adcc43e87bbd-plugins-config-volume\") pod \"payload-processing-e2e-worker-w5-2b425d-74d89cbc4b-rdcqc\" (UID: \"2d52d866-9725-4d46-9aa5-adcc43e87bbd\") " pod="openshift-ingress/payload-processing-e2e-worker-w5-2b425d-74d89cbc4b-rdcqc" Oct 30 20:50:13.625093 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:50:13.625071 5378 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2d52d866-9725-4d46-9aa5-adcc43e87bbd-plugins-config-volume\") pod \"payload-processing-e2e-worker-w5-2b425d-74d89cbc4b-rdcqc\" (UID: \"2d52d866-9725-4d46-9aa5-adcc43e87bbd\") " pod="openshift-ingress/payload-processing-e2e-worker-w5-2b425d-74d89cbc4b-rdcqc" Oct 30 20:50:13.683478 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:50:13.683445 5378 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9lrmh\" (UniqueName: \"kubernetes.io/projected/2d52d866-9725-4d46-9aa5-adcc43e87bbd-kube-api-access-9lrmh\") pod \"payload-processing-e2e-worker-w5-2b425d-74d89cbc4b-rdcqc\" (UID: \"2d52d866-9725-4d46-9aa5-adcc43e87bbd\") " pod="openshift-ingress/payload-processing-e2e-worker-w5-2b425d-74d89cbc4b-rdcqc" Oct 30 20:50:13.737592 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:50:13.737501 5378 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w5-2b425d-74d89cbc4b-rdcqc" Oct 30 20:50:13.741270 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:50:13.741238 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 20:50:13.922345 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:50:13.922319 5378 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w5-2b425d-59c9dc557f-sd78q"] Oct 30 20:50:13.923658 ip-10-0-138-149 kubenswrapper[5378]: W1030 20:50:13.923633 5378 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7715f68b_9673_4119_a43d_05372a096116.slice/crio-a2f226d103b2f262c28ca9cb80de67103a06892eb0938155e3068f68ac6cc8d3 WatchSource:0}: Error finding container a2f226d103b2f262c28ca9cb80de67103a06892eb0938155e3068f68ac6cc8d3: Status 404 returned error can't find the container with id a2f226d103b2f262c28ca9cb80de67103a06892eb0938155e3068f68ac6cc8d3 Oct 30 20:50:14.109270 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:50:14.109240 5378 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w5-2b425d-74d89cbc4b-rdcqc"] Oct 30 20:50:14.110420 ip-10-0-138-149 kubenswrapper[5378]: W1030 20:50:14.110398 5378 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2d52d866_9725_4d46_9aa5_adcc43e87bbd.slice/crio-127a3deaf6fc4aee396aee0e233b8b59463ba110ab61995339f74d392d23e0a3 WatchSource:0}: Error finding container 127a3deaf6fc4aee396aee0e233b8b59463ba110ab61995339f74d392d23e0a3: Status 404 returned error can't find the container with id 127a3deaf6fc4aee396aee0e233b8b59463ba110ab61995339f74d392d23e0a3 Oct 30 20:50:14.457802 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:50:14.457730 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 20:50:14.533642 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:50:14.533598 5378 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w5-2b425d-74d89cbc4b-rdcqc" event={"ID":"2d52d866-9725-4d46-9aa5-adcc43e87bbd","Type":"ContainerStarted","Data":"2bb51a5abb388bb86b1f21c45ba6d808e1a095613b591fbaa4c4213cffa4d783"} Oct 30 20:50:14.534073 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:50:14.533671 5378 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w5-2b425d-74d89cbc4b-rdcqc" event={"ID":"2d52d866-9725-4d46-9aa5-adcc43e87bbd","Type":"ContainerStarted","Data":"127a3deaf6fc4aee396aee0e233b8b59463ba110ab61995339f74d392d23e0a3"} Oct 30 20:50:14.534073 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:50:14.533701 5378 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w5-2b425d-74d89cbc4b-rdcqc" Oct 30 20:50:14.535128 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:50:14.535106 5378 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-2b425d-59c9dc557f-sd78q" event={"ID":"7715f68b-9673-4119-a43d-05372a096116","Type":"ContainerStarted","Data":"984fb00d89aa672a0cf010cd330a7da88d6a067932dfb5b87c0f9f5843f90b01"} Oct 30 20:50:14.535242 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:50:14.535143 5378 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-2b425d-59c9dc557f-sd78q" event={"ID":"7715f68b-9673-4119-a43d-05372a096116","Type":"ContainerStarted","Data":"a2f226d103b2f262c28ca9cb80de67103a06892eb0938155e3068f68ac6cc8d3"} Oct 30 20:50:14.535279 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:50:14.535243 5378 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-2b425d-59c9dc557f-sd78q" Oct 30 20:50:14.653778 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:50:14.653730 5378 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w5-2b425d-74d89cbc4b-rdcqc" podStartSLOduration=1.653715949 podStartE2EDuration="1.653715949s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:50:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:50:14.651150622 +0000 UTC m=+705.079264563" watchObservedRunningTime="2026-10-30 20:50:14.653715949 +0000 UTC m=+705.081829867" Oct 30 20:50:14.770461 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:50:14.770364 5378 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-2b425d-59c9dc557f-sd78q" podStartSLOduration=1.7703501959999999 podStartE2EDuration="1.770350196s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:50:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:50:14.769234789 +0000 UTC m=+705.197348709" watchObservedRunningTime="2026-10-30 20:50:14.770350196 +0000 UTC m=+705.198464114" Oct 30 20:50:16.190379 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:50:16.190340 5378 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-r9g59"] Oct 30 20:50:17.496138 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:50:17.496109 5378 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w1-f9f277-cf5c8b8bb-st9wf" Oct 30 20:50:17.496577 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:50:17.496187 5378 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-f9f277-68cfd5cbbc-bz22r" Oct 30 20:50:17.785101 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:50:17.785019 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 20:50:17.984284 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:50:17.984250 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 20:50:18.348972 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:50:18.348939 5378 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-781058-6d5676f8cb-fkrgx"] Oct 30 20:50:18.352756 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:50:18.352730 5378 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w0-781058-6d5676f8cb-fkrgx" Oct 30 20:50:18.399288 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:50:18.399256 5378 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9b4bb6e8-85e8-44a9-aaaa-485f6c6b0b79-plugins-config-volume\") pod \"payload-processing-e2e-worker-w0-781058-6d5676f8cb-fkrgx\" (UID: \"9b4bb6e8-85e8-44a9-aaaa-485f6c6b0b79\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-781058-6d5676f8cb-fkrgx" Oct 30 20:50:18.399418 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:50:18.399302 5378 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jfk9j\" (UniqueName: \"kubernetes.io/projected/9b4bb6e8-85e8-44a9-aaaa-485f6c6b0b79-kube-api-access-jfk9j\") pod \"payload-processing-e2e-worker-w0-781058-6d5676f8cb-fkrgx\" (UID: \"9b4bb6e8-85e8-44a9-aaaa-485f6c6b0b79\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-781058-6d5676f8cb-fkrgx" Oct 30 20:50:18.410479 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:50:18.410451 5378 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-781058-6d5676f8cb-fkrgx"] Oct 30 20:50:18.461921 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:50:18.461870 5378 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w0-781058\"" Oct 30 20:50:18.462156 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:50:18.462139 5378 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w0-781058-dockercfg-84vjz\"" Oct 30 20:50:18.500816 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:50:18.500783 5378 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9b4bb6e8-85e8-44a9-aaaa-485f6c6b0b79-plugins-config-volume\") pod \"payload-processing-e2e-worker-w0-781058-6d5676f8cb-fkrgx\" (UID: \"9b4bb6e8-85e8-44a9-aaaa-485f6c6b0b79\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-781058-6d5676f8cb-fkrgx" Oct 30 20:50:18.501148 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:50:18.500829 5378 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jfk9j\" (UniqueName: \"kubernetes.io/projected/9b4bb6e8-85e8-44a9-aaaa-485f6c6b0b79-kube-api-access-jfk9j\") pod \"payload-processing-e2e-worker-w0-781058-6d5676f8cb-fkrgx\" (UID: \"9b4bb6e8-85e8-44a9-aaaa-485f6c6b0b79\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-781058-6d5676f8cb-fkrgx" Oct 30 20:50:18.501453 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:50:18.501434 5378 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9b4bb6e8-85e8-44a9-aaaa-485f6c6b0b79-plugins-config-volume\") pod \"payload-processing-e2e-worker-w0-781058-6d5676f8cb-fkrgx\" (UID: \"9b4bb6e8-85e8-44a9-aaaa-485f6c6b0b79\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-781058-6d5676f8cb-fkrgx" Oct 30 20:50:18.578270 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:50:18.578230 5378 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jfk9j\" (UniqueName: \"kubernetes.io/projected/9b4bb6e8-85e8-44a9-aaaa-485f6c6b0b79-kube-api-access-jfk9j\") pod \"payload-processing-e2e-worker-w0-781058-6d5676f8cb-fkrgx\" (UID: \"9b4bb6e8-85e8-44a9-aaaa-485f6c6b0b79\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-781058-6d5676f8cb-fkrgx" Oct 30 20:50:18.686191 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:50:18.686108 5378 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w0-781058-6d5676f8cb-fkrgx" Oct 30 20:50:18.836793 ip-10-0-138-149 kubenswrapper[5378]: E1030 20:50:18.836749 5378 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q6mx7" containerName="extractor" Oct 30 20:50:18.990395 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:50:18.990316 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 20:50:19.081283 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:50:19.081241 5378 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-781058-6d5676f8cb-fkrgx"] Oct 30 20:50:19.085397 ip-10-0-138-149 kubenswrapper[5378]: W1030 20:50:19.085369 5378 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9b4bb6e8_85e8_44a9_aaaa_485f6c6b0b79.slice/crio-a5af72ea280f3664443ae07c193dde3dfff2e2022e3277d9d3c283ece25ce541 WatchSource:0}: Error finding container a5af72ea280f3664443ae07c193dde3dfff2e2022e3277d9d3c283ece25ce541: Status 404 returned error can't find the container with id a5af72ea280f3664443ae07c193dde3dfff2e2022e3277d9d3c283ece25ce541 Oct 30 20:50:19.581750 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:50:19.581711 5378 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w0-781058-6d5676f8cb-fkrgx" event={"ID":"9b4bb6e8-85e8-44a9-aaaa-485f6c6b0b79","Type":"ContainerStarted","Data":"3e25b19e11cd021147005f7589ac25af8e75d13b095e83dd162a5d0e96347bb1"} Oct 30 20:50:19.582139 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:50:19.581785 5378 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w0-781058-6d5676f8cb-fkrgx" Oct 30 20:50:19.582139 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:50:19.581811 5378 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w0-781058-6d5676f8cb-fkrgx" event={"ID":"9b4bb6e8-85e8-44a9-aaaa-485f6c6b0b79","Type":"ContainerStarted","Data":"a5af72ea280f3664443ae07c193dde3dfff2e2022e3277d9d3c283ece25ce541"} Oct 30 20:50:19.699577 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:50:19.699522 5378 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w0-781058-6d5676f8cb-fkrgx" podStartSLOduration=1.6995064050000002 podStartE2EDuration="1.699506405s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:50:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:50:19.698448906 +0000 UTC m=+710.126562824" watchObservedRunningTime="2026-10-30 20:50:19.699506405 +0000 UTC m=+710.127620341" Oct 30 20:50:20.378172 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:50:20.378131 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 20:50:20.579110 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:50:20.579080 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 20:50:20.937433 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:50:20.937398 5378 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w6-8fa2a7-7d66df94fb-clj44"] Oct 30 20:50:20.941587 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:50:20.941537 5378 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w6-8fa2a7-7d66df94fb-clj44" Oct 30 20:50:21.026794 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:50:21.026762 5378 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/75677fa4-95cf-4a72-bced-a11307c20ca0-plugins-config-volume\") pod \"payload-processing-e2e-worker-w6-8fa2a7-7d66df94fb-clj44\" (UID: \"75677fa4-95cf-4a72-bced-a11307c20ca0\") " pod="openshift-ingress/payload-processing-e2e-worker-w6-8fa2a7-7d66df94fb-clj44" Oct 30 20:50:21.026794 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:50:21.026797 5378 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vptpm\" (UniqueName: \"kubernetes.io/projected/75677fa4-95cf-4a72-bced-a11307c20ca0-kube-api-access-vptpm\") pod \"payload-processing-e2e-worker-w6-8fa2a7-7d66df94fb-clj44\" (UID: \"75677fa4-95cf-4a72-bced-a11307c20ca0\") " pod="openshift-ingress/payload-processing-e2e-worker-w6-8fa2a7-7d66df94fb-clj44" Oct 30 20:50:21.047728 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:50:21.047697 5378 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w6-8fa2a7\"" Oct 30 20:50:21.047846 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:50:21.047822 5378 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w6-8fa2a7-dockercfg-tfglb\"" Oct 30 20:50:21.049603 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:50:21.049586 5378 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w6-8fa2a7-7d66df94fb-clj44"] Oct 30 20:50:21.127682 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:50:21.127651 5378 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vptpm\" (UniqueName: \"kubernetes.io/projected/75677fa4-95cf-4a72-bced-a11307c20ca0-kube-api-access-vptpm\") pod \"payload-processing-e2e-worker-w6-8fa2a7-7d66df94fb-clj44\" (UID: \"75677fa4-95cf-4a72-bced-a11307c20ca0\") " pod="openshift-ingress/payload-processing-e2e-worker-w6-8fa2a7-7d66df94fb-clj44" Oct 30 20:50:21.127849 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:50:21.127786 5378 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/75677fa4-95cf-4a72-bced-a11307c20ca0-plugins-config-volume\") pod \"payload-processing-e2e-worker-w6-8fa2a7-7d66df94fb-clj44\" (UID: \"75677fa4-95cf-4a72-bced-a11307c20ca0\") " pod="openshift-ingress/payload-processing-e2e-worker-w6-8fa2a7-7d66df94fb-clj44" Oct 30 20:50:21.128394 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:50:21.128377 5378 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/75677fa4-95cf-4a72-bced-a11307c20ca0-plugins-config-volume\") pod \"payload-processing-e2e-worker-w6-8fa2a7-7d66df94fb-clj44\" (UID: \"75677fa4-95cf-4a72-bced-a11307c20ca0\") " pod="openshift-ingress/payload-processing-e2e-worker-w6-8fa2a7-7d66df94fb-clj44" Oct 30 20:50:21.186437 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:50:21.186410 5378 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vptpm\" (UniqueName: \"kubernetes.io/projected/75677fa4-95cf-4a72-bced-a11307c20ca0-kube-api-access-vptpm\") pod \"payload-processing-e2e-worker-w6-8fa2a7-7d66df94fb-clj44\" (UID: \"75677fa4-95cf-4a72-bced-a11307c20ca0\") " pod="openshift-ingress/payload-processing-e2e-worker-w6-8fa2a7-7d66df94fb-clj44" Oct 30 20:50:21.280601 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:50:21.280505 5378 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w6-8fa2a7-7d66df94fb-clj44" Oct 30 20:50:21.672590 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:50:21.666688 5378 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w6-8fa2a7-7d66df94fb-clj44"] Oct 30 20:50:21.682078 ip-10-0-138-149 kubenswrapper[5378]: W1030 20:50:21.682051 5378 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod75677fa4_95cf_4a72_bced_a11307c20ca0.slice/crio-ff5e586d01159dad4d08397592dc0c2da8f845980b2f067dc1bf269b5d854f05 WatchSource:0}: Error finding container ff5e586d01159dad4d08397592dc0c2da8f845980b2f067dc1bf269b5d854f05: Status 404 returned error can't find the container with id ff5e586d01159dad4d08397592dc0c2da8f845980b2f067dc1bf269b5d854f05 Oct 30 20:50:22.599767 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:50:22.599729 5378 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w6-8fa2a7-7d66df94fb-clj44" event={"ID":"75677fa4-95cf-4a72-bced-a11307c20ca0","Type":"ContainerStarted","Data":"b480467aa93228bbf419066e938f3fbbe9902aa8bec8eaff1d83a615d4fa301a"} Oct 30 20:50:22.600221 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:50:22.599828 5378 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w6-8fa2a7-7d66df94fb-clj44" event={"ID":"75677fa4-95cf-4a72-bced-a11307c20ca0","Type":"ContainerStarted","Data":"ff5e586d01159dad4d08397592dc0c2da8f845980b2f067dc1bf269b5d854f05"} Oct 30 20:50:22.600221 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:50:22.599885 5378 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w6-8fa2a7-7d66df94fb-clj44" Oct 30 20:50:22.718572 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:50:22.718507 5378 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w6-8fa2a7-7d66df94fb-clj44" podStartSLOduration=2.718494541 podStartE2EDuration="2.718494541s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:50:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:50:22.716790806 +0000 UTC m=+713.144904726" watchObservedRunningTime="2026-10-30 20:50:22.718494541 +0000 UTC m=+713.146608460" Oct 30 20:50:23.258630 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:50:23.258590 5378 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-r9g59"] Oct 30 20:50:23.539080 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:50:23.538995 5378 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w3-682999-69c9476f4b-qwh92" Oct 30 20:50:24.459605 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:50:24.459553 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 20:50:24.618318 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:50:24.618281 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 20:50:24.757078 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:50:24.756990 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 20:50:25.543240 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:50:25.543198 5378 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-2b425d-59c9dc557f-sd78q" Oct 30 20:50:25.543764 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:50:25.543464 5378 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w5-2b425d-74d89cbc4b-rdcqc" Oct 30 20:50:26.623781 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:50:26.623744 5378 generic.go:356] "Generic (PLEG): container finished" podID="9b76fbb2-f5a8-4fea-bdfd-30f40bee7008" containerID="f371283ad7cbbc8f6a41616e270dfc8087188b60af99301046ead6a2cc9070d9" exitCode=0 Oct 30 20:50:26.624325 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:50:26.623789 5378 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-682999-openshift-default-54dcb797b5-r8dcv" event={"ID":"9b76fbb2-f5a8-4fea-bdfd-30f40bee7008","Type":"ContainerDied","Data":"f371283ad7cbbc8f6a41616e270dfc8087188b60af99301046ead6a2cc9070d9"} Oct 30 20:50:26.624325 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:50:26.624249 5378 scope.go:122] "RemoveContainer" containerID="f371283ad7cbbc8f6a41616e270dfc8087188b60af99301046ead6a2cc9070d9" Oct 30 20:50:26.625547 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:50:26.625517 5378 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 20:50:26.625708 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:50:26.625633 5378 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 20:50:26.625708 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:50:26.625682 5378 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 20:50:27.630955 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:50:27.630919 5378 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-682999-openshift-default-54dcb797b5-r8dcv" event={"ID":"9b76fbb2-f5a8-4fea-bdfd-30f40bee7008","Type":"ContainerStarted","Data":"7966cdde4c6c95b093f0557ff6527111242add1fb6bd6213a8d49c79eea93a0b"} Oct 30 20:50:27.740518 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:50:27.740490 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 20:50:27.921652 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:50:27.921573 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 20:50:28.302863 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:50:28.302135 5378 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-682999-openshift-default-54dcb797b5-r8dcv" Oct 30 20:50:28.305067 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:50:28.303473 5378 patch_prober.go:29] interesting pod/e2e-worker-w3-682999-openshift-default-54dcb797b5-r8dcv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.43:15021/healthz/ready\": dial tcp 10.133.0.43:15021: connect: connection refused" start-of-body= Oct 30 20:50:28.305067 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:50:28.303522 5378 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-682999-openshift-default-54dcb797b5-r8dcv" podUID="9b76fbb2-f5a8-4fea-bdfd-30f40bee7008" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.43:15021/healthz/ready\": dial tcp 10.133.0.43:15021: connect: connection refused" Oct 30 20:50:29.299795 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:50:29.299762 5378 patch_prober.go:29] interesting pod/e2e-worker-w3-682999-openshift-default-54dcb797b5-r8dcv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.43:15021/healthz/ready\": dial tcp 10.133.0.43:15021: connect: connection refused" start-of-body= Oct 30 20:50:29.300200 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:50:29.299827 5378 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-682999-openshift-default-54dcb797b5-r8dcv" podUID="9b76fbb2-f5a8-4fea-bdfd-30f40bee7008" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.43:15021/healthz/ready\": dial tcp 10.133.0.43:15021: connect: connection refused" Oct 30 20:50:29.449131 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:50:29.449095 5378 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-682999-openshift-default-54dcb797b5-r8dcv" Oct 30 20:50:29.779162 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:50:29.779129 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 20:50:29.941144 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:50:29.941112 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 20:50:30.074973 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:50:30.074851 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 20:50:30.299781 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:50:30.299747 5378 patch_prober.go:29] interesting pod/e2e-worker-w3-682999-openshift-default-54dcb797b5-r8dcv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.43:15021/healthz/ready\": dial tcp 10.133.0.43:15021: connect: connection refused" start-of-body= Oct 30 20:50:30.299989 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:50:30.299805 5378 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-682999-openshift-default-54dcb797b5-r8dcv" podUID="9b76fbb2-f5a8-4fea-bdfd-30f40bee7008" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.43:15021/healthz/ready\": dial tcp 10.133.0.43:15021: connect: connection refused" Oct 30 20:50:30.478506 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:50:30.478466 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 20:50:30.599550 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:50:30.598305 5378 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w0-781058-6d5676f8cb-fkrgx" Oct 30 20:50:30.647151 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:50:30.647109 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 20:50:31.299355 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:50:31.299321 5378 patch_prober.go:29] interesting pod/e2e-worker-w3-682999-openshift-default-54dcb797b5-r8dcv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.43:15021/healthz/ready\": dial tcp 10.133.0.43:15021: connect: connection refused" start-of-body= Oct 30 20:50:31.299529 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:50:31.299386 5378 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-682999-openshift-default-54dcb797b5-r8dcv" podUID="9b76fbb2-f5a8-4fea-bdfd-30f40bee7008" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.43:15021/healthz/ready\": dial tcp 10.133.0.43:15021: connect: connection refused" Oct 30 20:50:32.300138 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:50:32.300101 5378 patch_prober.go:29] interesting pod/e2e-worker-w3-682999-openshift-default-54dcb797b5-r8dcv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.43:15021/healthz/ready\": dial tcp 10.133.0.43:15021: connect: connection refused" start-of-body= Oct 30 20:50:32.300544 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:50:32.300158 5378 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-682999-openshift-default-54dcb797b5-r8dcv" podUID="9b76fbb2-f5a8-4fea-bdfd-30f40bee7008" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.43:15021/healthz/ready\": dial tcp 10.133.0.43:15021: connect: connection refused" Oct 30 20:50:33.304024 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:50:33.303992 5378 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-682999-openshift-default-54dcb797b5-r8dcv" Oct 30 20:50:33.305121 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:50:33.305099 5378 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-682999-openshift-default-54dcb797b5-r8dcv" Oct 30 20:50:33.617210 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:50:33.617180 5378 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w6-8fa2a7-7d66df94fb-clj44" Oct 30 20:50:35.006192 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:50:35.006155 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 20:50:38.593516 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:50:38.593435 5378 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-r9g59"] Oct 30 20:50:39.797977 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:50:39.797898 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 20:50:39.935920 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:50:39.935886 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 20:50:40.084627 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:50:40.084541 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 20:50:40.215895 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:50:40.215865 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 20:50:41.827988 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:50:41.827952 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 20:50:42.601575 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:50:42.601526 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 20:50:42.757993 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:50:42.757952 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 20:50:45.235359 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:50:45.235331 5378 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-1fdaa7-547f998845-pckbc"] Oct 30 20:50:45.240301 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:50:45.240271 5378 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-1fdaa7-547f998845-pckbc" Oct 30 20:50:45.297813 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:50:45.297767 5378 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-w4-1fdaa7-serving-cert\"" Oct 30 20:50:45.348482 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:50:45.348447 5378 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-6mqfn\"" Oct 30 20:50:45.351275 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:50:45.351242 5378 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\"" Oct 30 20:50:45.351275 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:50:45.351251 5378 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 20:50:45.351444 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:50:45.351284 5378 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 20:50:45.353039 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:50:45.353020 5378 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-1fdaa7-547f998845-pckbc"] Oct 30 20:50:45.378025 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:50:45.377996 5378 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c0b8557d-4625-4dc6-a769-e03f2d95c684-openshift-service-ca\") pod \"maas-api-e2e-worker-w4-1fdaa7-547f998845-pckbc\" (UID: \"c0b8557d-4625-4dc6-a769-e03f2d95c684\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-1fdaa7-547f998845-pckbc" Oct 30 20:50:45.378161 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:50:45.378046 5378 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pxlzl\" (UniqueName: \"kubernetes.io/projected/c0b8557d-4625-4dc6-a769-e03f2d95c684-kube-api-access-pxlzl\") pod \"maas-api-e2e-worker-w4-1fdaa7-547f998845-pckbc\" (UID: \"c0b8557d-4625-4dc6-a769-e03f2d95c684\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-1fdaa7-547f998845-pckbc" Oct 30 20:50:45.378161 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:50:45.378126 5378 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c0b8557d-4625-4dc6-a769-e03f2d95c684-maas-api-tls\") pod \"maas-api-e2e-worker-w4-1fdaa7-547f998845-pckbc\" (UID: \"c0b8557d-4625-4dc6-a769-e03f2d95c684\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-1fdaa7-547f998845-pckbc" Oct 30 20:50:45.378271 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:50:45.378162 5378 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c0b8557d-4625-4dc6-a769-e03f2d95c684-metrics-certs\") pod \"maas-api-e2e-worker-w4-1fdaa7-547f998845-pckbc\" (UID: \"c0b8557d-4625-4dc6-a769-e03f2d95c684\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-1fdaa7-547f998845-pckbc" Oct 30 20:50:45.456769 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:50:45.456738 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 20:50:45.478830 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:50:45.478802 5378 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c0b8557d-4625-4dc6-a769-e03f2d95c684-openshift-service-ca\") pod \"maas-api-e2e-worker-w4-1fdaa7-547f998845-pckbc\" (UID: \"c0b8557d-4625-4dc6-a769-e03f2d95c684\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-1fdaa7-547f998845-pckbc" Oct 30 20:50:45.478962 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:50:45.478838 5378 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pxlzl\" (UniqueName: \"kubernetes.io/projected/c0b8557d-4625-4dc6-a769-e03f2d95c684-kube-api-access-pxlzl\") pod \"maas-api-e2e-worker-w4-1fdaa7-547f998845-pckbc\" (UID: \"c0b8557d-4625-4dc6-a769-e03f2d95c684\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-1fdaa7-547f998845-pckbc" Oct 30 20:50:45.478962 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:50:45.478860 5378 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c0b8557d-4625-4dc6-a769-e03f2d95c684-maas-api-tls\") pod \"maas-api-e2e-worker-w4-1fdaa7-547f998845-pckbc\" (UID: \"c0b8557d-4625-4dc6-a769-e03f2d95c684\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-1fdaa7-547f998845-pckbc" Oct 30 20:50:45.479100 ip-10-0-138-149 kubenswrapper[5378]: E1030 20:50:45.479074 5378 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-worker-w4-1fdaa7-serving-cert: secret "maas-api-e2e-worker-w4-1fdaa7-serving-cert" not found Oct 30 20:50:45.479239 ip-10-0-138-149 kubenswrapper[5378]: E1030 20:50:45.479147 5378 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c0b8557d-4625-4dc6-a769-e03f2d95c684-maas-api-tls podName:c0b8557d-4625-4dc6-a769-e03f2d95c684 nodeName:}" failed. No retries permitted until 2026-10-30 20:50:45.9791268 +0000 UTC m=+736.407240708 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/c0b8557d-4625-4dc6-a769-e03f2d95c684-maas-api-tls") pod "maas-api-e2e-worker-w4-1fdaa7-547f998845-pckbc" (UID: "c0b8557d-4625-4dc6-a769-e03f2d95c684") : secret "maas-api-e2e-worker-w4-1fdaa7-serving-cert" not found Oct 30 20:50:45.479239 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:50:45.479180 5378 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c0b8557d-4625-4dc6-a769-e03f2d95c684-metrics-certs\") pod \"maas-api-e2e-worker-w4-1fdaa7-547f998845-pckbc\" (UID: \"c0b8557d-4625-4dc6-a769-e03f2d95c684\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-1fdaa7-547f998845-pckbc" Oct 30 20:50:45.479698 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:50:45.479680 5378 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c0b8557d-4625-4dc6-a769-e03f2d95c684-openshift-service-ca\") pod \"maas-api-e2e-worker-w4-1fdaa7-547f998845-pckbc\" (UID: \"c0b8557d-4625-4dc6-a769-e03f2d95c684\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-1fdaa7-547f998845-pckbc" Oct 30 20:50:45.498261 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:50:45.497091 5378 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c0b8557d-4625-4dc6-a769-e03f2d95c684-metrics-certs\") pod \"maas-api-e2e-worker-w4-1fdaa7-547f998845-pckbc\" (UID: \"c0b8557d-4625-4dc6-a769-e03f2d95c684\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-1fdaa7-547f998845-pckbc" Oct 30 20:50:45.546301 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:50:45.546268 5378 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pxlzl\" (UniqueName: \"kubernetes.io/projected/c0b8557d-4625-4dc6-a769-e03f2d95c684-kube-api-access-pxlzl\") pod \"maas-api-e2e-worker-w4-1fdaa7-547f998845-pckbc\" (UID: \"c0b8557d-4625-4dc6-a769-e03f2d95c684\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-1fdaa7-547f998845-pckbc" Oct 30 20:50:45.663074 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:50:45.660185 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 20:50:45.795477 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:50:45.795392 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 20:50:45.985331 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:50:45.985301 5378 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c0b8557d-4625-4dc6-a769-e03f2d95c684-maas-api-tls\") pod \"maas-api-e2e-worker-w4-1fdaa7-547f998845-pckbc\" (UID: \"c0b8557d-4625-4dc6-a769-e03f2d95c684\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-1fdaa7-547f998845-pckbc" Oct 30 20:50:45.987689 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:50:45.987662 5378 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c0b8557d-4625-4dc6-a769-e03f2d95c684-maas-api-tls\") pod \"maas-api-e2e-worker-w4-1fdaa7-547f998845-pckbc\" (UID: \"c0b8557d-4625-4dc6-a769-e03f2d95c684\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-1fdaa7-547f998845-pckbc" Oct 30 20:50:46.153248 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:50:46.153220 5378 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-1fdaa7-547f998845-pckbc" Oct 30 20:50:46.535505 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:50:46.535444 5378 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-1fdaa7-547f998845-pckbc"] Oct 30 20:50:46.538605 ip-10-0-138-149 kubenswrapper[5378]: W1030 20:50:46.538576 5378 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc0b8557d_4625_4dc6_a769_e03f2d95c684.slice/crio-bc45038a976bd0d96c825fc779d6ea6b14ed3ecb52b62ee041f9abdfc6b2b2da WatchSource:0}: Error finding container bc45038a976bd0d96c825fc779d6ea6b14ed3ecb52b62ee041f9abdfc6b2b2da: Status 404 returned error can't find the container with id bc45038a976bd0d96c825fc779d6ea6b14ed3ecb52b62ee041f9abdfc6b2b2da Oct 30 20:50:46.744027 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:50:46.743995 5378 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-1fdaa7-547f998845-pckbc" event={"ID":"c0b8557d-4625-4dc6-a769-e03f2d95c684","Type":"ContainerStarted","Data":"bc45038a976bd0d96c825fc779d6ea6b14ed3ecb52b62ee041f9abdfc6b2b2da"} Oct 30 20:50:47.122476 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:50:47.122445 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 20:50:48.112516 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:50:48.112480 5378 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-r9g59"] Oct 30 20:50:48.755942 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:50:48.755904 5378 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-1fdaa7-547f998845-pckbc" event={"ID":"c0b8557d-4625-4dc6-a769-e03f2d95c684","Type":"ContainerStarted","Data":"32f99068fbd0e980e08f61e39cf04ab37074e01a9924799fce5b86401365823e"} Oct 30 20:50:48.756127 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:50:48.756020 5378 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-1fdaa7-547f998845-pckbc" Oct 30 20:50:48.835706 ip-10-0-138-149 kubenswrapper[5378]: E1030 20:50:48.835628 5378 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q6mx7" containerName="extractor" Oct 30 20:50:48.876310 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:50:48.876258 5378 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-1fdaa7-547f998845-pckbc" podStartSLOduration=1.8633262959999999 podStartE2EDuration="3.876245093s" totalImagesPullingTime="2.012918797s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:50:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:50:48.875687703 +0000 UTC m=+739.303801626" watchObservedRunningTime="2026-10-30 20:50:48.876245093 +0000 UTC m=+739.304359012" Oct 30 20:50:50.416509 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:50:50.416476 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 20:50:50.589498 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:50:50.589464 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 20:50:50.731950 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:50:50.731871 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 20:50:54.765438 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:50:54.765407 5378 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-1fdaa7-547f998845-pckbc" Oct 30 20:50:55.678754 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:50:55.678717 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 20:50:59.542839 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:50:59.542800 5378 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-r9g59"] Oct 30 20:51:00.789757 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:51:00.789724 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 20:51:02.578392 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:51:02.578348 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 20:51:03.178223 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:51:03.178169 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 20:51:03.317829 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:51:03.317795 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 20:51:03.466113 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:51:03.466023 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 20:51:06.188343 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:51:06.188311 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 20:51:08.863221 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:51:08.863190 5378 generic.go:356] "Generic (PLEG): container finished" podID="2391a719-0c9b-4c56-b229-8f286a9aac96" containerID="4c4e053ee467b0b8deb3acae00ce4858327a6294f588d162c000012246a5364b" exitCode=0 Oct 30 20:51:08.863557 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:51:08.863240 5378 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-1fdaa7-openshift-default-84586db7d5-sbpk8" event={"ID":"2391a719-0c9b-4c56-b229-8f286a9aac96","Type":"ContainerDied","Data":"4c4e053ee467b0b8deb3acae00ce4858327a6294f588d162c000012246a5364b"} Oct 30 20:51:08.863632 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:51:08.863616 5378 scope.go:122] "RemoveContainer" containerID="4c4e053ee467b0b8deb3acae00ce4858327a6294f588d162c000012246a5364b" Oct 30 20:51:08.864881 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:51:08.864851 5378 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 20:51:08.864986 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:51:08.864938 5378 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 20:51:08.865049 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:51:08.864981 5378 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 20:51:09.883262 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:51:09.883226 5378 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-1fdaa7-openshift-default-84586db7d5-sbpk8" event={"ID":"2391a719-0c9b-4c56-b229-8f286a9aac96","Type":"ContainerStarted","Data":"92f7f85bab6bc2319d456ad69b2778e842c9505a32745d738d82a63bfbb854b5"} Oct 30 20:51:10.043033 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:51:10.042994 5378 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-1fdaa7-openshift-default-84586db7d5-sbpk8" Oct 30 20:51:10.043207 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:51:10.043141 5378 patch_prober.go:29] interesting pod/e2e-worker-w4-1fdaa7-openshift-default-84586db7d5-sbpk8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.44:15021/healthz/ready\": dial tcp 10.133.0.44:15021: connect: connection refused" start-of-body= Oct 30 20:51:10.043260 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:51:10.043204 5378 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-1fdaa7-openshift-default-84586db7d5-sbpk8" podUID="2391a719-0c9b-4c56-b229-8f286a9aac96" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.44:15021/healthz/ready\": dial tcp 10.133.0.44:15021: connect: connection refused" Oct 30 20:51:11.043379 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:51:11.043338 5378 patch_prober.go:29] interesting pod/e2e-worker-w4-1fdaa7-openshift-default-84586db7d5-sbpk8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.44:15021/healthz/ready\": dial tcp 10.133.0.44:15021: connect: connection refused" start-of-body= Oct 30 20:51:11.043852 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:51:11.043402 5378 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-1fdaa7-openshift-default-84586db7d5-sbpk8" podUID="2391a719-0c9b-4c56-b229-8f286a9aac96" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.44:15021/healthz/ready\": dial tcp 10.133.0.44:15021: connect: connection refused" Oct 30 20:51:11.103587 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:51:11.103533 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 20:51:11.784584 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:51:11.784432 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 20:51:12.043255 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:51:12.043177 5378 patch_prober.go:29] interesting pod/e2e-worker-w4-1fdaa7-openshift-default-84586db7d5-sbpk8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.44:15021/healthz/ready\": dial tcp 10.133.0.44:15021: connect: connection refused" start-of-body= Oct 30 20:51:12.043255 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:51:12.043237 5378 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-1fdaa7-openshift-default-84586db7d5-sbpk8" podUID="2391a719-0c9b-4c56-b229-8f286a9aac96" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.44:15021/healthz/ready\": dial tcp 10.133.0.44:15021: connect: connection refused" Oct 30 20:51:12.389268 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:51:12.389219 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 20:51:12.535138 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:51:12.535106 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 20:51:12.683489 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:51:12.683396 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 20:51:12.920712 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:51:12.920675 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 20:51:13.044059 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:51:13.043976 5378 patch_prober.go:29] interesting pod/e2e-worker-w4-1fdaa7-openshift-default-84586db7d5-sbpk8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.44:15021/healthz/ready\": dial tcp 10.133.0.44:15021: connect: connection refused" start-of-body= Oct 30 20:51:13.044059 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:51:13.044038 5378 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-1fdaa7-openshift-default-84586db7d5-sbpk8" podUID="2391a719-0c9b-4c56-b229-8f286a9aac96" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.44:15021/healthz/ready\": dial tcp 10.133.0.44:15021: connect: connection refused" Oct 30 20:51:14.043443 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:51:14.043409 5378 patch_prober.go:29] interesting pod/e2e-worker-w4-1fdaa7-openshift-default-84586db7d5-sbpk8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.44:15021/healthz/ready\": dial tcp 10.133.0.44:15021: connect: connection refused" start-of-body= Oct 30 20:51:14.043929 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:51:14.043497 5378 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-1fdaa7-openshift-default-84586db7d5-sbpk8" podUID="2391a719-0c9b-4c56-b229-8f286a9aac96" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.44:15021/healthz/ready\": dial tcp 10.133.0.44:15021: connect: connection refused" Oct 30 20:51:15.076583 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:51:15.076540 5378 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-1fdaa7-openshift-default-84586db7d5-sbpk8" Oct 30 20:51:15.079624 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:51:15.079606 5378 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-1fdaa7-openshift-default-84586db7d5-sbpk8" Oct 30 20:51:15.084385 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:51:15.084358 5378 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-1fdaa7-openshift-default-84586db7d5-sbpk8" Oct 30 20:51:15.675475 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:51:15.675440 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 20:51:16.834939 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:51:16.834905 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 20:51:18.175959 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:51:18.175921 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 20:51:18.349450 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:51:18.349407 5378 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-r9g59"] Oct 30 20:51:18.817636 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:51:18.817557 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 20:51:18.842015 ip-10-0-138-149 kubenswrapper[5378]: E1030 20:51:18.841977 5378 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q6mx7" containerName="extractor" Oct 30 20:51:18.983348 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:51:18.983304 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 20:51:19.155472 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:51:19.155422 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 20:51:19.982973 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:51:19.982941 5378 generic.go:356] "Generic (PLEG): container finished" podID="261c5b5a-84f8-4527-9b51-567d6dbf17ca" containerID="1ae019c888ef9d4c5b0a8b8d7e7b434bbad1b7c253644c927a9f6aa6276aa201" exitCode=0 Oct 30 20:51:19.983396 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:51:19.982992 5378 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-f9f277-openshift-default-869d58fb77-p2wtm" event={"ID":"261c5b5a-84f8-4527-9b51-567d6dbf17ca","Type":"ContainerDied","Data":"1ae019c888ef9d4c5b0a8b8d7e7b434bbad1b7c253644c927a9f6aa6276aa201"} Oct 30 20:51:19.983396 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:51:19.983339 5378 scope.go:122] "RemoveContainer" containerID="1ae019c888ef9d4c5b0a8b8d7e7b434bbad1b7c253644c927a9f6aa6276aa201" Oct 30 20:51:19.984543 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:51:19.984514 5378 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 20:51:19.984617 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:51:19.984603 5378 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 20:51:19.984658 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:51:19.984641 5378 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 20:51:20.991204 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:51:20.991170 5378 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-f9f277-openshift-default-869d58fb77-p2wtm" event={"ID":"261c5b5a-84f8-4527-9b51-567d6dbf17ca","Type":"ContainerStarted","Data":"470d851689964db6c4bc6e0521e64737e2b9b70fd5544f90a068a21fbeb8d14f"} Oct 30 20:51:21.605373 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:51:21.605337 5378 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-f9f277-openshift-default-869d58fb77-p2wtm" Oct 30 20:51:21.605578 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:51:21.605544 5378 patch_prober.go:29] interesting pod/e2e-worker-w1-f9f277-openshift-default-869d58fb77-p2wtm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.42:15021/healthz/ready\": dial tcp 10.133.0.42:15021: connect: connection refused" start-of-body= Oct 30 20:51:21.605652 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:51:21.605634 5378 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-f9f277-openshift-default-869d58fb77-p2wtm" podUID="261c5b5a-84f8-4527-9b51-567d6dbf17ca" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.42:15021/healthz/ready\": dial tcp 10.133.0.42:15021: connect: connection refused" Oct 30 20:51:22.336736 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:51:22.336698 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 20:51:22.605587 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:51:22.605533 5378 patch_prober.go:29] interesting pod/e2e-worker-w1-f9f277-openshift-default-869d58fb77-p2wtm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.42:15021/healthz/ready\": dial tcp 10.133.0.42:15021: connect: connection refused" start-of-body= Oct 30 20:51:22.605735 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:51:22.605643 5378 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-f9f277-openshift-default-869d58fb77-p2wtm" podUID="261c5b5a-84f8-4527-9b51-567d6dbf17ca" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.42:15021/healthz/ready\": dial tcp 10.133.0.42:15021: connect: connection refused" Oct 30 20:51:23.605657 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:51:23.605617 5378 patch_prober.go:29] interesting pod/e2e-worker-w1-f9f277-openshift-default-869d58fb77-p2wtm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.42:15021/healthz/ready\": dial tcp 10.133.0.42:15021: connect: connection refused" start-of-body= Oct 30 20:51:23.606161 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:51:23.605682 5378 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-f9f277-openshift-default-869d58fb77-p2wtm" podUID="261c5b5a-84f8-4527-9b51-567d6dbf17ca" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.42:15021/healthz/ready\": dial tcp 10.133.0.42:15021: connect: connection refused" Oct 30 20:51:24.605905 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:51:24.605870 5378 patch_prober.go:29] interesting pod/e2e-worker-w1-f9f277-openshift-default-869d58fb77-p2wtm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.42:15021/healthz/ready\": dial tcp 10.133.0.42:15021: connect: connection refused" start-of-body= Oct 30 20:51:24.606304 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:51:24.605934 5378 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-f9f277-openshift-default-869d58fb77-p2wtm" podUID="261c5b5a-84f8-4527-9b51-567d6dbf17ca" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.42:15021/healthz/ready\": dial tcp 10.133.0.42:15021: connect: connection refused" Oct 30 20:51:25.605498 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:51:25.605462 5378 patch_prober.go:29] interesting pod/e2e-worker-w1-f9f277-openshift-default-869d58fb77-p2wtm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.42:15021/healthz/ready\": dial tcp 10.133.0.42:15021: connect: connection refused" start-of-body= Oct 30 20:51:25.605702 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:51:25.605518 5378 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-f9f277-openshift-default-869d58fb77-p2wtm" podUID="261c5b5a-84f8-4527-9b51-567d6dbf17ca" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.42:15021/healthz/ready\": dial tcp 10.133.0.42:15021: connect: connection refused" Oct 30 20:51:26.610198 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:51:26.610167 5378 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-f9f277-openshift-default-869d58fb77-p2wtm" Oct 30 20:51:26.610721 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:51:26.610546 5378 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-f9f277-openshift-default-869d58fb77-p2wtm" Oct 30 20:51:26.611265 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:51:26.611249 5378 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-f9f277-openshift-default-869d58fb77-p2wtm" Oct 30 20:51:27.367393 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:51:27.367355 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 20:51:29.612232 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:51:29.612201 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 20:51:30.222038 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:51:30.221972 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 20:51:30.356437 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:51:30.356403 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 20:51:30.510557 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:51:30.510476 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 20:51:32.564392 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:51:32.564330 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 20:51:34.212608 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:51:34.212576 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 20:51:34.807985 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:51:34.807949 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 20:51:34.951202 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:51:34.951169 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 20:51:35.106886 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:51:35.106858 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 20:51:36.541992 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:51:36.541958 5378 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w1-f9f277/simulator-w1-f9f277-kserve-5d9685f85c-2h78x"] Oct 30 20:51:36.546003 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:51:36.545980 5378 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-f9f277/simulator-w1-f9f277-kserve-5d9685f85c-2h78x" Oct 30 20:51:36.602308 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:51:36.602278 5378 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w1-f9f277\"/\"kube-root-ca.crt\"" Oct 30 20:51:36.618283 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:51:36.618236 5378 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/dc662e9d-914c-4dfb-a86f-0fb048f7b690-home\") pod \"simulator-w1-f9f277-kserve-5d9685f85c-2h78x\" (UID: \"dc662e9d-914c-4dfb-a86f-0fb048f7b690\") " pod="e2e-models-e2e-worker-w1-f9f277/simulator-w1-f9f277-kserve-5d9685f85c-2h78x" Oct 30 20:51:36.618444 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:51:36.618352 5378 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dc662e9d-914c-4dfb-a86f-0fb048f7b690-tmp-dir\") pod \"simulator-w1-f9f277-kserve-5d9685f85c-2h78x\" (UID: \"dc662e9d-914c-4dfb-a86f-0fb048f7b690\") " pod="e2e-models-e2e-worker-w1-f9f277/simulator-w1-f9f277-kserve-5d9685f85c-2h78x" Oct 30 20:51:36.618444 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:51:36.618406 5378 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/dc662e9d-914c-4dfb-a86f-0fb048f7b690-model-cache\") pod \"simulator-w1-f9f277-kserve-5d9685f85c-2h78x\" (UID: \"dc662e9d-914c-4dfb-a86f-0fb048f7b690\") " pod="e2e-models-e2e-worker-w1-f9f277/simulator-w1-f9f277-kserve-5d9685f85c-2h78x" Oct 30 20:51:36.618444 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:51:36.618422 5378 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dc662e9d-914c-4dfb-a86f-0fb048f7b690-tls-certs\") pod \"simulator-w1-f9f277-kserve-5d9685f85c-2h78x\" (UID: \"dc662e9d-914c-4dfb-a86f-0fb048f7b690\") " pod="e2e-models-e2e-worker-w1-f9f277/simulator-w1-f9f277-kserve-5d9685f85c-2h78x" Oct 30 20:51:36.618544 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:51:36.618496 5378 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-msjgx\" (UniqueName: \"kubernetes.io/projected/dc662e9d-914c-4dfb-a86f-0fb048f7b690-kube-api-access-msjgx\") pod \"simulator-w1-f9f277-kserve-5d9685f85c-2h78x\" (UID: \"dc662e9d-914c-4dfb-a86f-0fb048f7b690\") " pod="e2e-models-e2e-worker-w1-f9f277/simulator-w1-f9f277-kserve-5d9685f85c-2h78x" Oct 30 20:51:36.618689 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:51:36.618670 5378 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/dc662e9d-914c-4dfb-a86f-0fb048f7b690-dshm\") pod \"simulator-w1-f9f277-kserve-5d9685f85c-2h78x\" (UID: \"dc662e9d-914c-4dfb-a86f-0fb048f7b690\") " pod="e2e-models-e2e-worker-w1-f9f277/simulator-w1-f9f277-kserve-5d9685f85c-2h78x" Oct 30 20:51:36.647686 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:51:36.647655 5378 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w1-f9f277\"/\"openshift-service-ca.crt\"" Oct 30 20:51:36.652807 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:51:36.652783 5378 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w1-f9f277\"/\"default-dockercfg-xbx2p\"" Oct 30 20:51:36.652933 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:51:36.652818 5378 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w1-f9f277\"/\"simulator-w1-f9f277-kserve-self-signed-certs\"" Oct 30 20:51:36.654795 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:51:36.654776 5378 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-f9f277/simulator-w1-f9f277-kserve-5d9685f85c-2h78x"] Oct 30 20:51:36.717323 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:51:36.717287 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 20:51:36.719221 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:51:36.719197 5378 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/dc662e9d-914c-4dfb-a86f-0fb048f7b690-model-cache\") pod \"simulator-w1-f9f277-kserve-5d9685f85c-2h78x\" (UID: \"dc662e9d-914c-4dfb-a86f-0fb048f7b690\") " pod="e2e-models-e2e-worker-w1-f9f277/simulator-w1-f9f277-kserve-5d9685f85c-2h78x" Oct 30 20:51:36.719346 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:51:36.719226 5378 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dc662e9d-914c-4dfb-a86f-0fb048f7b690-tls-certs\") pod \"simulator-w1-f9f277-kserve-5d9685f85c-2h78x\" (UID: \"dc662e9d-914c-4dfb-a86f-0fb048f7b690\") " pod="e2e-models-e2e-worker-w1-f9f277/simulator-w1-f9f277-kserve-5d9685f85c-2h78x" Oct 30 20:51:36.719346 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:51:36.719265 5378 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-msjgx\" (UniqueName: \"kubernetes.io/projected/dc662e9d-914c-4dfb-a86f-0fb048f7b690-kube-api-access-msjgx\") pod \"simulator-w1-f9f277-kserve-5d9685f85c-2h78x\" (UID: \"dc662e9d-914c-4dfb-a86f-0fb048f7b690\") " pod="e2e-models-e2e-worker-w1-f9f277/simulator-w1-f9f277-kserve-5d9685f85c-2h78x" Oct 30 20:51:36.719346 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:51:36.719322 5378 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/dc662e9d-914c-4dfb-a86f-0fb048f7b690-dshm\") pod \"simulator-w1-f9f277-kserve-5d9685f85c-2h78x\" (UID: \"dc662e9d-914c-4dfb-a86f-0fb048f7b690\") " pod="e2e-models-e2e-worker-w1-f9f277/simulator-w1-f9f277-kserve-5d9685f85c-2h78x" Oct 30 20:51:36.719522 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:51:36.719425 5378 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/dc662e9d-914c-4dfb-a86f-0fb048f7b690-home\") pod \"simulator-w1-f9f277-kserve-5d9685f85c-2h78x\" (UID: \"dc662e9d-914c-4dfb-a86f-0fb048f7b690\") " pod="e2e-models-e2e-worker-w1-f9f277/simulator-w1-f9f277-kserve-5d9685f85c-2h78x" Oct 30 20:51:36.719522 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:51:36.719488 5378 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dc662e9d-914c-4dfb-a86f-0fb048f7b690-tmp-dir\") pod \"simulator-w1-f9f277-kserve-5d9685f85c-2h78x\" (UID: \"dc662e9d-914c-4dfb-a86f-0fb048f7b690\") " pod="e2e-models-e2e-worker-w1-f9f277/simulator-w1-f9f277-kserve-5d9685f85c-2h78x" Oct 30 20:51:36.719697 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:51:36.719678 5378 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/dc662e9d-914c-4dfb-a86f-0fb048f7b690-model-cache\") pod \"simulator-w1-f9f277-kserve-5d9685f85c-2h78x\" (UID: \"dc662e9d-914c-4dfb-a86f-0fb048f7b690\") " pod="e2e-models-e2e-worker-w1-f9f277/simulator-w1-f9f277-kserve-5d9685f85c-2h78x" Oct 30 20:51:36.719803 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:51:36.719782 5378 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/dc662e9d-914c-4dfb-a86f-0fb048f7b690-home\") pod \"simulator-w1-f9f277-kserve-5d9685f85c-2h78x\" (UID: \"dc662e9d-914c-4dfb-a86f-0fb048f7b690\") " pod="e2e-models-e2e-worker-w1-f9f277/simulator-w1-f9f277-kserve-5d9685f85c-2h78x" Oct 30 20:51:36.719844 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:51:36.719832 5378 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dc662e9d-914c-4dfb-a86f-0fb048f7b690-tmp-dir\") pod \"simulator-w1-f9f277-kserve-5d9685f85c-2h78x\" (UID: \"dc662e9d-914c-4dfb-a86f-0fb048f7b690\") " pod="e2e-models-e2e-worker-w1-f9f277/simulator-w1-f9f277-kserve-5d9685f85c-2h78x" Oct 30 20:51:36.721774 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:51:36.721752 5378 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dc662e9d-914c-4dfb-a86f-0fb048f7b690-tls-certs\") pod \"simulator-w1-f9f277-kserve-5d9685f85c-2h78x\" (UID: \"dc662e9d-914c-4dfb-a86f-0fb048f7b690\") " pod="e2e-models-e2e-worker-w1-f9f277/simulator-w1-f9f277-kserve-5d9685f85c-2h78x" Oct 30 20:51:36.722221 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:51:36.722201 5378 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/dc662e9d-914c-4dfb-a86f-0fb048f7b690-dshm\") pod \"simulator-w1-f9f277-kserve-5d9685f85c-2h78x\" (UID: \"dc662e9d-914c-4dfb-a86f-0fb048f7b690\") " pod="e2e-models-e2e-worker-w1-f9f277/simulator-w1-f9f277-kserve-5d9685f85c-2h78x" Oct 30 20:51:36.776547 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:51:36.776519 5378 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-msjgx\" (UniqueName: \"kubernetes.io/projected/dc662e9d-914c-4dfb-a86f-0fb048f7b690-kube-api-access-msjgx\") pod \"simulator-w1-f9f277-kserve-5d9685f85c-2h78x\" (UID: \"dc662e9d-914c-4dfb-a86f-0fb048f7b690\") " pod="e2e-models-e2e-worker-w1-f9f277/simulator-w1-f9f277-kserve-5d9685f85c-2h78x" Oct 30 20:51:36.859532 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:51:36.859504 5378 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-f9f277/simulator-w1-f9f277-kserve-5d9685f85c-2h78x" Oct 30 20:51:37.250622 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:51:37.250592 5378 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-f9f277/simulator-w1-f9f277-kserve-5d9685f85c-2h78x"] Oct 30 20:51:37.252358 ip-10-0-138-149 kubenswrapper[5378]: W1030 20:51:37.252327 5378 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poddc662e9d_914c_4dfb_a86f_0fb048f7b690.slice/crio-91e64839ddba643b5f406aacc92dcab4f72496ae37403f03c17fda8e0a1f174f WatchSource:0}: Error finding container 91e64839ddba643b5f406aacc92dcab4f72496ae37403f03c17fda8e0a1f174f: Status 404 returned error can't find the container with id 91e64839ddba643b5f406aacc92dcab4f72496ae37403f03c17fda8e0a1f174f Oct 30 20:51:37.864839 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:51:37.864806 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 20:51:38.086356 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:51:38.086317 5378 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-f9f277/simulator-w1-f9f277-kserve-5d9685f85c-2h78x" event={"ID":"dc662e9d-914c-4dfb-a86f-0fb048f7b690","Type":"ContainerStarted","Data":"a6d16baadcb84866120ccf6156a24bd15e3f789892937fa292bcb2462dde73a0"} Oct 30 20:51:38.086356 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:51:38.086355 5378 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-f9f277/simulator-w1-f9f277-kserve-5d9685f85c-2h78x" event={"ID":"dc662e9d-914c-4dfb-a86f-0fb048f7b690","Type":"ContainerStarted","Data":"91e64839ddba643b5f406aacc92dcab4f72496ae37403f03c17fda8e0a1f174f"} Oct 30 20:51:38.202551 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:51:38.202449 5378 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w1-f9f277/simulator-w1-f9f277-kserve-5d9685f85c-2h78x" podStartSLOduration=2.20243377 podStartE2EDuration="2.20243377s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:51:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:51:38.201455475 +0000 UTC m=+788.629569407" watchObservedRunningTime="2026-10-30 20:51:38.20243377 +0000 UTC m=+788.630547688" Oct 30 20:51:38.859908 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:51:38.859872 5378 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-f9f277/simulator-w1-f9f277-kserve-5d9685f85c-2h78x" Oct 30 20:51:39.517839 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:51:39.517810 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 20:51:40.132265 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:51:40.132229 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 20:51:40.270550 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:51:40.270520 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 20:51:40.423067 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:51:40.422990 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 20:51:41.572144 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:51:41.572110 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 20:51:42.387500 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:51:42.387462 5378 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-r9g59"] Oct 30 20:51:42.986906 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:51:42.986876 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 20:51:46.860401 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:51:46.860370 5378 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w1-f9f277/simulator-w1-f9f277-kserve-5d9685f85c-2h78x" Oct 30 20:51:46.872925 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:51:46.872900 5378 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w1-f9f277/simulator-w1-f9f277-kserve-5d9685f85c-2h78x" Oct 30 20:51:47.142600 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:51:47.142509 5378 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-f9f277/simulator-w1-f9f277-kserve-5d9685f85c-2h78x" Oct 30 20:51:47.873880 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:51:47.873845 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 20:51:48.701671 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:51:48.701636 5378 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w6-8fa2a7/simulator-w6-8fa2a7-kserve-6cbf5bd894-lfpcq"] Oct 30 20:51:48.705660 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:51:48.705639 5378 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-8fa2a7/simulator-w6-8fa2a7-kserve-6cbf5bd894-lfpcq" Oct 30 20:51:48.742685 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:51:48.742653 5378 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f409f469-71fc-445b-bdc8-b8e84a4d1dc6-model-cache\") pod \"simulator-w6-8fa2a7-kserve-6cbf5bd894-lfpcq\" (UID: \"f409f469-71fc-445b-bdc8-b8e84a4d1dc6\") " pod="e2e-models-e2e-worker-w6-8fa2a7/simulator-w6-8fa2a7-kserve-6cbf5bd894-lfpcq" Oct 30 20:51:48.742818 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:51:48.742696 5378 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gkjj8\" (UniqueName: \"kubernetes.io/projected/f409f469-71fc-445b-bdc8-b8e84a4d1dc6-kube-api-access-gkjj8\") pod \"simulator-w6-8fa2a7-kserve-6cbf5bd894-lfpcq\" (UID: \"f409f469-71fc-445b-bdc8-b8e84a4d1dc6\") " pod="e2e-models-e2e-worker-w6-8fa2a7/simulator-w6-8fa2a7-kserve-6cbf5bd894-lfpcq" Oct 30 20:51:48.742818 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:51:48.742804 5378 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f409f469-71fc-445b-bdc8-b8e84a4d1dc6-tls-certs\") pod \"simulator-w6-8fa2a7-kserve-6cbf5bd894-lfpcq\" (UID: \"f409f469-71fc-445b-bdc8-b8e84a4d1dc6\") " pod="e2e-models-e2e-worker-w6-8fa2a7/simulator-w6-8fa2a7-kserve-6cbf5bd894-lfpcq" Oct 30 20:51:48.742934 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:51:48.742905 5378 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f409f469-71fc-445b-bdc8-b8e84a4d1dc6-tmp-dir\") pod \"simulator-w6-8fa2a7-kserve-6cbf5bd894-lfpcq\" (UID: \"f409f469-71fc-445b-bdc8-b8e84a4d1dc6\") " pod="e2e-models-e2e-worker-w6-8fa2a7/simulator-w6-8fa2a7-kserve-6cbf5bd894-lfpcq" Oct 30 20:51:48.743037 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:51:48.743014 5378 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f409f469-71fc-445b-bdc8-b8e84a4d1dc6-dshm\") pod \"simulator-w6-8fa2a7-kserve-6cbf5bd894-lfpcq\" (UID: \"f409f469-71fc-445b-bdc8-b8e84a4d1dc6\") " pod="e2e-models-e2e-worker-w6-8fa2a7/simulator-w6-8fa2a7-kserve-6cbf5bd894-lfpcq" Oct 30 20:51:48.743083 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:51:48.743055 5378 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f409f469-71fc-445b-bdc8-b8e84a4d1dc6-home\") pod \"simulator-w6-8fa2a7-kserve-6cbf5bd894-lfpcq\" (UID: \"f409f469-71fc-445b-bdc8-b8e84a4d1dc6\") " pod="e2e-models-e2e-worker-w6-8fa2a7/simulator-w6-8fa2a7-kserve-6cbf5bd894-lfpcq" Oct 30 20:51:48.758779 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:51:48.758742 5378 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w6-8fa2a7\"/\"openshift-service-ca.crt\"" Oct 30 20:51:48.794430 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:51:48.794408 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 20:51:48.809326 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:51:48.809301 5378 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w6-8fa2a7\"/\"default-dockercfg-q9gbm\"" Oct 30 20:51:48.809424 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:51:48.809344 5378 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w6-8fa2a7\"/\"simulator-w6-8fa2a7-kserve-self-signed-certs\"" Oct 30 20:51:48.811310 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:51:48.811286 5378 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w6-8fa2a7/simulator-w6-8fa2a7-kserve-6cbf5bd894-lfpcq"] Oct 30 20:51:48.839553 ip-10-0-138-149 kubenswrapper[5378]: E1030 20:51:48.839523 5378 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q6mx7" containerName="extractor" Oct 30 20:51:48.844580 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:51:48.844550 5378 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f409f469-71fc-445b-bdc8-b8e84a4d1dc6-tls-certs\") pod \"simulator-w6-8fa2a7-kserve-6cbf5bd894-lfpcq\" (UID: \"f409f469-71fc-445b-bdc8-b8e84a4d1dc6\") " pod="e2e-models-e2e-worker-w6-8fa2a7/simulator-w6-8fa2a7-kserve-6cbf5bd894-lfpcq" Oct 30 20:51:48.844662 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:51:48.844615 5378 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f409f469-71fc-445b-bdc8-b8e84a4d1dc6-tmp-dir\") pod \"simulator-w6-8fa2a7-kserve-6cbf5bd894-lfpcq\" (UID: \"f409f469-71fc-445b-bdc8-b8e84a4d1dc6\") " pod="e2e-models-e2e-worker-w6-8fa2a7/simulator-w6-8fa2a7-kserve-6cbf5bd894-lfpcq" Oct 30 20:51:48.844715 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:51:48.844661 5378 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f409f469-71fc-445b-bdc8-b8e84a4d1dc6-dshm\") pod \"simulator-w6-8fa2a7-kserve-6cbf5bd894-lfpcq\" (UID: \"f409f469-71fc-445b-bdc8-b8e84a4d1dc6\") " pod="e2e-models-e2e-worker-w6-8fa2a7/simulator-w6-8fa2a7-kserve-6cbf5bd894-lfpcq" Oct 30 20:51:48.844715 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:51:48.844701 5378 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f409f469-71fc-445b-bdc8-b8e84a4d1dc6-home\") pod \"simulator-w6-8fa2a7-kserve-6cbf5bd894-lfpcq\" (UID: \"f409f469-71fc-445b-bdc8-b8e84a4d1dc6\") " pod="e2e-models-e2e-worker-w6-8fa2a7/simulator-w6-8fa2a7-kserve-6cbf5bd894-lfpcq" Oct 30 20:51:48.844812 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:51:48.844784 5378 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f409f469-71fc-445b-bdc8-b8e84a4d1dc6-model-cache\") pod \"simulator-w6-8fa2a7-kserve-6cbf5bd894-lfpcq\" (UID: \"f409f469-71fc-445b-bdc8-b8e84a4d1dc6\") " pod="e2e-models-e2e-worker-w6-8fa2a7/simulator-w6-8fa2a7-kserve-6cbf5bd894-lfpcq" Oct 30 20:51:48.844865 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:51:48.844810 5378 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gkjj8\" (UniqueName: \"kubernetes.io/projected/f409f469-71fc-445b-bdc8-b8e84a4d1dc6-kube-api-access-gkjj8\") pod \"simulator-w6-8fa2a7-kserve-6cbf5bd894-lfpcq\" (UID: \"f409f469-71fc-445b-bdc8-b8e84a4d1dc6\") " pod="e2e-models-e2e-worker-w6-8fa2a7/simulator-w6-8fa2a7-kserve-6cbf5bd894-lfpcq" Oct 30 20:51:48.845095 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:51:48.845074 5378 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f409f469-71fc-445b-bdc8-b8e84a4d1dc6-tmp-dir\") pod \"simulator-w6-8fa2a7-kserve-6cbf5bd894-lfpcq\" (UID: \"f409f469-71fc-445b-bdc8-b8e84a4d1dc6\") " pod="e2e-models-e2e-worker-w6-8fa2a7/simulator-w6-8fa2a7-kserve-6cbf5bd894-lfpcq" Oct 30 20:51:48.845220 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:51:48.845139 5378 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f409f469-71fc-445b-bdc8-b8e84a4d1dc6-home\") pod \"simulator-w6-8fa2a7-kserve-6cbf5bd894-lfpcq\" (UID: \"f409f469-71fc-445b-bdc8-b8e84a4d1dc6\") " pod="e2e-models-e2e-worker-w6-8fa2a7/simulator-w6-8fa2a7-kserve-6cbf5bd894-lfpcq" Oct 30 20:51:48.845220 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:51:48.845189 5378 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f409f469-71fc-445b-bdc8-b8e84a4d1dc6-model-cache\") pod \"simulator-w6-8fa2a7-kserve-6cbf5bd894-lfpcq\" (UID: \"f409f469-71fc-445b-bdc8-b8e84a4d1dc6\") " pod="e2e-models-e2e-worker-w6-8fa2a7/simulator-w6-8fa2a7-kserve-6cbf5bd894-lfpcq" Oct 30 20:51:48.846684 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:51:48.846668 5378 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f409f469-71fc-445b-bdc8-b8e84a4d1dc6-dshm\") pod \"simulator-w6-8fa2a7-kserve-6cbf5bd894-lfpcq\" (UID: \"f409f469-71fc-445b-bdc8-b8e84a4d1dc6\") " pod="e2e-models-e2e-worker-w6-8fa2a7/simulator-w6-8fa2a7-kserve-6cbf5bd894-lfpcq" Oct 30 20:51:48.847128 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:51:48.847111 5378 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f409f469-71fc-445b-bdc8-b8e84a4d1dc6-tls-certs\") pod \"simulator-w6-8fa2a7-kserve-6cbf5bd894-lfpcq\" (UID: \"f409f469-71fc-445b-bdc8-b8e84a4d1dc6\") " pod="e2e-models-e2e-worker-w6-8fa2a7/simulator-w6-8fa2a7-kserve-6cbf5bd894-lfpcq" Oct 30 20:51:48.872591 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:51:48.872361 5378 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w6-8fa2a7\"/\"kube-root-ca.crt\"" Oct 30 20:51:48.903359 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:51:48.903328 5378 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gkjj8\" (UniqueName: \"kubernetes.io/projected/f409f469-71fc-445b-bdc8-b8e84a4d1dc6-kube-api-access-gkjj8\") pod \"simulator-w6-8fa2a7-kserve-6cbf5bd894-lfpcq\" (UID: \"f409f469-71fc-445b-bdc8-b8e84a4d1dc6\") " pod="e2e-models-e2e-worker-w6-8fa2a7/simulator-w6-8fa2a7-kserve-6cbf5bd894-lfpcq" Oct 30 20:51:49.017874 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:51:49.017788 5378 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-8fa2a7/simulator-w6-8fa2a7-kserve-6cbf5bd894-lfpcq" Oct 30 20:51:49.388488 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:51:49.388459 5378 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w6-8fa2a7/simulator-w6-8fa2a7-kserve-6cbf5bd894-lfpcq"] Oct 30 20:51:49.390320 ip-10-0-138-149 kubenswrapper[5378]: W1030 20:51:49.390294 5378 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podf409f469_71fc_445b_bdc8_b8e84a4d1dc6.slice/crio-4e5802167291b0752fb50965139888bfe6b4f4a5c36d3c5663e8389d085c4cd8 WatchSource:0}: Error finding container 4e5802167291b0752fb50965139888bfe6b4f4a5c36d3c5663e8389d085c4cd8: Status 404 returned error can't find the container with id 4e5802167291b0752fb50965139888bfe6b4f4a5c36d3c5663e8389d085c4cd8 Oct 30 20:51:50.150061 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:51:50.150027 5378 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-8fa2a7/simulator-w6-8fa2a7-kserve-6cbf5bd894-lfpcq" event={"ID":"f409f469-71fc-445b-bdc8-b8e84a4d1dc6","Type":"ContainerStarted","Data":"ec8df931af9b57998441e9214b15235987b3a0fe13581ac2c52e3deeb7c9f85d"} Oct 30 20:51:50.150506 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:51:50.150070 5378 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-8fa2a7/simulator-w6-8fa2a7-kserve-6cbf5bd894-lfpcq" event={"ID":"f409f469-71fc-445b-bdc8-b8e84a4d1dc6","Type":"ContainerStarted","Data":"4e5802167291b0752fb50965139888bfe6b4f4a5c36d3c5663e8389d085c4cd8"} Oct 30 20:51:50.268714 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:51:50.268661 5378 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w6-8fa2a7/simulator-w6-8fa2a7-kserve-6cbf5bd894-lfpcq" podStartSLOduration=2.268647274 podStartE2EDuration="2.268647274s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:51:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:51:50.267419265 +0000 UTC m=+800.695533187" watchObservedRunningTime="2026-10-30 20:51:50.268647274 +0000 UTC m=+800.696761187" Oct 30 20:51:51.018850 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:51:51.018815 5378 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-8fa2a7/simulator-w6-8fa2a7-kserve-6cbf5bd894-lfpcq" Oct 30 20:51:52.687878 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:51:52.687848 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 20:51:54.602443 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:51:54.602407 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 20:51:57.464673 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:51:57.464636 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 20:51:59.018471 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:51:59.018438 5378 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w6-8fa2a7/simulator-w6-8fa2a7-kserve-6cbf5bd894-lfpcq" Oct 30 20:51:59.030672 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:51:59.030644 5378 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w6-8fa2a7/simulator-w6-8fa2a7-kserve-6cbf5bd894-lfpcq" Oct 30 20:51:59.212549 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:51:59.212514 5378 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-8fa2a7/simulator-w6-8fa2a7-kserve-6cbf5bd894-lfpcq" Oct 30 20:52:00.035521 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:52:00.035485 5378 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-r9g59"] Oct 30 20:52:02.366299 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:52:02.366249 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 20:52:07.076091 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:52:07.076059 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 20:52:11.967082 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:52:11.966996 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 20:52:13.258297 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:52:13.258262 5378 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w4-1fdaa7/simulator-w4-1fdaa7-kserve-9866d468c-fwmsk"] Oct 30 20:52:13.261972 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:52:13.261951 5378 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-1fdaa7/simulator-w4-1fdaa7-kserve-9866d468c-fwmsk" Oct 30 20:52:13.340945 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:52:13.340908 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 20:52:13.367863 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:52:13.367834 5378 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w4-1fdaa7\"/\"openshift-service-ca.crt\"" Oct 30 20:52:13.367863 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:52:13.367834 5378 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w4-1fdaa7\"/\"kube-root-ca.crt\"" Oct 30 20:52:13.368954 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:52:13.368937 5378 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-1fdaa7\"/\"default-dockercfg-ffzcn\"" Oct 30 20:52:13.368954 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:52:13.368945 5378 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-1fdaa7\"/\"simulator-w4-1fdaa7-kserve-self-signed-certs\"" Oct 30 20:52:13.369839 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:52:13.369821 5378 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-1fdaa7/simulator-w4-1fdaa7-kserve-9866d468c-fwmsk"] Oct 30 20:52:13.411381 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:52:13.402585 5378 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/09219832-010e-4ba8-bfb9-5a2884b358e1-model-cache\") pod \"simulator-w4-1fdaa7-kserve-9866d468c-fwmsk\" (UID: \"09219832-010e-4ba8-bfb9-5a2884b358e1\") " pod="e2e-models-e2e-worker-w4-1fdaa7/simulator-w4-1fdaa7-kserve-9866d468c-fwmsk" Oct 30 20:52:13.411381 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:52:13.402684 5378 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/09219832-010e-4ba8-bfb9-5a2884b358e1-tmp-dir\") pod \"simulator-w4-1fdaa7-kserve-9866d468c-fwmsk\" (UID: \"09219832-010e-4ba8-bfb9-5a2884b358e1\") " pod="e2e-models-e2e-worker-w4-1fdaa7/simulator-w4-1fdaa7-kserve-9866d468c-fwmsk" Oct 30 20:52:13.411381 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:52:13.402777 5378 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/09219832-010e-4ba8-bfb9-5a2884b358e1-home\") pod \"simulator-w4-1fdaa7-kserve-9866d468c-fwmsk\" (UID: \"09219832-010e-4ba8-bfb9-5a2884b358e1\") " pod="e2e-models-e2e-worker-w4-1fdaa7/simulator-w4-1fdaa7-kserve-9866d468c-fwmsk" Oct 30 20:52:13.411381 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:52:13.402824 5378 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/09219832-010e-4ba8-bfb9-5a2884b358e1-tls-certs\") pod \"simulator-w4-1fdaa7-kserve-9866d468c-fwmsk\" (UID: \"09219832-010e-4ba8-bfb9-5a2884b358e1\") " pod="e2e-models-e2e-worker-w4-1fdaa7/simulator-w4-1fdaa7-kserve-9866d468c-fwmsk" Oct 30 20:52:13.411381 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:52:13.402854 5378 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bxm9w\" (UniqueName: \"kubernetes.io/projected/09219832-010e-4ba8-bfb9-5a2884b358e1-kube-api-access-bxm9w\") pod \"simulator-w4-1fdaa7-kserve-9866d468c-fwmsk\" (UID: \"09219832-010e-4ba8-bfb9-5a2884b358e1\") " pod="e2e-models-e2e-worker-w4-1fdaa7/simulator-w4-1fdaa7-kserve-9866d468c-fwmsk" Oct 30 20:52:13.411381 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:52:13.402888 5378 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/09219832-010e-4ba8-bfb9-5a2884b358e1-dshm\") pod \"simulator-w4-1fdaa7-kserve-9866d468c-fwmsk\" (UID: \"09219832-010e-4ba8-bfb9-5a2884b358e1\") " pod="e2e-models-e2e-worker-w4-1fdaa7/simulator-w4-1fdaa7-kserve-9866d468c-fwmsk" Oct 30 20:52:13.503416 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:52:13.503378 5378 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/09219832-010e-4ba8-bfb9-5a2884b358e1-tmp-dir\") pod \"simulator-w4-1fdaa7-kserve-9866d468c-fwmsk\" (UID: \"09219832-010e-4ba8-bfb9-5a2884b358e1\") " pod="e2e-models-e2e-worker-w4-1fdaa7/simulator-w4-1fdaa7-kserve-9866d468c-fwmsk" Oct 30 20:52:13.503610 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:52:13.503445 5378 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/09219832-010e-4ba8-bfb9-5a2884b358e1-home\") pod \"simulator-w4-1fdaa7-kserve-9866d468c-fwmsk\" (UID: \"09219832-010e-4ba8-bfb9-5a2884b358e1\") " pod="e2e-models-e2e-worker-w4-1fdaa7/simulator-w4-1fdaa7-kserve-9866d468c-fwmsk" Oct 30 20:52:13.503610 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:52:13.503477 5378 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/09219832-010e-4ba8-bfb9-5a2884b358e1-tls-certs\") pod \"simulator-w4-1fdaa7-kserve-9866d468c-fwmsk\" (UID: \"09219832-010e-4ba8-bfb9-5a2884b358e1\") " pod="e2e-models-e2e-worker-w4-1fdaa7/simulator-w4-1fdaa7-kserve-9866d468c-fwmsk" Oct 30 20:52:13.503610 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:52:13.503496 5378 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bxm9w\" (UniqueName: \"kubernetes.io/projected/09219832-010e-4ba8-bfb9-5a2884b358e1-kube-api-access-bxm9w\") pod \"simulator-w4-1fdaa7-kserve-9866d468c-fwmsk\" (UID: \"09219832-010e-4ba8-bfb9-5a2884b358e1\") " pod="e2e-models-e2e-worker-w4-1fdaa7/simulator-w4-1fdaa7-kserve-9866d468c-fwmsk" Oct 30 20:52:13.503610 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:52:13.503517 5378 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/09219832-010e-4ba8-bfb9-5a2884b358e1-dshm\") pod \"simulator-w4-1fdaa7-kserve-9866d468c-fwmsk\" (UID: \"09219832-010e-4ba8-bfb9-5a2884b358e1\") " pod="e2e-models-e2e-worker-w4-1fdaa7/simulator-w4-1fdaa7-kserve-9866d468c-fwmsk" Oct 30 20:52:13.503832 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:52:13.503743 5378 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/09219832-010e-4ba8-bfb9-5a2884b358e1-model-cache\") pod \"simulator-w4-1fdaa7-kserve-9866d468c-fwmsk\" (UID: \"09219832-010e-4ba8-bfb9-5a2884b358e1\") " pod="e2e-models-e2e-worker-w4-1fdaa7/simulator-w4-1fdaa7-kserve-9866d468c-fwmsk" Oct 30 20:52:13.503888 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:52:13.503859 5378 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/09219832-010e-4ba8-bfb9-5a2884b358e1-tmp-dir\") pod \"simulator-w4-1fdaa7-kserve-9866d468c-fwmsk\" (UID: \"09219832-010e-4ba8-bfb9-5a2884b358e1\") " pod="e2e-models-e2e-worker-w4-1fdaa7/simulator-w4-1fdaa7-kserve-9866d468c-fwmsk" Oct 30 20:52:13.503888 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:52:13.503875 5378 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/09219832-010e-4ba8-bfb9-5a2884b358e1-home\") pod \"simulator-w4-1fdaa7-kserve-9866d468c-fwmsk\" (UID: \"09219832-010e-4ba8-bfb9-5a2884b358e1\") " pod="e2e-models-e2e-worker-w4-1fdaa7/simulator-w4-1fdaa7-kserve-9866d468c-fwmsk" Oct 30 20:52:13.504031 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:52:13.504013 5378 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/09219832-010e-4ba8-bfb9-5a2884b358e1-model-cache\") pod \"simulator-w4-1fdaa7-kserve-9866d468c-fwmsk\" (UID: \"09219832-010e-4ba8-bfb9-5a2884b358e1\") " pod="e2e-models-e2e-worker-w4-1fdaa7/simulator-w4-1fdaa7-kserve-9866d468c-fwmsk" Oct 30 20:52:13.505820 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:52:13.505796 5378 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/09219832-010e-4ba8-bfb9-5a2884b358e1-dshm\") pod \"simulator-w4-1fdaa7-kserve-9866d468c-fwmsk\" (UID: \"09219832-010e-4ba8-bfb9-5a2884b358e1\") " pod="e2e-models-e2e-worker-w4-1fdaa7/simulator-w4-1fdaa7-kserve-9866d468c-fwmsk" Oct 30 20:52:13.505986 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:52:13.505969 5378 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/09219832-010e-4ba8-bfb9-5a2884b358e1-tls-certs\") pod \"simulator-w4-1fdaa7-kserve-9866d468c-fwmsk\" (UID: \"09219832-010e-4ba8-bfb9-5a2884b358e1\") " pod="e2e-models-e2e-worker-w4-1fdaa7/simulator-w4-1fdaa7-kserve-9866d468c-fwmsk" Oct 30 20:52:13.561621 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:52:13.561536 5378 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bxm9w\" (UniqueName: \"kubernetes.io/projected/09219832-010e-4ba8-bfb9-5a2884b358e1-kube-api-access-bxm9w\") pod \"simulator-w4-1fdaa7-kserve-9866d468c-fwmsk\" (UID: \"09219832-010e-4ba8-bfb9-5a2884b358e1\") " pod="e2e-models-e2e-worker-w4-1fdaa7/simulator-w4-1fdaa7-kserve-9866d468c-fwmsk" Oct 30 20:52:13.573926 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:52:13.573900 5378 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-1fdaa7/simulator-w4-1fdaa7-kserve-9866d468c-fwmsk" Oct 30 20:52:13.944005 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:52:13.943978 5378 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-1fdaa7/simulator-w4-1fdaa7-kserve-9866d468c-fwmsk"] Oct 30 20:52:13.945915 ip-10-0-138-149 kubenswrapper[5378]: W1030 20:52:13.945882 5378 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod09219832_010e_4ba8_bfb9_5a2884b358e1.slice/crio-d1292e3c66c3d98af24e6c68f24d992a1a06c1982f6b391d6a7ac03d12b445f8 WatchSource:0}: Error finding container d1292e3c66c3d98af24e6c68f24d992a1a06c1982f6b391d6a7ac03d12b445f8: Status 404 returned error can't find the container with id d1292e3c66c3d98af24e6c68f24d992a1a06c1982f6b391d6a7ac03d12b445f8 Oct 30 20:52:14.293870 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:52:14.293777 5378 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-1fdaa7/simulator-w4-1fdaa7-kserve-9866d468c-fwmsk" event={"ID":"09219832-010e-4ba8-bfb9-5a2884b358e1","Type":"ContainerStarted","Data":"6be1583f151b9b2619c98618b8a94907756a9c4ca205c56dc206922b30ce5998"} Oct 30 20:52:14.293870 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:52:14.293826 5378 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-1fdaa7/simulator-w4-1fdaa7-kserve-9866d468c-fwmsk" event={"ID":"09219832-010e-4ba8-bfb9-5a2884b358e1","Type":"ContainerStarted","Data":"d1292e3c66c3d98af24e6c68f24d992a1a06c1982f6b391d6a7ac03d12b445f8"} Oct 30 20:52:14.409783 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:52:14.409733 5378 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w4-1fdaa7/simulator-w4-1fdaa7-kserve-9866d468c-fwmsk" podStartSLOduration=1.409721285 podStartE2EDuration="1.409721285s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:52:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:52:14.408700544 +0000 UTC m=+824.836814464" watchObservedRunningTime="2026-10-30 20:52:14.409721285 +0000 UTC m=+824.837835203" Oct 30 20:52:14.574157 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:52:14.574083 5378 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-1fdaa7/simulator-w4-1fdaa7-kserve-9866d468c-fwmsk" Oct 30 20:52:16.772199 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:52:16.772166 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 20:52:17.457642 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:52:17.457609 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 20:52:18.836802 ip-10-0-138-149 kubenswrapper[5378]: E1030 20:52:18.836764 5378 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q6mx7" containerName="extractor" Oct 30 20:52:21.675662 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:52:21.675629 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 20:52:23.554178 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:52:23.554144 5378 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-r9g59"] Oct 30 20:52:23.574717 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:52:23.574685 5378 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w4-1fdaa7/simulator-w4-1fdaa7-kserve-9866d468c-fwmsk" Oct 30 20:52:23.594585 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:52:23.593830 5378 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w4-1fdaa7/simulator-w4-1fdaa7-kserve-9866d468c-fwmsk" Oct 30 20:52:23.858591 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:52:23.858549 5378 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w2-d061d0/simulator-w2-d061d0-kserve-8484f9678b-lnszm"] Oct 30 20:52:23.862737 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:52:23.862709 5378 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-d061d0/simulator-w2-d061d0-kserve-8484f9678b-lnszm" Oct 30 20:52:23.921063 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:52:23.921032 5378 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-d061d0/simulator-w2-d061d0-kserve-8484f9678b-lnszm"] Oct 30 20:52:23.922008 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:52:23.921981 5378 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-d061d0\"/\"simulator-w2-d061d0-kserve-self-signed-certs\"" Oct 30 20:52:23.922297 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:52:23.922267 5378 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-d061d0\"/\"default-dockercfg-nxxqc\"" Oct 30 20:52:23.922476 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:52:23.922461 5378 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w2-d061d0\"/\"kube-root-ca.crt\"" Oct 30 20:52:23.922650 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:52:23.922627 5378 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w2-d061d0\"/\"openshift-service-ca.crt\"" Oct 30 20:52:23.928173 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:52:23.928149 5378 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/525fac02-b21c-4dd5-b553-72bcc36062c5-home\") pod \"simulator-w2-d061d0-kserve-8484f9678b-lnszm\" (UID: \"525fac02-b21c-4dd5-b553-72bcc36062c5\") " pod="e2e-models-e2e-worker-w2-d061d0/simulator-w2-d061d0-kserve-8484f9678b-lnszm" Oct 30 20:52:23.928315 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:52:23.928297 5378 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n2nfj\" (UniqueName: \"kubernetes.io/projected/525fac02-b21c-4dd5-b553-72bcc36062c5-kube-api-access-n2nfj\") pod \"simulator-w2-d061d0-kserve-8484f9678b-lnszm\" (UID: \"525fac02-b21c-4dd5-b553-72bcc36062c5\") " pod="e2e-models-e2e-worker-w2-d061d0/simulator-w2-d061d0-kserve-8484f9678b-lnszm" Oct 30 20:52:23.928419 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:52:23.928402 5378 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/525fac02-b21c-4dd5-b553-72bcc36062c5-tmp-dir\") pod \"simulator-w2-d061d0-kserve-8484f9678b-lnszm\" (UID: \"525fac02-b21c-4dd5-b553-72bcc36062c5\") " pod="e2e-models-e2e-worker-w2-d061d0/simulator-w2-d061d0-kserve-8484f9678b-lnszm" Oct 30 20:52:23.928520 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:52:23.928504 5378 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/525fac02-b21c-4dd5-b553-72bcc36062c5-dshm\") pod \"simulator-w2-d061d0-kserve-8484f9678b-lnszm\" (UID: \"525fac02-b21c-4dd5-b553-72bcc36062c5\") " pod="e2e-models-e2e-worker-w2-d061d0/simulator-w2-d061d0-kserve-8484f9678b-lnszm" Oct 30 20:52:23.928608 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:52:23.928551 5378 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/525fac02-b21c-4dd5-b553-72bcc36062c5-tls-certs\") pod \"simulator-w2-d061d0-kserve-8484f9678b-lnszm\" (UID: \"525fac02-b21c-4dd5-b553-72bcc36062c5\") " pod="e2e-models-e2e-worker-w2-d061d0/simulator-w2-d061d0-kserve-8484f9678b-lnszm" Oct 30 20:52:23.928690 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:52:23.928672 5378 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/525fac02-b21c-4dd5-b553-72bcc36062c5-model-cache\") pod \"simulator-w2-d061d0-kserve-8484f9678b-lnszm\" (UID: \"525fac02-b21c-4dd5-b553-72bcc36062c5\") " pod="e2e-models-e2e-worker-w2-d061d0/simulator-w2-d061d0-kserve-8484f9678b-lnszm" Oct 30 20:52:23.956604 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:52:23.956575 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 20:52:24.042676 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:52:24.031290 5378 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n2nfj\" (UniqueName: \"kubernetes.io/projected/525fac02-b21c-4dd5-b553-72bcc36062c5-kube-api-access-n2nfj\") pod \"simulator-w2-d061d0-kserve-8484f9678b-lnszm\" (UID: \"525fac02-b21c-4dd5-b553-72bcc36062c5\") " pod="e2e-models-e2e-worker-w2-d061d0/simulator-w2-d061d0-kserve-8484f9678b-lnszm" Oct 30 20:52:24.042676 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:52:24.031367 5378 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/525fac02-b21c-4dd5-b553-72bcc36062c5-tmp-dir\") pod \"simulator-w2-d061d0-kserve-8484f9678b-lnszm\" (UID: \"525fac02-b21c-4dd5-b553-72bcc36062c5\") " pod="e2e-models-e2e-worker-w2-d061d0/simulator-w2-d061d0-kserve-8484f9678b-lnszm" Oct 30 20:52:24.042676 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:52:24.031457 5378 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/525fac02-b21c-4dd5-b553-72bcc36062c5-dshm\") pod \"simulator-w2-d061d0-kserve-8484f9678b-lnszm\" (UID: \"525fac02-b21c-4dd5-b553-72bcc36062c5\") " pod="e2e-models-e2e-worker-w2-d061d0/simulator-w2-d061d0-kserve-8484f9678b-lnszm" Oct 30 20:52:24.042676 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:52:24.031495 5378 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/525fac02-b21c-4dd5-b553-72bcc36062c5-tls-certs\") pod \"simulator-w2-d061d0-kserve-8484f9678b-lnszm\" (UID: \"525fac02-b21c-4dd5-b553-72bcc36062c5\") " pod="e2e-models-e2e-worker-w2-d061d0/simulator-w2-d061d0-kserve-8484f9678b-lnszm" Oct 30 20:52:24.042676 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:52:24.031612 5378 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/525fac02-b21c-4dd5-b553-72bcc36062c5-model-cache\") pod \"simulator-w2-d061d0-kserve-8484f9678b-lnszm\" (UID: \"525fac02-b21c-4dd5-b553-72bcc36062c5\") " pod="e2e-models-e2e-worker-w2-d061d0/simulator-w2-d061d0-kserve-8484f9678b-lnszm" Oct 30 20:52:24.042676 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:52:24.031666 5378 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/525fac02-b21c-4dd5-b553-72bcc36062c5-home\") pod \"simulator-w2-d061d0-kserve-8484f9678b-lnszm\" (UID: \"525fac02-b21c-4dd5-b553-72bcc36062c5\") " pod="e2e-models-e2e-worker-w2-d061d0/simulator-w2-d061d0-kserve-8484f9678b-lnszm" Oct 30 20:52:24.042676 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:52:24.032188 5378 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/525fac02-b21c-4dd5-b553-72bcc36062c5-home\") pod \"simulator-w2-d061d0-kserve-8484f9678b-lnszm\" (UID: \"525fac02-b21c-4dd5-b553-72bcc36062c5\") " pod="e2e-models-e2e-worker-w2-d061d0/simulator-w2-d061d0-kserve-8484f9678b-lnszm" Oct 30 20:52:24.042676 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:52:24.032789 5378 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/525fac02-b21c-4dd5-b553-72bcc36062c5-tmp-dir\") pod \"simulator-w2-d061d0-kserve-8484f9678b-lnszm\" (UID: \"525fac02-b21c-4dd5-b553-72bcc36062c5\") " pod="e2e-models-e2e-worker-w2-d061d0/simulator-w2-d061d0-kserve-8484f9678b-lnszm" Oct 30 20:52:24.042676 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:52:24.033779 5378 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/525fac02-b21c-4dd5-b553-72bcc36062c5-model-cache\") pod \"simulator-w2-d061d0-kserve-8484f9678b-lnszm\" (UID: \"525fac02-b21c-4dd5-b553-72bcc36062c5\") " pod="e2e-models-e2e-worker-w2-d061d0/simulator-w2-d061d0-kserve-8484f9678b-lnszm" Oct 30 20:52:24.043264 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:52:24.042950 5378 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/525fac02-b21c-4dd5-b553-72bcc36062c5-dshm\") pod \"simulator-w2-d061d0-kserve-8484f9678b-lnszm\" (UID: \"525fac02-b21c-4dd5-b553-72bcc36062c5\") " pod="e2e-models-e2e-worker-w2-d061d0/simulator-w2-d061d0-kserve-8484f9678b-lnszm" Oct 30 20:52:24.047586 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:52:24.045228 5378 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/525fac02-b21c-4dd5-b553-72bcc36062c5-tls-certs\") pod \"simulator-w2-d061d0-kserve-8484f9678b-lnszm\" (UID: \"525fac02-b21c-4dd5-b553-72bcc36062c5\") " pod="e2e-models-e2e-worker-w2-d061d0/simulator-w2-d061d0-kserve-8484f9678b-lnszm" Oct 30 20:52:24.089583 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:52:24.089525 5378 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n2nfj\" (UniqueName: \"kubernetes.io/projected/525fac02-b21c-4dd5-b553-72bcc36062c5-kube-api-access-n2nfj\") pod \"simulator-w2-d061d0-kserve-8484f9678b-lnszm\" (UID: \"525fac02-b21c-4dd5-b553-72bcc36062c5\") " pod="e2e-models-e2e-worker-w2-d061d0/simulator-w2-d061d0-kserve-8484f9678b-lnszm" Oct 30 20:52:24.178795 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:52:24.178697 5378 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-d061d0/simulator-w2-d061d0-kserve-8484f9678b-lnszm" Oct 30 20:52:24.369173 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:52:24.369136 5378 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-1fdaa7/simulator-w4-1fdaa7-kserve-9866d468c-fwmsk" Oct 30 20:52:24.628795 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:52:24.628763 5378 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-d061d0/simulator-w2-d061d0-kserve-8484f9678b-lnszm"] Oct 30 20:52:24.629733 ip-10-0-138-149 kubenswrapper[5378]: W1030 20:52:24.629706 5378 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod525fac02_b21c_4dd5_b553_72bcc36062c5.slice/crio-4a5c2994f6a916c441c0b7b26adee925ceaa6169372f6cd5ca3e911d57fb23fe WatchSource:0}: Error finding container 4a5c2994f6a916c441c0b7b26adee925ceaa6169372f6cd5ca3e911d57fb23fe: Status 404 returned error can't find the container with id 4a5c2994f6a916c441c0b7b26adee925ceaa6169372f6cd5ca3e911d57fb23fe Oct 30 20:52:25.356634 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:52:25.356594 5378 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-d061d0/simulator-w2-d061d0-kserve-8484f9678b-lnszm" event={"ID":"525fac02-b21c-4dd5-b553-72bcc36062c5","Type":"ContainerStarted","Data":"d9dc8acba58ff31b072e5ad4cf311194b0404765220cf6074ef3265a4246dd5c"} Oct 30 20:52:25.356802 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:52:25.356651 5378 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-d061d0/simulator-w2-d061d0-kserve-8484f9678b-lnszm" event={"ID":"525fac02-b21c-4dd5-b553-72bcc36062c5","Type":"ContainerStarted","Data":"4a5c2994f6a916c441c0b7b26adee925ceaa6169372f6cd5ca3e911d57fb23fe"} Oct 30 20:52:25.474642 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:52:25.474560 5378 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w2-d061d0/simulator-w2-d061d0-kserve-8484f9678b-lnszm" podStartSLOduration=2.474545075 podStartE2EDuration="2.474545075s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:52:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:52:25.473525569 +0000 UTC m=+835.901639510" watchObservedRunningTime="2026-10-30 20:52:25.474545075 +0000 UTC m=+835.902658993" Oct 30 20:52:26.179294 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:52:26.179260 5378 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-d061d0/simulator-w2-d061d0-kserve-8484f9678b-lnszm" Oct 30 20:52:26.767553 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:52:26.767520 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 20:52:31.680928 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:52:31.680885 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 20:52:34.179833 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:52:34.179797 5378 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w2-d061d0/simulator-w2-d061d0-kserve-8484f9678b-lnszm" Oct 30 20:52:34.192167 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:52:34.192142 5378 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w2-d061d0/simulator-w2-d061d0-kserve-8484f9678b-lnszm" Oct 30 20:52:34.412441 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:52:34.412411 5378 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-d061d0/simulator-w2-d061d0-kserve-8484f9678b-lnszm" Oct 30 20:52:36.580634 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:52:36.580603 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 20:52:40.477033 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:52:40.476996 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 20:52:41.650498 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:52:41.650454 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 20:52:43.515936 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:52:43.515900 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 20:52:46.315808 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:52:46.315776 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 20:52:48.836267 ip-10-0-138-149 kubenswrapper[5378]: E1030 20:52:48.836228 5378 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q6mx7" containerName="extractor" Oct 30 20:52:51.263093 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:52:51.263063 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 20:52:51.787861 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:52:51.787829 5378 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-r9g59"] Oct 30 20:52:56.267167 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:52:56.267130 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 20:53:01.162903 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:53:01.162872 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 20:53:06.097260 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:53:06.097222 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 20:53:10.885889 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:53:10.885817 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 20:53:14.301198 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:53:14.301165 5378 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w6-8fa2a7/premium-w6-8fa2a7-kserve-64bf4bf6c-dxrdr"] Oct 30 20:53:14.304908 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:53:14.304890 5378 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-8fa2a7/premium-w6-8fa2a7-kserve-64bf4bf6c-dxrdr" Oct 30 20:53:14.368978 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:53:14.368948 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 20:53:14.410796 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:53:14.410769 5378 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w6-8fa2a7\"/\"premium-w6-8fa2a7-kserve-self-signed-certs\"" Oct 30 20:53:14.413579 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:53:14.413546 5378 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w6-8fa2a7/premium-w6-8fa2a7-kserve-64bf4bf6c-dxrdr"] Oct 30 20:53:14.490121 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:53:14.490035 5378 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/484cba5f-b415-430a-b52d-50b1293be948-tmp-dir\") pod \"premium-w6-8fa2a7-kserve-64bf4bf6c-dxrdr\" (UID: \"484cba5f-b415-430a-b52d-50b1293be948\") " pod="e2e-models-e2e-worker-w6-8fa2a7/premium-w6-8fa2a7-kserve-64bf4bf6c-dxrdr" Oct 30 20:53:14.490340 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:53:14.490190 5378 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/484cba5f-b415-430a-b52d-50b1293be948-dshm\") pod \"premium-w6-8fa2a7-kserve-64bf4bf6c-dxrdr\" (UID: \"484cba5f-b415-430a-b52d-50b1293be948\") " pod="e2e-models-e2e-worker-w6-8fa2a7/premium-w6-8fa2a7-kserve-64bf4bf6c-dxrdr" Oct 30 20:53:14.490340 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:53:14.490265 5378 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/484cba5f-b415-430a-b52d-50b1293be948-tls-certs\") pod \"premium-w6-8fa2a7-kserve-64bf4bf6c-dxrdr\" (UID: \"484cba5f-b415-430a-b52d-50b1293be948\") " pod="e2e-models-e2e-worker-w6-8fa2a7/premium-w6-8fa2a7-kserve-64bf4bf6c-dxrdr" Oct 30 20:53:14.490340 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:53:14.490314 5378 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-78w6l\" (UniqueName: \"kubernetes.io/projected/484cba5f-b415-430a-b52d-50b1293be948-kube-api-access-78w6l\") pod \"premium-w6-8fa2a7-kserve-64bf4bf6c-dxrdr\" (UID: \"484cba5f-b415-430a-b52d-50b1293be948\") " pod="e2e-models-e2e-worker-w6-8fa2a7/premium-w6-8fa2a7-kserve-64bf4bf6c-dxrdr" Oct 30 20:53:14.490503 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:53:14.490421 5378 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/484cba5f-b415-430a-b52d-50b1293be948-home\") pod \"premium-w6-8fa2a7-kserve-64bf4bf6c-dxrdr\" (UID: \"484cba5f-b415-430a-b52d-50b1293be948\") " pod="e2e-models-e2e-worker-w6-8fa2a7/premium-w6-8fa2a7-kserve-64bf4bf6c-dxrdr" Oct 30 20:53:14.490503 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:53:14.490462 5378 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/484cba5f-b415-430a-b52d-50b1293be948-model-cache\") pod \"premium-w6-8fa2a7-kserve-64bf4bf6c-dxrdr\" (UID: \"484cba5f-b415-430a-b52d-50b1293be948\") " pod="e2e-models-e2e-worker-w6-8fa2a7/premium-w6-8fa2a7-kserve-64bf4bf6c-dxrdr" Oct 30 20:53:14.499670 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:53:14.499639 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 20:53:14.591201 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:53:14.591120 5378 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/484cba5f-b415-430a-b52d-50b1293be948-dshm\") pod \"premium-w6-8fa2a7-kserve-64bf4bf6c-dxrdr\" (UID: \"484cba5f-b415-430a-b52d-50b1293be948\") " pod="e2e-models-e2e-worker-w6-8fa2a7/premium-w6-8fa2a7-kserve-64bf4bf6c-dxrdr" Oct 30 20:53:14.591201 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:53:14.591199 5378 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/484cba5f-b415-430a-b52d-50b1293be948-tls-certs\") pod \"premium-w6-8fa2a7-kserve-64bf4bf6c-dxrdr\" (UID: \"484cba5f-b415-430a-b52d-50b1293be948\") " pod="e2e-models-e2e-worker-w6-8fa2a7/premium-w6-8fa2a7-kserve-64bf4bf6c-dxrdr" Oct 30 20:53:14.591415 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:53:14.591229 5378 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-78w6l\" (UniqueName: \"kubernetes.io/projected/484cba5f-b415-430a-b52d-50b1293be948-kube-api-access-78w6l\") pod \"premium-w6-8fa2a7-kserve-64bf4bf6c-dxrdr\" (UID: \"484cba5f-b415-430a-b52d-50b1293be948\") " pod="e2e-models-e2e-worker-w6-8fa2a7/premium-w6-8fa2a7-kserve-64bf4bf6c-dxrdr" Oct 30 20:53:14.591415 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:53:14.591262 5378 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/484cba5f-b415-430a-b52d-50b1293be948-home\") pod \"premium-w6-8fa2a7-kserve-64bf4bf6c-dxrdr\" (UID: \"484cba5f-b415-430a-b52d-50b1293be948\") " pod="e2e-models-e2e-worker-w6-8fa2a7/premium-w6-8fa2a7-kserve-64bf4bf6c-dxrdr" Oct 30 20:53:14.591415 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:53:14.591366 5378 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/484cba5f-b415-430a-b52d-50b1293be948-model-cache\") pod \"premium-w6-8fa2a7-kserve-64bf4bf6c-dxrdr\" (UID: \"484cba5f-b415-430a-b52d-50b1293be948\") " pod="e2e-models-e2e-worker-w6-8fa2a7/premium-w6-8fa2a7-kserve-64bf4bf6c-dxrdr" Oct 30 20:53:14.591630 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:53:14.591442 5378 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/484cba5f-b415-430a-b52d-50b1293be948-tmp-dir\") pod \"premium-w6-8fa2a7-kserve-64bf4bf6c-dxrdr\" (UID: \"484cba5f-b415-430a-b52d-50b1293be948\") " pod="e2e-models-e2e-worker-w6-8fa2a7/premium-w6-8fa2a7-kserve-64bf4bf6c-dxrdr" Oct 30 20:53:14.591742 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:53:14.591718 5378 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/484cba5f-b415-430a-b52d-50b1293be948-home\") pod \"premium-w6-8fa2a7-kserve-64bf4bf6c-dxrdr\" (UID: \"484cba5f-b415-430a-b52d-50b1293be948\") " pod="e2e-models-e2e-worker-w6-8fa2a7/premium-w6-8fa2a7-kserve-64bf4bf6c-dxrdr" Oct 30 20:53:14.591819 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:53:14.591794 5378 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/484cba5f-b415-430a-b52d-50b1293be948-model-cache\") pod \"premium-w6-8fa2a7-kserve-64bf4bf6c-dxrdr\" (UID: \"484cba5f-b415-430a-b52d-50b1293be948\") " pod="e2e-models-e2e-worker-w6-8fa2a7/premium-w6-8fa2a7-kserve-64bf4bf6c-dxrdr" Oct 30 20:53:14.591881 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:53:14.591838 5378 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/484cba5f-b415-430a-b52d-50b1293be948-tmp-dir\") pod \"premium-w6-8fa2a7-kserve-64bf4bf6c-dxrdr\" (UID: \"484cba5f-b415-430a-b52d-50b1293be948\") " pod="e2e-models-e2e-worker-w6-8fa2a7/premium-w6-8fa2a7-kserve-64bf4bf6c-dxrdr" Oct 30 20:53:14.593548 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:53:14.593522 5378 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/484cba5f-b415-430a-b52d-50b1293be948-dshm\") pod \"premium-w6-8fa2a7-kserve-64bf4bf6c-dxrdr\" (UID: \"484cba5f-b415-430a-b52d-50b1293be948\") " pod="e2e-models-e2e-worker-w6-8fa2a7/premium-w6-8fa2a7-kserve-64bf4bf6c-dxrdr" Oct 30 20:53:14.593879 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:53:14.593864 5378 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/484cba5f-b415-430a-b52d-50b1293be948-tls-certs\") pod \"premium-w6-8fa2a7-kserve-64bf4bf6c-dxrdr\" (UID: \"484cba5f-b415-430a-b52d-50b1293be948\") " pod="e2e-models-e2e-worker-w6-8fa2a7/premium-w6-8fa2a7-kserve-64bf4bf6c-dxrdr" Oct 30 20:53:14.648736 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:53:14.648711 5378 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-78w6l\" (UniqueName: \"kubernetes.io/projected/484cba5f-b415-430a-b52d-50b1293be948-kube-api-access-78w6l\") pod \"premium-w6-8fa2a7-kserve-64bf4bf6c-dxrdr\" (UID: \"484cba5f-b415-430a-b52d-50b1293be948\") " pod="e2e-models-e2e-worker-w6-8fa2a7/premium-w6-8fa2a7-kserve-64bf4bf6c-dxrdr" Oct 30 20:53:14.917377 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:53:14.917339 5378 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-8fa2a7/premium-w6-8fa2a7-kserve-64bf4bf6c-dxrdr" Oct 30 20:53:15.291974 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:53:15.291946 5378 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w6-8fa2a7/premium-w6-8fa2a7-kserve-64bf4bf6c-dxrdr"] Oct 30 20:53:15.293415 ip-10-0-138-149 kubenswrapper[5378]: W1030 20:53:15.293384 5378 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod484cba5f_b415_430a_b52d_50b1293be948.slice/crio-c12109ae609d50415f5a179b8abc64886aa92c88edda51853e4094c865576024 WatchSource:0}: Error finding container c12109ae609d50415f5a179b8abc64886aa92c88edda51853e4094c865576024: Status 404 returned error can't find the container with id c12109ae609d50415f5a179b8abc64886aa92c88edda51853e4094c865576024 Oct 30 20:53:15.629292 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:53:15.629252 5378 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-8fa2a7/premium-w6-8fa2a7-kserve-64bf4bf6c-dxrdr" event={"ID":"484cba5f-b415-430a-b52d-50b1293be948","Type":"ContainerStarted","Data":"e39bdfd69306cfe0ef86ecd80cdbece184e6a6c7740bdee328493f0ab43ce75f"} Oct 30 20:53:15.629292 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:53:15.629297 5378 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-8fa2a7/premium-w6-8fa2a7-kserve-64bf4bf6c-dxrdr" event={"ID":"484cba5f-b415-430a-b52d-50b1293be948","Type":"ContainerStarted","Data":"c12109ae609d50415f5a179b8abc64886aa92c88edda51853e4094c865576024"} Oct 30 20:53:15.699057 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:53:15.699020 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 20:53:15.762608 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:53:15.749298 5378 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w6-8fa2a7/premium-w6-8fa2a7-kserve-64bf4bf6c-dxrdr" podStartSLOduration=1.749279221 podStartE2EDuration="1.749279221s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:53:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:53:15.745141674 +0000 UTC m=+886.173255593" watchObservedRunningTime="2026-10-30 20:53:15.749279221 +0000 UTC m=+886.177393138" Oct 30 20:53:15.918041 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:53:15.917965 5378 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-8fa2a7/premium-w6-8fa2a7-kserve-64bf4bf6c-dxrdr" Oct 30 20:53:18.836531 ip-10-0-138-149 kubenswrapper[5378]: E1030 20:53:18.836498 5378 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q6mx7" containerName="extractor" Oct 30 20:53:20.497775 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:53:20.497737 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 20:53:23.610283 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:53:23.610249 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 20:53:24.917561 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:53:24.917524 5378 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w6-8fa2a7/premium-w6-8fa2a7-kserve-64bf4bf6c-dxrdr" Oct 30 20:53:24.930642 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:53:24.930617 5378 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w6-8fa2a7/premium-w6-8fa2a7-kserve-64bf4bf6c-dxrdr" Oct 30 20:53:25.455767 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:53:25.455729 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 20:53:25.697659 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:53:25.697625 5378 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-8fa2a7/premium-w6-8fa2a7-kserve-64bf4bf6c-dxrdr" Oct 30 20:53:25.880253 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:53:25.880222 5378 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-r9g59"] Oct 30 20:53:28.851763 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:53:28.851728 5378 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4939c02d-fm4s4"] Oct 30 20:53:28.855956 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:53:28.855939 5378 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4939c02d-fm4s4" Oct 30 20:53:28.912021 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:53:28.911986 5378 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 20:53:28.912188 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:53:28.912117 5378 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-q2mgc\"" Oct 30 20:53:28.940235 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:53:28.940205 5378 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/563c5c7c-a42c-417c-b883-98efbbef92ca-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4939c02d-fm4s4\" (UID: \"563c5c7c-a42c-417c-b883-98efbbef92ca\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4939c02d-fm4s4" Oct 30 20:53:28.940403 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:53:28.940384 5378 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7ck2p\" (UniqueName: \"kubernetes.io/projected/563c5c7c-a42c-417c-b883-98efbbef92ca-kube-api-access-7ck2p\") pod \"maas-api-revoke-sub-4939c02d-fm4s4\" (UID: \"563c5c7c-a42c-417c-b883-98efbbef92ca\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4939c02d-fm4s4" Oct 30 20:53:28.940514 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:53:28.940493 5378 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/563c5c7c-a42c-417c-b883-98efbbef92ca-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4939c02d-fm4s4\" (UID: \"563c5c7c-a42c-417c-b883-98efbbef92ca\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4939c02d-fm4s4" Oct 30 20:53:28.964607 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:53:28.964580 5378 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4939c02d-fm4s4"] Oct 30 20:53:29.041067 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:53:29.041031 5378 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7ck2p\" (UniqueName: \"kubernetes.io/projected/563c5c7c-a42c-417c-b883-98efbbef92ca-kube-api-access-7ck2p\") pod \"maas-api-revoke-sub-4939c02d-fm4s4\" (UID: \"563c5c7c-a42c-417c-b883-98efbbef92ca\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4939c02d-fm4s4" Oct 30 20:53:29.041234 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:53:29.041092 5378 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/563c5c7c-a42c-417c-b883-98efbbef92ca-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4939c02d-fm4s4\" (UID: \"563c5c7c-a42c-417c-b883-98efbbef92ca\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4939c02d-fm4s4" Oct 30 20:53:29.041234 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:53:29.041131 5378 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/563c5c7c-a42c-417c-b883-98efbbef92ca-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4939c02d-fm4s4\" (UID: \"563c5c7c-a42c-417c-b883-98efbbef92ca\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4939c02d-fm4s4" Oct 30 20:53:29.041921 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:53:29.041894 5378 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/563c5c7c-a42c-417c-b883-98efbbef92ca-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4939c02d-fm4s4\" (UID: \"563c5c7c-a42c-417c-b883-98efbbef92ca\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4939c02d-fm4s4" Oct 30 20:53:29.043576 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:53:29.043542 5378 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/563c5c7c-a42c-417c-b883-98efbbef92ca-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4939c02d-fm4s4\" (UID: \"563c5c7c-a42c-417c-b883-98efbbef92ca\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4939c02d-fm4s4" Oct 30 20:53:29.099033 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:53:29.099002 5378 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7ck2p\" (UniqueName: \"kubernetes.io/projected/563c5c7c-a42c-417c-b883-98efbbef92ca-kube-api-access-7ck2p\") pod \"maas-api-revoke-sub-4939c02d-fm4s4\" (UID: \"563c5c7c-a42c-417c-b883-98efbbef92ca\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4939c02d-fm4s4" Oct 30 20:53:29.169800 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:53:29.169735 5378 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4939c02d-fm4s4" Oct 30 20:53:29.547711 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:53:29.547681 5378 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4939c02d-fm4s4"] Oct 30 20:53:29.708733 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:53:29.708701 5378 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4939c02d-fm4s4" event={"ID":"563c5c7c-a42c-417c-b883-98efbbef92ca","Type":"ContainerStarted","Data":"8bedd3f6d031bd4375772614be24a465ca047fd036159030a9f1ed238069c8b8"} Oct 30 20:53:29.816587 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:53:29.816500 5378 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-7x9jg_c2ff359c-cf92-4a5b-8610-f6eb41acbe73/ovn-acl-logging/0.log" Oct 30 20:53:29.818273 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:53:29.818250 5378 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-7x9jg_c2ff359c-cf92-4a5b-8610-f6eb41acbe73/ovn-acl-logging/0.log" Oct 30 20:53:30.250808 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:53:30.250780 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 20:53:33.008235 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:53:33.008203 5378 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w4-1fdaa7/premium-w4-1fdaa7-kserve-55dffbd5cd-gvpxq"] Oct 30 20:53:33.013449 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:53:33.013430 5378 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-1fdaa7/premium-w4-1fdaa7-kserve-55dffbd5cd-gvpxq" Oct 30 20:53:33.080476 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:53:33.080440 5378 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ec282a4b-742f-4382-a296-335dcd2e9636-home\") pod \"premium-w4-1fdaa7-kserve-55dffbd5cd-gvpxq\" (UID: \"ec282a4b-742f-4382-a296-335dcd2e9636\") " pod="e2e-models-e2e-worker-w4-1fdaa7/premium-w4-1fdaa7-kserve-55dffbd5cd-gvpxq" Oct 30 20:53:33.080675 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:53:33.080493 5378 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ec282a4b-742f-4382-a296-335dcd2e9636-model-cache\") pod \"premium-w4-1fdaa7-kserve-55dffbd5cd-gvpxq\" (UID: \"ec282a4b-742f-4382-a296-335dcd2e9636\") " pod="e2e-models-e2e-worker-w4-1fdaa7/premium-w4-1fdaa7-kserve-55dffbd5cd-gvpxq" Oct 30 20:53:33.080675 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:53:33.080595 5378 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ec282a4b-742f-4382-a296-335dcd2e9636-tmp-dir\") pod \"premium-w4-1fdaa7-kserve-55dffbd5cd-gvpxq\" (UID: \"ec282a4b-742f-4382-a296-335dcd2e9636\") " pod="e2e-models-e2e-worker-w4-1fdaa7/premium-w4-1fdaa7-kserve-55dffbd5cd-gvpxq" Oct 30 20:53:33.080675 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:53:33.080650 5378 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2qfhl\" (UniqueName: \"kubernetes.io/projected/ec282a4b-742f-4382-a296-335dcd2e9636-kube-api-access-2qfhl\") pod \"premium-w4-1fdaa7-kserve-55dffbd5cd-gvpxq\" (UID: \"ec282a4b-742f-4382-a296-335dcd2e9636\") " pod="e2e-models-e2e-worker-w4-1fdaa7/premium-w4-1fdaa7-kserve-55dffbd5cd-gvpxq" Oct 30 20:53:33.080865 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:53:33.080751 5378 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ec282a4b-742f-4382-a296-335dcd2e9636-dshm\") pod \"premium-w4-1fdaa7-kserve-55dffbd5cd-gvpxq\" (UID: \"ec282a4b-742f-4382-a296-335dcd2e9636\") " pod="e2e-models-e2e-worker-w4-1fdaa7/premium-w4-1fdaa7-kserve-55dffbd5cd-gvpxq" Oct 30 20:53:33.080865 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:53:33.080796 5378 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ec282a4b-742f-4382-a296-335dcd2e9636-tls-certs\") pod \"premium-w4-1fdaa7-kserve-55dffbd5cd-gvpxq\" (UID: \"ec282a4b-742f-4382-a296-335dcd2e9636\") " pod="e2e-models-e2e-worker-w4-1fdaa7/premium-w4-1fdaa7-kserve-55dffbd5cd-gvpxq" Oct 30 20:53:33.101265 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:53:33.101227 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 20:53:33.116868 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:53:33.116842 5378 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-1fdaa7\"/\"premium-w4-1fdaa7-kserve-self-signed-certs\"" Oct 30 20:53:33.118874 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:53:33.118856 5378 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-1fdaa7/premium-w4-1fdaa7-kserve-55dffbd5cd-gvpxq"] Oct 30 20:53:33.186205 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:53:33.186165 5378 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ec282a4b-742f-4382-a296-335dcd2e9636-tls-certs\") pod \"premium-w4-1fdaa7-kserve-55dffbd5cd-gvpxq\" (UID: \"ec282a4b-742f-4382-a296-335dcd2e9636\") " pod="e2e-models-e2e-worker-w4-1fdaa7/premium-w4-1fdaa7-kserve-55dffbd5cd-gvpxq" Oct 30 20:53:33.186404 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:53:33.186285 5378 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ec282a4b-742f-4382-a296-335dcd2e9636-home\") pod \"premium-w4-1fdaa7-kserve-55dffbd5cd-gvpxq\" (UID: \"ec282a4b-742f-4382-a296-335dcd2e9636\") " pod="e2e-models-e2e-worker-w4-1fdaa7/premium-w4-1fdaa7-kserve-55dffbd5cd-gvpxq" Oct 30 20:53:33.186404 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:53:33.186361 5378 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ec282a4b-742f-4382-a296-335dcd2e9636-model-cache\") pod \"premium-w4-1fdaa7-kserve-55dffbd5cd-gvpxq\" (UID: \"ec282a4b-742f-4382-a296-335dcd2e9636\") " pod="e2e-models-e2e-worker-w4-1fdaa7/premium-w4-1fdaa7-kserve-55dffbd5cd-gvpxq" Oct 30 20:53:33.186404 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:53:33.186394 5378 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ec282a4b-742f-4382-a296-335dcd2e9636-tmp-dir\") pod \"premium-w4-1fdaa7-kserve-55dffbd5cd-gvpxq\" (UID: \"ec282a4b-742f-4382-a296-335dcd2e9636\") " pod="e2e-models-e2e-worker-w4-1fdaa7/premium-w4-1fdaa7-kserve-55dffbd5cd-gvpxq" Oct 30 20:53:33.186624 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:53:33.186434 5378 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2qfhl\" (UniqueName: \"kubernetes.io/projected/ec282a4b-742f-4382-a296-335dcd2e9636-kube-api-access-2qfhl\") pod \"premium-w4-1fdaa7-kserve-55dffbd5cd-gvpxq\" (UID: \"ec282a4b-742f-4382-a296-335dcd2e9636\") " pod="e2e-models-e2e-worker-w4-1fdaa7/premium-w4-1fdaa7-kserve-55dffbd5cd-gvpxq" Oct 30 20:53:33.186624 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:53:33.186520 5378 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ec282a4b-742f-4382-a296-335dcd2e9636-dshm\") pod \"premium-w4-1fdaa7-kserve-55dffbd5cd-gvpxq\" (UID: \"ec282a4b-742f-4382-a296-335dcd2e9636\") " pod="e2e-models-e2e-worker-w4-1fdaa7/premium-w4-1fdaa7-kserve-55dffbd5cd-gvpxq" Oct 30 20:53:33.186936 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:53:33.186913 5378 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ec282a4b-742f-4382-a296-335dcd2e9636-model-cache\") pod \"premium-w4-1fdaa7-kserve-55dffbd5cd-gvpxq\" (UID: \"ec282a4b-742f-4382-a296-335dcd2e9636\") " pod="e2e-models-e2e-worker-w4-1fdaa7/premium-w4-1fdaa7-kserve-55dffbd5cd-gvpxq" Oct 30 20:53:33.187204 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:53:33.187184 5378 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ec282a4b-742f-4382-a296-335dcd2e9636-home\") pod \"premium-w4-1fdaa7-kserve-55dffbd5cd-gvpxq\" (UID: \"ec282a4b-742f-4382-a296-335dcd2e9636\") " pod="e2e-models-e2e-worker-w4-1fdaa7/premium-w4-1fdaa7-kserve-55dffbd5cd-gvpxq" Oct 30 20:53:33.187278 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:53:33.187220 5378 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ec282a4b-742f-4382-a296-335dcd2e9636-tmp-dir\") pod \"premium-w4-1fdaa7-kserve-55dffbd5cd-gvpxq\" (UID: \"ec282a4b-742f-4382-a296-335dcd2e9636\") " pod="e2e-models-e2e-worker-w4-1fdaa7/premium-w4-1fdaa7-kserve-55dffbd5cd-gvpxq" Oct 30 20:53:33.189162 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:53:33.189136 5378 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ec282a4b-742f-4382-a296-335dcd2e9636-dshm\") pod \"premium-w4-1fdaa7-kserve-55dffbd5cd-gvpxq\" (UID: \"ec282a4b-742f-4382-a296-335dcd2e9636\") " pod="e2e-models-e2e-worker-w4-1fdaa7/premium-w4-1fdaa7-kserve-55dffbd5cd-gvpxq" Oct 30 20:53:33.189330 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:53:33.189307 5378 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ec282a4b-742f-4382-a296-335dcd2e9636-tls-certs\") pod \"premium-w4-1fdaa7-kserve-55dffbd5cd-gvpxq\" (UID: \"ec282a4b-742f-4382-a296-335dcd2e9636\") " pod="e2e-models-e2e-worker-w4-1fdaa7/premium-w4-1fdaa7-kserve-55dffbd5cd-gvpxq" Oct 30 20:53:33.244886 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:53:33.244856 5378 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2qfhl\" (UniqueName: \"kubernetes.io/projected/ec282a4b-742f-4382-a296-335dcd2e9636-kube-api-access-2qfhl\") pod \"premium-w4-1fdaa7-kserve-55dffbd5cd-gvpxq\" (UID: \"ec282a4b-742f-4382-a296-335dcd2e9636\") " pod="e2e-models-e2e-worker-w4-1fdaa7/premium-w4-1fdaa7-kserve-55dffbd5cd-gvpxq" Oct 30 20:53:33.325868 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:53:33.325798 5378 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-1fdaa7/premium-w4-1fdaa7-kserve-55dffbd5cd-gvpxq" Oct 30 20:53:33.729473 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:53:33.729417 5378 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-1fdaa7/premium-w4-1fdaa7-kserve-55dffbd5cd-gvpxq"] Oct 30 20:53:33.730419 ip-10-0-138-149 kubenswrapper[5378]: W1030 20:53:33.730394 5378 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podec282a4b_742f_4382_a296_335dcd2e9636.slice/crio-f455a19947696c1ad00004267e6fde85cdaa293e6c9291f8698ec4178792ed5a WatchSource:0}: Error finding container f455a19947696c1ad00004267e6fde85cdaa293e6c9291f8698ec4178792ed5a: Status 404 returned error can't find the container with id f455a19947696c1ad00004267e6fde85cdaa293e6c9291f8698ec4178792ed5a Oct 30 20:53:34.737044 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:53:34.737006 5378 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-1fdaa7/premium-w4-1fdaa7-kserve-55dffbd5cd-gvpxq" event={"ID":"ec282a4b-742f-4382-a296-335dcd2e9636","Type":"ContainerStarted","Data":"582c981270282cf4b1bf2ee8310ecda4e6f18fde9ace8f30167166f6d9be40cb"} Oct 30 20:53:34.737044 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:53:34.737043 5378 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-1fdaa7/premium-w4-1fdaa7-kserve-55dffbd5cd-gvpxq" event={"ID":"ec282a4b-742f-4382-a296-335dcd2e9636","Type":"ContainerStarted","Data":"f455a19947696c1ad00004267e6fde85cdaa293e6c9291f8698ec4178792ed5a"} Oct 30 20:53:34.853416 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:53:34.853363 5378 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w4-1fdaa7/premium-w4-1fdaa7-kserve-55dffbd5cd-gvpxq" podStartSLOduration=2.853350395 podStartE2EDuration="2.853350395s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:53:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:53:34.851976824 +0000 UTC m=+905.280090744" watchObservedRunningTime="2026-10-30 20:53:34.853350395 +0000 UTC m=+905.281464328" Oct 30 20:53:35.218109 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:53:35.217995 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 20:53:35.326187 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:53:35.326153 5378 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-1fdaa7/premium-w4-1fdaa7-kserve-55dffbd5cd-gvpxq" Oct 30 20:53:38.633832 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:53:38.633729 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 20:53:40.054066 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:53:40.054034 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 20:53:40.773072 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:53:40.773039 5378 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4939c02d-fm4s4" event={"ID":"563c5c7c-a42c-417c-b883-98efbbef92ca","Type":"ContainerStarted","Data":"1f1196e486a3a5be083266069fbdacdb2adb74abb055d3457513abd9edae8aac"} Oct 30 20:53:40.806978 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:53:40.806950 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 20:53:40.887804 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:53:40.887758 5378 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4939c02d-fm4s4" podStartSLOduration=2.4382953 podStartE2EDuration="12.887745387s" totalImagesPullingTime="10.449450087s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:53:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:53:40.886926022 +0000 UTC m=+911.315039943" watchObservedRunningTime="2026-10-30 20:53:40.887745387 +0000 UTC m=+911.315859306" Oct 30 20:53:41.779810 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:53:41.779778 5378 generic.go:356] "Generic (PLEG): container finished" podID="563c5c7c-a42c-417c-b883-98efbbef92ca" containerID="1f1196e486a3a5be083266069fbdacdb2adb74abb055d3457513abd9edae8aac" exitCode=0 Oct 30 20:53:41.780199 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:53:41.779868 5378 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4939c02d-fm4s4" event={"ID":"563c5c7c-a42c-417c-b883-98efbbef92ca","Type":"ContainerDied","Data":"1f1196e486a3a5be083266069fbdacdb2adb74abb055d3457513abd9edae8aac"} Oct 30 20:53:42.929046 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:53:42.929020 5378 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4939c02d-fm4s4" Oct 30 20:53:43.004764 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:53:43.004730 5378 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/563c5c7c-a42c-417c-b883-98efbbef92ca-kube-api-access-7ck2p\" (UniqueName: \"kubernetes.io/projected/563c5c7c-a42c-417c-b883-98efbbef92ca-kube-api-access-7ck2p\") pod \"563c5c7c-a42c-417c-b883-98efbbef92ca\" (UID: \"563c5c7c-a42c-417c-b883-98efbbef92ca\") " Oct 30 20:53:43.004956 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:53:43.004804 5378 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/563c5c7c-a42c-417c-b883-98efbbef92ca-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/563c5c7c-a42c-417c-b883-98efbbef92ca-maas-api-openshift-ca\") pod \"563c5c7c-a42c-417c-b883-98efbbef92ca\" (UID: \"563c5c7c-a42c-417c-b883-98efbbef92ca\") " Oct 30 20:53:43.004956 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:53:43.004926 5378 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/563c5c7c-a42c-417c-b883-98efbbef92ca-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/563c5c7c-a42c-417c-b883-98efbbef92ca-maas-api-xks-ca\") pod \"563c5c7c-a42c-417c-b883-98efbbef92ca\" (UID: \"563c5c7c-a42c-417c-b883-98efbbef92ca\") " Oct 30 20:53:43.005221 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:53:43.005191 5378 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/563c5c7c-a42c-417c-b883-98efbbef92ca-maas-api-openshift-ca" pod "563c5c7c-a42c-417c-b883-98efbbef92ca" (UID: "563c5c7c-a42c-417c-b883-98efbbef92ca"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:53:43.005382 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:53:43.005367 5378 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/563c5c7c-a42c-417c-b883-98efbbef92ca-maas-api-openshift-ca\") on node \"ip-10-0-138-149.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:53:43.007093 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:53:43.007067 5378 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/563c5c7c-a42c-417c-b883-98efbbef92ca-kube-api-access-7ck2p" pod "563c5c7c-a42c-417c-b883-98efbbef92ca" (UID: "563c5c7c-a42c-417c-b883-98efbbef92ca"). InnerVolumeSpecName "kube-api-access-7ck2p". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:53:43.007256 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:53:43.007233 5378 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/563c5c7c-a42c-417c-b883-98efbbef92ca-maas-api-xks-ca" pod "563c5c7c-a42c-417c-b883-98efbbef92ca" (UID: "563c5c7c-a42c-417c-b883-98efbbef92ca"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 20:53:43.106672 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:53:43.106637 5378 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7ck2p\" (UniqueName: \"kubernetes.io/projected/563c5c7c-a42c-417c-b883-98efbbef92ca-kube-api-access-7ck2p\") on node \"ip-10-0-138-149.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:53:43.106672 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:53:43.106668 5378 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/563c5c7c-a42c-417c-b883-98efbbef92ca-maas-api-xks-ca\") on node \"ip-10-0-138-149.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:53:43.326760 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:53:43.326724 5378 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w4-1fdaa7/premium-w4-1fdaa7-kserve-55dffbd5cd-gvpxq" Oct 30 20:53:43.339104 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:53:43.339072 5378 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w4-1fdaa7/premium-w4-1fdaa7-kserve-55dffbd5cd-gvpxq" Oct 30 20:53:43.791531 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:53:43.791492 5378 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4939c02d-fm4s4" event={"ID":"563c5c7c-a42c-417c-b883-98efbbef92ca","Type":"ContainerDied","Data":"8bedd3f6d031bd4375772614be24a465ca047fd036159030a9f1ed238069c8b8"} Oct 30 20:53:43.791531 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:53:43.791516 5378 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4939c02d-fm4s4" Oct 30 20:53:43.791757 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:53:43.791547 5378 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8bedd3f6d031bd4375772614be24a465ca047fd036159030a9f1ed238069c8b8" Oct 30 20:53:43.803052 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:53:43.803022 5378 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-1fdaa7/premium-w4-1fdaa7-kserve-55dffbd5cd-gvpxq" Oct 30 20:53:44.955377 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:53:44.955342 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 20:53:48.836339 ip-10-0-138-149 kubenswrapper[5378]: E1030 20:53:48.836303 5378 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q6mx7" containerName="extractor" Oct 30 20:53:49.258383 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:53:49.258347 5378 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-r9g59"] Oct 30 20:53:49.783168 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:53:49.783132 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 20:53:54.761854 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:53:54.761814 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 20:53:59.405889 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:53:59.405857 5378 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c0c05792-jrcz7"] Oct 30 20:53:59.406390 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:53:59.406350 5378 cpu_manager.go:414] "containerMap: removing container" podUID="563c5c7c-a42c-417c-b883-98efbbef92ca" containerName="revoke-keys" Oct 30 20:53:59.406390 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:53:59.406363 5378 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="563c5c7c-a42c-417c-b883-98efbbef92ca" containerName="revoke-keys" Oct 30 20:53:59.406493 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:53:59.406462 5378 memory_manager.go:365] "RemoveStaleState removing state" podUID="563c5c7c-a42c-417c-b883-98efbbef92ca" containerName="revoke-keys" Oct 30 20:53:59.411602 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:53:59.410167 5378 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c0c05792-jrcz7" Oct 30 20:53:59.465865 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:53:59.465827 5378 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c0c05792-jrcz7"] Oct 30 20:53:59.510749 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:53:59.510721 5378 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a5d5a88d-6977-4900-948e-7171ce1cb445-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c0c05792-jrcz7\" (UID: \"a5d5a88d-6977-4900-948e-7171ce1cb445\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c0c05792-jrcz7" Oct 30 20:53:59.510916 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:53:59.510875 5378 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zjp5w\" (UniqueName: \"kubernetes.io/projected/a5d5a88d-6977-4900-948e-7171ce1cb445-kube-api-access-zjp5w\") pod \"maas-api-revoke-sub-c0c05792-jrcz7\" (UID: \"a5d5a88d-6977-4900-948e-7171ce1cb445\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c0c05792-jrcz7" Oct 30 20:53:59.510969 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:53:59.510933 5378 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a5d5a88d-6977-4900-948e-7171ce1cb445-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c0c05792-jrcz7\" (UID: \"a5d5a88d-6977-4900-948e-7171ce1cb445\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c0c05792-jrcz7" Oct 30 20:53:59.513916 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:53:59.513893 5378 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 20:53:59.514043 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:53:59.513963 5378 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-q2mgc\"" Oct 30 20:53:59.579714 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:53:59.579689 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 20:53:59.612105 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:53:59.612082 5378 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zjp5w\" (UniqueName: \"kubernetes.io/projected/a5d5a88d-6977-4900-948e-7171ce1cb445-kube-api-access-zjp5w\") pod \"maas-api-revoke-sub-c0c05792-jrcz7\" (UID: \"a5d5a88d-6977-4900-948e-7171ce1cb445\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c0c05792-jrcz7" Oct 30 20:53:59.612200 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:53:59.612120 5378 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a5d5a88d-6977-4900-948e-7171ce1cb445-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c0c05792-jrcz7\" (UID: \"a5d5a88d-6977-4900-948e-7171ce1cb445\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c0c05792-jrcz7" Oct 30 20:53:59.612241 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:53:59.612207 5378 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a5d5a88d-6977-4900-948e-7171ce1cb445-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c0c05792-jrcz7\" (UID: \"a5d5a88d-6977-4900-948e-7171ce1cb445\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c0c05792-jrcz7" Oct 30 20:53:59.612943 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:53:59.612917 5378 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a5d5a88d-6977-4900-948e-7171ce1cb445-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c0c05792-jrcz7\" (UID: \"a5d5a88d-6977-4900-948e-7171ce1cb445\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c0c05792-jrcz7" Oct 30 20:53:59.614585 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:53:59.614553 5378 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a5d5a88d-6977-4900-948e-7171ce1cb445-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c0c05792-jrcz7\" (UID: \"a5d5a88d-6977-4900-948e-7171ce1cb445\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c0c05792-jrcz7" Oct 30 20:53:59.670980 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:53:59.670897 5378 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zjp5w\" (UniqueName: \"kubernetes.io/projected/a5d5a88d-6977-4900-948e-7171ce1cb445-kube-api-access-zjp5w\") pod \"maas-api-revoke-sub-c0c05792-jrcz7\" (UID: \"a5d5a88d-6977-4900-948e-7171ce1cb445\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c0c05792-jrcz7" Oct 30 20:53:59.728236 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:53:59.728205 5378 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c0c05792-jrcz7" Oct 30 20:54:00.103684 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:54:00.103660 5378 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c0c05792-jrcz7"] Oct 30 20:54:00.106260 ip-10-0-138-149 kubenswrapper[5378]: W1030 20:54:00.106229 5378 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda5d5a88d_6977_4900_948e_7171ce1cb445.slice/crio-408eda428a52cb377ef44a9179571d4dc58637da6735ba97b8ce35dc86511104 WatchSource:0}: Error finding container 408eda428a52cb377ef44a9179571d4dc58637da6735ba97b8ce35dc86511104: Status 404 returned error can't find the container with id 408eda428a52cb377ef44a9179571d4dc58637da6735ba97b8ce35dc86511104 Oct 30 20:54:00.885354 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:54:00.885317 5378 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c0c05792-jrcz7" event={"ID":"a5d5a88d-6977-4900-948e-7171ce1cb445","Type":"ContainerStarted","Data":"6560457e1da2e780652f5972cf84904e5f3358c9e801458326fd6e5b7ded1736"} Oct 30 20:54:00.885818 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:54:00.885391 5378 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c0c05792-jrcz7" event={"ID":"a5d5a88d-6977-4900-948e-7171ce1cb445","Type":"ContainerStarted","Data":"408eda428a52cb377ef44a9179571d4dc58637da6735ba97b8ce35dc86511104"} Oct 30 20:54:01.054771 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:54:01.054718 5378 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c0c05792-jrcz7" podStartSLOduration=2.054704923 podStartE2EDuration="2.054704923s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:53:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:54:01.002717641 +0000 UTC m=+931.430831564" watchObservedRunningTime="2026-10-30 20:54:01.054704923 +0000 UTC m=+931.482818842" Oct 30 20:54:01.892618 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:54:01.892584 5378 generic.go:356] "Generic (PLEG): container finished" podID="a5d5a88d-6977-4900-948e-7171ce1cb445" containerID="6560457e1da2e780652f5972cf84904e5f3358c9e801458326fd6e5b7ded1736" exitCode=0 Oct 30 20:54:01.893003 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:54:01.892634 5378 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c0c05792-jrcz7" event={"ID":"a5d5a88d-6977-4900-948e-7171ce1cb445","Type":"ContainerDied","Data":"6560457e1da2e780652f5972cf84904e5f3358c9e801458326fd6e5b7ded1736"} Oct 30 20:54:03.036500 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:54:03.036473 5378 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c0c05792-jrcz7" Oct 30 20:54:03.151803 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:54:03.151735 5378 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a5d5a88d-6977-4900-948e-7171ce1cb445-kube-api-access-zjp5w\" (UniqueName: \"kubernetes.io/projected/a5d5a88d-6977-4900-948e-7171ce1cb445-kube-api-access-zjp5w\") pod \"a5d5a88d-6977-4900-948e-7171ce1cb445\" (UID: \"a5d5a88d-6977-4900-948e-7171ce1cb445\") " Oct 30 20:54:03.151942 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:54:03.151877 5378 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a5d5a88d-6977-4900-948e-7171ce1cb445-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a5d5a88d-6977-4900-948e-7171ce1cb445-maas-api-openshift-ca\") pod \"a5d5a88d-6977-4900-948e-7171ce1cb445\" (UID: \"a5d5a88d-6977-4900-948e-7171ce1cb445\") " Oct 30 20:54:03.151942 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:54:03.151898 5378 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a5d5a88d-6977-4900-948e-7171ce1cb445-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a5d5a88d-6977-4900-948e-7171ce1cb445-maas-api-xks-ca\") pod \"a5d5a88d-6977-4900-948e-7171ce1cb445\" (UID: \"a5d5a88d-6977-4900-948e-7171ce1cb445\") " Oct 30 20:54:03.152233 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:54:03.152208 5378 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a5d5a88d-6977-4900-948e-7171ce1cb445-maas-api-openshift-ca" pod "a5d5a88d-6977-4900-948e-7171ce1cb445" (UID: "a5d5a88d-6977-4900-948e-7171ce1cb445"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:54:03.154041 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:54:03.153880 5378 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a5d5a88d-6977-4900-948e-7171ce1cb445-kube-api-access-zjp5w" pod "a5d5a88d-6977-4900-948e-7171ce1cb445" (UID: "a5d5a88d-6977-4900-948e-7171ce1cb445"). InnerVolumeSpecName "kube-api-access-zjp5w". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:54:03.154605 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:54:03.154488 5378 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a5d5a88d-6977-4900-948e-7171ce1cb445-maas-api-xks-ca" pod "a5d5a88d-6977-4900-948e-7171ce1cb445" (UID: "a5d5a88d-6977-4900-948e-7171ce1cb445"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 20:54:03.253136 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:54:03.253106 5378 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a5d5a88d-6977-4900-948e-7171ce1cb445-maas-api-openshift-ca\") on node \"ip-10-0-138-149.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:54:03.253136 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:54:03.253131 5378 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a5d5a88d-6977-4900-948e-7171ce1cb445-maas-api-xks-ca\") on node \"ip-10-0-138-149.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:54:03.253136 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:54:03.253141 5378 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zjp5w\" (UniqueName: \"kubernetes.io/projected/a5d5a88d-6977-4900-948e-7171ce1cb445-kube-api-access-zjp5w\") on node \"ip-10-0-138-149.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:54:03.904793 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:54:03.904764 5378 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c0c05792-jrcz7" Oct 30 20:54:03.904955 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:54:03.904795 5378 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c0c05792-jrcz7" event={"ID":"a5d5a88d-6977-4900-948e-7171ce1cb445","Type":"ContainerDied","Data":"408eda428a52cb377ef44a9179571d4dc58637da6735ba97b8ce35dc86511104"} Oct 30 20:54:03.904955 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:54:03.904844 5378 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="408eda428a52cb377ef44a9179571d4dc58637da6735ba97b8ce35dc86511104" Oct 30 20:54:04.468414 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:54:04.468380 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 20:54:09.281722 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:54:09.281690 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 20:54:14.190367 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:54:14.190330 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 20:54:18.837290 ip-10-0-138-149 kubenswrapper[5378]: E1030 20:54:18.837251 5378 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q6mx7" containerName="extractor" Oct 30 20:54:18.981682 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:54:18.981642 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 20:54:23.187277 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:54:23.187237 5378 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-r9g59"] Oct 30 20:54:23.865989 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:54:23.865939 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 20:54:28.753920 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:54:28.753886 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 20:54:33.669744 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:54:33.669708 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 20:54:38.455902 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:54:38.455872 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 20:54:39.776676 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:54:39.776645 5378 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-769df7846c-8jkpf"] Oct 30 20:54:39.777081 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:54:39.776818 5378 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-8jkpf" podUID="0d0a80d9-4ec9-43cb-8368-15e6b0cb7b22" containerName="istio-proxy" containerID="cri-o://683bd8651a4c13ca27c4d4126f9342bf1f02e18227d22e733dae96c651abc3d9" gracePeriod=30 Oct 30 20:54:41.170833 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:54:41.170803 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 20:54:43.345860 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:54:43.345819 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 20:54:44.194603 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:54:44.194545 5378 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0adcf026-8wzrg"] Oct 30 20:54:44.195088 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:54:44.195074 5378 cpu_manager.go:414] "containerMap: removing container" podUID="a5d5a88d-6977-4900-948e-7171ce1cb445" containerName="revoke-keys" Oct 30 20:54:44.195132 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:54:44.195089 5378 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a5d5a88d-6977-4900-948e-7171ce1cb445" containerName="revoke-keys" Oct 30 20:54:44.195208 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:54:44.195198 5378 memory_manager.go:365] "RemoveStaleState removing state" podUID="a5d5a88d-6977-4900-948e-7171ce1cb445" containerName="revoke-keys" Oct 30 20:54:44.198342 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:54:44.198327 5378 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0adcf026-8wzrg" Oct 30 20:54:44.227864 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:54:44.227841 5378 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7c00b187-6511-415b-bf45-1db52954347e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0adcf026-8wzrg\" (UID: \"7c00b187-6511-415b-bf45-1db52954347e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0adcf026-8wzrg" Oct 30 20:54:44.227998 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:54:44.227903 5378 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7c00b187-6511-415b-bf45-1db52954347e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0adcf026-8wzrg\" (UID: \"7c00b187-6511-415b-bf45-1db52954347e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0adcf026-8wzrg" Oct 30 20:54:44.227998 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:54:44.227980 5378 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bl26h\" (UniqueName: \"kubernetes.io/projected/7c00b187-6511-415b-bf45-1db52954347e-kube-api-access-bl26h\") pod \"maas-api-revoke-sub-0adcf026-8wzrg\" (UID: \"7c00b187-6511-415b-bf45-1db52954347e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0adcf026-8wzrg" Oct 30 20:54:44.254523 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:54:44.254496 5378 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 20:54:44.257043 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:54:44.257014 5378 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0adcf026-8wzrg"] Oct 30 20:54:44.304528 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:54:44.304497 5378 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-q2mgc\"" Oct 30 20:54:44.329334 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:54:44.329308 5378 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7c00b187-6511-415b-bf45-1db52954347e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0adcf026-8wzrg\" (UID: \"7c00b187-6511-415b-bf45-1db52954347e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0adcf026-8wzrg" Oct 30 20:54:44.329472 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:54:44.329343 5378 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bl26h\" (UniqueName: \"kubernetes.io/projected/7c00b187-6511-415b-bf45-1db52954347e-kube-api-access-bl26h\") pod \"maas-api-revoke-sub-0adcf026-8wzrg\" (UID: \"7c00b187-6511-415b-bf45-1db52954347e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0adcf026-8wzrg" Oct 30 20:54:44.329472 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:54:44.329454 5378 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7c00b187-6511-415b-bf45-1db52954347e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0adcf026-8wzrg\" (UID: \"7c00b187-6511-415b-bf45-1db52954347e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0adcf026-8wzrg" Oct 30 20:54:44.330208 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:54:44.330180 5378 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7c00b187-6511-415b-bf45-1db52954347e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0adcf026-8wzrg\" (UID: \"7c00b187-6511-415b-bf45-1db52954347e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0adcf026-8wzrg" Oct 30 20:54:44.331709 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:54:44.331686 5378 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7c00b187-6511-415b-bf45-1db52954347e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0adcf026-8wzrg\" (UID: \"7c00b187-6511-415b-bf45-1db52954347e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0adcf026-8wzrg" Oct 30 20:54:44.387060 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:54:44.387026 5378 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bl26h\" (UniqueName: \"kubernetes.io/projected/7c00b187-6511-415b-bf45-1db52954347e-kube-api-access-bl26h\") pod \"maas-api-revoke-sub-0adcf026-8wzrg\" (UID: \"7c00b187-6511-415b-bf45-1db52954347e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0adcf026-8wzrg" Oct 30 20:54:44.513516 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:54:44.513438 5378 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0adcf026-8wzrg" Oct 30 20:54:44.888486 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:54:44.888461 5378 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0adcf026-8wzrg"] Oct 30 20:54:44.889740 ip-10-0-138-149 kubenswrapper[5378]: W1030 20:54:44.889706 5378 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7c00b187_6511_415b_bf45_1db52954347e.slice/crio-f6a046ccfa775d72fde9be83c7e1b8378a93f835e250717070bbec5bff3a7a40 WatchSource:0}: Error finding container f6a046ccfa775d72fde9be83c7e1b8378a93f835e250717070bbec5bff3a7a40: Status 404 returned error can't find the container with id f6a046ccfa775d72fde9be83c7e1b8378a93f835e250717070bbec5bff3a7a40 Oct 30 20:54:45.030395 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:54:45.030369 5378 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-8jkpf" Oct 30 20:54:45.136412 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:54:45.136381 5378 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0d0a80d9-4ec9-43cb-8368-15e6b0cb7b22-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0d0a80d9-4ec9-43cb-8368-15e6b0cb7b22-workload-socket\") pod \"0d0a80d9-4ec9-43cb-8368-15e6b0cb7b22\" (UID: \"0d0a80d9-4ec9-43cb-8368-15e6b0cb7b22\") " Oct 30 20:54:45.136591 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:54:45.136486 5378 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/0d0a80d9-4ec9-43cb-8368-15e6b0cb7b22-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0d0a80d9-4ec9-43cb-8368-15e6b0cb7b22-istio-podinfo\") pod \"0d0a80d9-4ec9-43cb-8368-15e6b0cb7b22\" (UID: \"0d0a80d9-4ec9-43cb-8368-15e6b0cb7b22\") " Oct 30 20:54:45.136591 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:54:45.136506 5378 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0d0a80d9-4ec9-43cb-8368-15e6b0cb7b22-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0d0a80d9-4ec9-43cb-8368-15e6b0cb7b22-istio-data\") pod \"0d0a80d9-4ec9-43cb-8368-15e6b0cb7b22\" (UID: \"0d0a80d9-4ec9-43cb-8368-15e6b0cb7b22\") " Oct 30 20:54:45.136591 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:54:45.136543 5378 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0d0a80d9-4ec9-43cb-8368-15e6b0cb7b22-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0d0a80d9-4ec9-43cb-8368-15e6b0cb7b22-istio-envoy\") pod \"0d0a80d9-4ec9-43cb-8368-15e6b0cb7b22\" (UID: \"0d0a80d9-4ec9-43cb-8368-15e6b0cb7b22\") " Oct 30 20:54:45.136766 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:54:45.136604 5378 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0d0a80d9-4ec9-43cb-8368-15e6b0cb7b22-istio-token\" (UniqueName: \"kubernetes.io/projected/0d0a80d9-4ec9-43cb-8368-15e6b0cb7b22-istio-token\") pod \"0d0a80d9-4ec9-43cb-8368-15e6b0cb7b22\" (UID: \"0d0a80d9-4ec9-43cb-8368-15e6b0cb7b22\") " Oct 30 20:54:45.136766 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:54:45.136658 5378 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0d0a80d9-4ec9-43cb-8368-15e6b0cb7b22-workload-socket" pod "0d0a80d9-4ec9-43cb-8368-15e6b0cb7b22" (UID: "0d0a80d9-4ec9-43cb-8368-15e6b0cb7b22"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:54:45.136766 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:54:45.136669 5378 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0d0a80d9-4ec9-43cb-8368-15e6b0cb7b22-kube-api-access-2dnhg\" (UniqueName: \"kubernetes.io/projected/0d0a80d9-4ec9-43cb-8368-15e6b0cb7b22-kube-api-access-2dnhg\") pod \"0d0a80d9-4ec9-43cb-8368-15e6b0cb7b22\" (UID: \"0d0a80d9-4ec9-43cb-8368-15e6b0cb7b22\") " Oct 30 20:54:45.136766 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:54:45.136727 5378 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0d0a80d9-4ec9-43cb-8368-15e6b0cb7b22-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0d0a80d9-4ec9-43cb-8368-15e6b0cb7b22-credential-socket\") pod \"0d0a80d9-4ec9-43cb-8368-15e6b0cb7b22\" (UID: \"0d0a80d9-4ec9-43cb-8368-15e6b0cb7b22\") " Oct 30 20:54:45.136975 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:54:45.136792 5378 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0d0a80d9-4ec9-43cb-8368-15e6b0cb7b22-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0d0a80d9-4ec9-43cb-8368-15e6b0cb7b22-istiod-ca-cert\") pod \"0d0a80d9-4ec9-43cb-8368-15e6b0cb7b22\" (UID: \"0d0a80d9-4ec9-43cb-8368-15e6b0cb7b22\") " Oct 30 20:54:45.136975 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:54:45.136821 5378 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0d0a80d9-4ec9-43cb-8368-15e6b0cb7b22-istio-data" pod "0d0a80d9-4ec9-43cb-8368-15e6b0cb7b22" (UID: "0d0a80d9-4ec9-43cb-8368-15e6b0cb7b22"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:54:45.136975 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:54:45.136831 5378 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0d0a80d9-4ec9-43cb-8368-15e6b0cb7b22-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0d0a80d9-4ec9-43cb-8368-15e6b0cb7b22-workload-certs\") pod \"0d0a80d9-4ec9-43cb-8368-15e6b0cb7b22\" (UID: \"0d0a80d9-4ec9-43cb-8368-15e6b0cb7b22\") " Oct 30 20:54:45.137145 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:54:45.137031 5378 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0d0a80d9-4ec9-43cb-8368-15e6b0cb7b22-credential-socket" pod "0d0a80d9-4ec9-43cb-8368-15e6b0cb7b22" (UID: "0d0a80d9-4ec9-43cb-8368-15e6b0cb7b22"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:54:45.137400 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:54:45.137328 5378 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0d0a80d9-4ec9-43cb-8368-15e6b0cb7b22-credential-socket\") on node \"ip-10-0-138-149.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:54:45.137400 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:54:45.137352 5378 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0d0a80d9-4ec9-43cb-8368-15e6b0cb7b22-workload-socket\") on node \"ip-10-0-138-149.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:54:45.137400 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:54:45.137366 5378 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0d0a80d9-4ec9-43cb-8368-15e6b0cb7b22-istio-data\") on node \"ip-10-0-138-149.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:54:45.137684 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:54:45.137487 5378 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0d0a80d9-4ec9-43cb-8368-15e6b0cb7b22-istiod-ca-cert" pod "0d0a80d9-4ec9-43cb-8368-15e6b0cb7b22" (UID: "0d0a80d9-4ec9-43cb-8368-15e6b0cb7b22"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:54:45.137684 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:54:45.137550 5378 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0d0a80d9-4ec9-43cb-8368-15e6b0cb7b22-workload-certs" pod "0d0a80d9-4ec9-43cb-8368-15e6b0cb7b22" (UID: "0d0a80d9-4ec9-43cb-8368-15e6b0cb7b22"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:54:45.138960 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:54:45.138932 5378 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0d0a80d9-4ec9-43cb-8368-15e6b0cb7b22-istio-envoy" pod "0d0a80d9-4ec9-43cb-8368-15e6b0cb7b22" (UID: "0d0a80d9-4ec9-43cb-8368-15e6b0cb7b22"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:54:45.139068 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:54:45.138959 5378 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0d0a80d9-4ec9-43cb-8368-15e6b0cb7b22-kube-api-access-2dnhg" pod "0d0a80d9-4ec9-43cb-8368-15e6b0cb7b22" (UID: "0d0a80d9-4ec9-43cb-8368-15e6b0cb7b22"). InnerVolumeSpecName "kube-api-access-2dnhg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:54:45.139068 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:54:45.139009 5378 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0d0a80d9-4ec9-43cb-8368-15e6b0cb7b22-istio-token" pod "0d0a80d9-4ec9-43cb-8368-15e6b0cb7b22" (UID: "0d0a80d9-4ec9-43cb-8368-15e6b0cb7b22"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:54:45.139068 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:54:45.139053 5378 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/0d0a80d9-4ec9-43cb-8368-15e6b0cb7b22-istio-podinfo" pod "0d0a80d9-4ec9-43cb-8368-15e6b0cb7b22" (UID: "0d0a80d9-4ec9-43cb-8368-15e6b0cb7b22"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 20:54:45.180906 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:54:45.180877 5378 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0adcf026-8wzrg" event={"ID":"7c00b187-6511-415b-bf45-1db52954347e","Type":"ContainerStarted","Data":"378176a6c61d5325db1dab1d835d3f89e4ef00780dcd3f1227615023f9a6326e"} Oct 30 20:54:45.181025 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:54:45.180935 5378 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0adcf026-8wzrg" event={"ID":"7c00b187-6511-415b-bf45-1db52954347e","Type":"ContainerStarted","Data":"f6a046ccfa775d72fde9be83c7e1b8378a93f835e250717070bbec5bff3a7a40"} Oct 30 20:54:45.182276 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:54:45.182243 5378 generic.go:356] "Generic (PLEG): container finished" podID="0d0a80d9-4ec9-43cb-8368-15e6b0cb7b22" containerID="683bd8651a4c13ca27c4d4126f9342bf1f02e18227d22e733dae96c651abc3d9" exitCode=0 Oct 30 20:54:45.182382 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:54:45.182289 5378 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-8jkpf" Oct 30 20:54:45.182382 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:54:45.182331 5378 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-8jkpf" event={"ID":"0d0a80d9-4ec9-43cb-8368-15e6b0cb7b22","Type":"ContainerDied","Data":"683bd8651a4c13ca27c4d4126f9342bf1f02e18227d22e733dae96c651abc3d9"} Oct 30 20:54:45.182382 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:54:45.182381 5378 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-8jkpf" event={"ID":"0d0a80d9-4ec9-43cb-8368-15e6b0cb7b22","Type":"ContainerDied","Data":"0e076fc52f98d805515a39fcfcd0d341ef591022ddeddac40efe4cc3730a4c77"} Oct 30 20:54:45.182555 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:54:45.182403 5378 scope.go:122] "RemoveContainer" containerID="683bd8651a4c13ca27c4d4126f9342bf1f02e18227d22e733dae96c651abc3d9" Oct 30 20:54:45.191309 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:54:45.191290 5378 scope.go:122] "RemoveContainer" containerID="683bd8651a4c13ca27c4d4126f9342bf1f02e18227d22e733dae96c651abc3d9" Oct 30 20:54:45.191662 ip-10-0-138-149 kubenswrapper[5378]: E1030 20:54:45.191633 5378 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"683bd8651a4c13ca27c4d4126f9342bf1f02e18227d22e733dae96c651abc3d9\": container with ID starting with 683bd8651a4c13ca27c4d4126f9342bf1f02e18227d22e733dae96c651abc3d9 not found: ID does not exist" containerID="683bd8651a4c13ca27c4d4126f9342bf1f02e18227d22e733dae96c651abc3d9" Oct 30 20:54:45.191735 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:54:45.191664 5378 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"683bd8651a4c13ca27c4d4126f9342bf1f02e18227d22e733dae96c651abc3d9"} err="failed to get container status \"683bd8651a4c13ca27c4d4126f9342bf1f02e18227d22e733dae96c651abc3d9\": rpc error: code = NotFound desc = could not find container \"683bd8651a4c13ca27c4d4126f9342bf1f02e18227d22e733dae96c651abc3d9\": container with ID starting with 683bd8651a4c13ca27c4d4126f9342bf1f02e18227d22e733dae96c651abc3d9 not found: ID does not exist" Oct 30 20:54:45.238263 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:54:45.238239 5378 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0d0a80d9-4ec9-43cb-8368-15e6b0cb7b22-istio-podinfo\") on node \"ip-10-0-138-149.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:54:45.238387 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:54:45.238266 5378 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0d0a80d9-4ec9-43cb-8368-15e6b0cb7b22-istio-envoy\") on node \"ip-10-0-138-149.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:54:45.238387 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:54:45.238280 5378 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0d0a80d9-4ec9-43cb-8368-15e6b0cb7b22-istio-token\") on node \"ip-10-0-138-149.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:54:45.238387 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:54:45.238292 5378 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2dnhg\" (UniqueName: \"kubernetes.io/projected/0d0a80d9-4ec9-43cb-8368-15e6b0cb7b22-kube-api-access-2dnhg\") on node \"ip-10-0-138-149.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:54:45.238387 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:54:45.238312 5378 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0d0a80d9-4ec9-43cb-8368-15e6b0cb7b22-istiod-ca-cert\") on node \"ip-10-0-138-149.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:54:45.238387 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:54:45.238326 5378 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0d0a80d9-4ec9-43cb-8368-15e6b0cb7b22-workload-certs\") on node \"ip-10-0-138-149.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:54:45.297084 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:54:45.297013 5378 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0adcf026-8wzrg" podStartSLOduration=1.296993518 podStartE2EDuration="1.296993518s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:54:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:54:45.296846728 +0000 UTC m=+975.724960660" watchObservedRunningTime="2026-10-30 20:54:45.296993518 +0000 UTC m=+975.725107438" Oct 30 20:54:45.417679 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:54:45.417647 5378 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-769df7846c-8jkpf"] Oct 30 20:54:45.424980 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:54:45.424942 5378 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-769df7846c-8jkpf"] Oct 30 20:54:45.740684 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:54:45.740655 5378 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0d0a80d9-4ec9-43cb-8368-15e6b0cb7b22" path="/var/lib/kubelet/pods/0d0a80d9-4ec9-43cb-8368-15e6b0cb7b22/volumes" Oct 30 20:54:46.189892 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:54:46.189851 5378 generic.go:356] "Generic (PLEG): container finished" podID="7c00b187-6511-415b-bf45-1db52954347e" containerID="378176a6c61d5325db1dab1d835d3f89e4ef00780dcd3f1227615023f9a6326e" exitCode=0 Oct 30 20:54:46.189892 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:54:46.189890 5378 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0adcf026-8wzrg" event={"ID":"7c00b187-6511-415b-bf45-1db52954347e","Type":"ContainerDied","Data":"378176a6c61d5325db1dab1d835d3f89e4ef00780dcd3f1227615023f9a6326e"} Oct 30 20:54:47.339102 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:54:47.339080 5378 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0adcf026-8wzrg" Oct 30 20:54:47.356333 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:54:47.356309 5378 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7c00b187-6511-415b-bf45-1db52954347e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7c00b187-6511-415b-bf45-1db52954347e-maas-api-xks-ca\") pod \"7c00b187-6511-415b-bf45-1db52954347e\" (UID: \"7c00b187-6511-415b-bf45-1db52954347e\") " Oct 30 20:54:47.356430 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:54:47.356379 5378 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7c00b187-6511-415b-bf45-1db52954347e-kube-api-access-bl26h\" (UniqueName: \"kubernetes.io/projected/7c00b187-6511-415b-bf45-1db52954347e-kube-api-access-bl26h\") pod \"7c00b187-6511-415b-bf45-1db52954347e\" (UID: \"7c00b187-6511-415b-bf45-1db52954347e\") " Oct 30 20:54:47.356495 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:54:47.356472 5378 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7c00b187-6511-415b-bf45-1db52954347e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7c00b187-6511-415b-bf45-1db52954347e-maas-api-openshift-ca\") pod \"7c00b187-6511-415b-bf45-1db52954347e\" (UID: \"7c00b187-6511-415b-bf45-1db52954347e\") " Oct 30 20:54:47.356905 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:54:47.356876 5378 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7c00b187-6511-415b-bf45-1db52954347e-maas-api-openshift-ca" pod "7c00b187-6511-415b-bf45-1db52954347e" (UID: "7c00b187-6511-415b-bf45-1db52954347e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:54:47.358408 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:54:47.358383 5378 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7c00b187-6511-415b-bf45-1db52954347e-maas-api-xks-ca" pod "7c00b187-6511-415b-bf45-1db52954347e" (UID: "7c00b187-6511-415b-bf45-1db52954347e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 20:54:47.358507 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:54:47.358489 5378 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7c00b187-6511-415b-bf45-1db52954347e-kube-api-access-bl26h" pod "7c00b187-6511-415b-bf45-1db52954347e" (UID: "7c00b187-6511-415b-bf45-1db52954347e"). InnerVolumeSpecName "kube-api-access-bl26h". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:54:47.458523 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:54:47.458444 5378 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7c00b187-6511-415b-bf45-1db52954347e-maas-api-openshift-ca\") on node \"ip-10-0-138-149.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:54:47.458523 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:54:47.458471 5378 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7c00b187-6511-415b-bf45-1db52954347e-maas-api-xks-ca\") on node \"ip-10-0-138-149.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:54:47.458523 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:54:47.458481 5378 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bl26h\" (UniqueName: \"kubernetes.io/projected/7c00b187-6511-415b-bf45-1db52954347e-kube-api-access-bl26h\") on node \"ip-10-0-138-149.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:54:48.159135 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:54:48.159099 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 20:54:48.202464 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:54:48.202436 5378 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0adcf026-8wzrg" Oct 30 20:54:48.202647 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:54:48.202441 5378 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0adcf026-8wzrg" event={"ID":"7c00b187-6511-415b-bf45-1db52954347e","Type":"ContainerDied","Data":"f6a046ccfa775d72fde9be83c7e1b8378a93f835e250717070bbec5bff3a7a40"} Oct 30 20:54:48.202647 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:54:48.202582 5378 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f6a046ccfa775d72fde9be83c7e1b8378a93f835e250717070bbec5bff3a7a40" Oct 30 20:54:48.837007 ip-10-0-138-149 kubenswrapper[5378]: E1030 20:54:48.836970 5378 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q6mx7" containerName="extractor" Oct 30 20:54:53.017207 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:54:53.017175 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 20:54:57.895036 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:54:57.894994 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 20:55:00.659700 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:55:00.659663 5378 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3ed48889-mnnln"] Oct 30 20:55:00.660183 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:55:00.660167 5378 cpu_manager.go:414] "containerMap: removing container" podUID="0d0a80d9-4ec9-43cb-8368-15e6b0cb7b22" containerName="istio-proxy" Oct 30 20:55:00.660226 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:55:00.660185 5378 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0d0a80d9-4ec9-43cb-8368-15e6b0cb7b22" containerName="istio-proxy" Oct 30 20:55:00.660226 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:55:00.660195 5378 cpu_manager.go:414] "containerMap: removing container" podUID="7c00b187-6511-415b-bf45-1db52954347e" containerName="revoke-keys" Oct 30 20:55:00.660226 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:55:00.660201 5378 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7c00b187-6511-415b-bf45-1db52954347e" containerName="revoke-keys" Oct 30 20:55:00.660316 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:55:00.660304 5378 memory_manager.go:365] "RemoveStaleState removing state" podUID="0d0a80d9-4ec9-43cb-8368-15e6b0cb7b22" containerName="istio-proxy" Oct 30 20:55:00.660316 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:55:00.660311 5378 memory_manager.go:365] "RemoveStaleState removing state" podUID="7c00b187-6511-415b-bf45-1db52954347e" containerName="revoke-keys" Oct 30 20:55:00.664026 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:55:00.664006 5378 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3ed48889-mnnln" Oct 30 20:55:00.767774 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:55:00.767731 5378 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 20:55:00.767953 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:55:00.767877 5378 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-q2mgc\"" Oct 30 20:55:00.771329 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:55:00.771304 5378 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3ed48889-mnnln"] Oct 30 20:55:00.789086 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:55:00.789048 5378 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gvwsw\" (UniqueName: \"kubernetes.io/projected/fb6656bf-9b3b-4373-bb00-cf650d4f2bcb-kube-api-access-gvwsw\") pod \"maas-api-revoke-sub-3ed48889-mnnln\" (UID: \"fb6656bf-9b3b-4373-bb00-cf650d4f2bcb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3ed48889-mnnln" Oct 30 20:55:00.789243 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:55:00.789168 5378 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fb6656bf-9b3b-4373-bb00-cf650d4f2bcb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3ed48889-mnnln\" (UID: \"fb6656bf-9b3b-4373-bb00-cf650d4f2bcb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3ed48889-mnnln" Oct 30 20:55:00.789243 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:55:00.789190 5378 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fb6656bf-9b3b-4373-bb00-cf650d4f2bcb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3ed48889-mnnln\" (UID: \"fb6656bf-9b3b-4373-bb00-cf650d4f2bcb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3ed48889-mnnln" Oct 30 20:55:00.890769 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:55:00.890741 5378 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fb6656bf-9b3b-4373-bb00-cf650d4f2bcb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3ed48889-mnnln\" (UID: \"fb6656bf-9b3b-4373-bb00-cf650d4f2bcb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3ed48889-mnnln" Oct 30 20:55:00.890910 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:55:00.890775 5378 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fb6656bf-9b3b-4373-bb00-cf650d4f2bcb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3ed48889-mnnln\" (UID: \"fb6656bf-9b3b-4373-bb00-cf650d4f2bcb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3ed48889-mnnln" Oct 30 20:55:00.890910 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:55:00.890864 5378 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gvwsw\" (UniqueName: \"kubernetes.io/projected/fb6656bf-9b3b-4373-bb00-cf650d4f2bcb-kube-api-access-gvwsw\") pod \"maas-api-revoke-sub-3ed48889-mnnln\" (UID: \"fb6656bf-9b3b-4373-bb00-cf650d4f2bcb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3ed48889-mnnln" Oct 30 20:55:00.891393 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:55:00.891367 5378 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fb6656bf-9b3b-4373-bb00-cf650d4f2bcb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3ed48889-mnnln\" (UID: \"fb6656bf-9b3b-4373-bb00-cf650d4f2bcb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3ed48889-mnnln" Oct 30 20:55:00.893055 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:55:00.893036 5378 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fb6656bf-9b3b-4373-bb00-cf650d4f2bcb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3ed48889-mnnln\" (UID: \"fb6656bf-9b3b-4373-bb00-cf650d4f2bcb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3ed48889-mnnln" Oct 30 20:55:00.948763 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:55:00.948703 5378 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gvwsw\" (UniqueName: \"kubernetes.io/projected/fb6656bf-9b3b-4373-bb00-cf650d4f2bcb-kube-api-access-gvwsw\") pod \"maas-api-revoke-sub-3ed48889-mnnln\" (UID: \"fb6656bf-9b3b-4373-bb00-cf650d4f2bcb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3ed48889-mnnln" Oct 30 20:55:00.977376 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:55:00.977353 5378 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3ed48889-mnnln" Oct 30 20:55:01.350241 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:55:01.350211 5378 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3ed48889-mnnln"] Oct 30 20:55:01.351704 ip-10-0-138-149 kubenswrapper[5378]: W1030 20:55:01.351674 5378 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfb6656bf_9b3b_4373_bb00_cf650d4f2bcb.slice/crio-ec6657eb82d20f4c85483e29b2a46b2f57f2b96c951c0351f393f7e9890a377b WatchSource:0}: Error finding container ec6657eb82d20f4c85483e29b2a46b2f57f2b96c951c0351f393f7e9890a377b: Status 404 returned error can't find the container with id ec6657eb82d20f4c85483e29b2a46b2f57f2b96c951c0351f393f7e9890a377b Oct 30 20:55:02.278020 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:55:02.277982 5378 generic.go:356] "Generic (PLEG): container finished" podID="fb6656bf-9b3b-4373-bb00-cf650d4f2bcb" containerID="e0936b412ac0ad9f042a7be6dcdc2d2683f9173b65d7c870419ff04e7ca54bda" exitCode=0 Oct 30 20:55:02.278412 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:55:02.278034 5378 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3ed48889-mnnln" event={"ID":"fb6656bf-9b3b-4373-bb00-cf650d4f2bcb","Type":"ContainerDied","Data":"e0936b412ac0ad9f042a7be6dcdc2d2683f9173b65d7c870419ff04e7ca54bda"} Oct 30 20:55:02.278412 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:55:02.278079 5378 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3ed48889-mnnln" event={"ID":"fb6656bf-9b3b-4373-bb00-cf650d4f2bcb","Type":"ContainerStarted","Data":"ec6657eb82d20f4c85483e29b2a46b2f57f2b96c951c0351f393f7e9890a377b"} Oct 30 20:55:02.765722 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:55:02.765687 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 20:55:03.426136 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:55:03.426109 5378 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3ed48889-mnnln" Oct 30 20:55:03.517278 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:55:03.517246 5378 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fb6656bf-9b3b-4373-bb00-cf650d4f2bcb-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fb6656bf-9b3b-4373-bb00-cf650d4f2bcb-maas-api-openshift-ca\") pod \"fb6656bf-9b3b-4373-bb00-cf650d4f2bcb\" (UID: \"fb6656bf-9b3b-4373-bb00-cf650d4f2bcb\") " Oct 30 20:55:03.517418 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:55:03.517377 5378 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fb6656bf-9b3b-4373-bb00-cf650d4f2bcb-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fb6656bf-9b3b-4373-bb00-cf650d4f2bcb-maas-api-xks-ca\") pod \"fb6656bf-9b3b-4373-bb00-cf650d4f2bcb\" (UID: \"fb6656bf-9b3b-4373-bb00-cf650d4f2bcb\") " Oct 30 20:55:03.517477 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:55:03.517452 5378 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fb6656bf-9b3b-4373-bb00-cf650d4f2bcb-kube-api-access-gvwsw\" (UniqueName: \"kubernetes.io/projected/fb6656bf-9b3b-4373-bb00-cf650d4f2bcb-kube-api-access-gvwsw\") pod \"fb6656bf-9b3b-4373-bb00-cf650d4f2bcb\" (UID: \"fb6656bf-9b3b-4373-bb00-cf650d4f2bcb\") " Oct 30 20:55:03.517645 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:55:03.517620 5378 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fb6656bf-9b3b-4373-bb00-cf650d4f2bcb-maas-api-openshift-ca" pod "fb6656bf-9b3b-4373-bb00-cf650d4f2bcb" (UID: "fb6656bf-9b3b-4373-bb00-cf650d4f2bcb"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:55:03.517853 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:55:03.517827 5378 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fb6656bf-9b3b-4373-bb00-cf650d4f2bcb-maas-api-openshift-ca\") on node \"ip-10-0-138-149.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:55:03.519476 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:55:03.519444 5378 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fb6656bf-9b3b-4373-bb00-cf650d4f2bcb-maas-api-xks-ca" pod "fb6656bf-9b3b-4373-bb00-cf650d4f2bcb" (UID: "fb6656bf-9b3b-4373-bb00-cf650d4f2bcb"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 20:55:03.519612 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:55:03.519498 5378 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fb6656bf-9b3b-4373-bb00-cf650d4f2bcb-kube-api-access-gvwsw" pod "fb6656bf-9b3b-4373-bb00-cf650d4f2bcb" (UID: "fb6656bf-9b3b-4373-bb00-cf650d4f2bcb"). InnerVolumeSpecName "kube-api-access-gvwsw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:55:03.618928 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:55:03.618898 5378 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gvwsw\" (UniqueName: \"kubernetes.io/projected/fb6656bf-9b3b-4373-bb00-cf650d4f2bcb-kube-api-access-gvwsw\") on node \"ip-10-0-138-149.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:55:03.618928 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:55:03.618928 5378 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fb6656bf-9b3b-4373-bb00-cf650d4f2bcb-maas-api-xks-ca\") on node \"ip-10-0-138-149.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:55:04.289410 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:55:04.289383 5378 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3ed48889-mnnln" Oct 30 20:55:04.289557 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:55:04.289374 5378 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3ed48889-mnnln" event={"ID":"fb6656bf-9b3b-4373-bb00-cf650d4f2bcb","Type":"ContainerDied","Data":"ec6657eb82d20f4c85483e29b2a46b2f57f2b96c951c0351f393f7e9890a377b"} Oct 30 20:55:04.289557 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:55:04.289512 5378 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ec6657eb82d20f4c85483e29b2a46b2f57f2b96c951c0351f393f7e9890a377b" Oct 30 20:55:07.676938 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:55:07.676904 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 20:55:10.700987 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:55:10.700910 5378 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w0-781058/distinct-w0-781058-kserve-8bdf9bfcd-hzdfk"] Oct 30 20:55:10.701498 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:55:10.701481 5378 cpu_manager.go:414] "containerMap: removing container" podUID="fb6656bf-9b3b-4373-bb00-cf650d4f2bcb" containerName="revoke-keys" Oct 30 20:55:10.701548 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:55:10.701500 5378 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fb6656bf-9b3b-4373-bb00-cf650d4f2bcb" containerName="revoke-keys" Oct 30 20:55:10.701616 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:55:10.701605 5378 memory_manager.go:365] "RemoveStaleState removing state" podUID="fb6656bf-9b3b-4373-bb00-cf650d4f2bcb" containerName="revoke-keys" Oct 30 20:55:10.705129 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:55:10.705109 5378 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-781058/distinct-w0-781058-kserve-8bdf9bfcd-hzdfk" Oct 30 20:55:10.758811 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:55:10.758775 5378 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-781058/distinct-w0-781058-kserve-8bdf9bfcd-hzdfk"] Oct 30 20:55:10.778361 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:55:10.778331 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 20:55:10.787971 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:55:10.787942 5378 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-781058\"/\"openshift-service-ca.crt\"" Oct 30 20:55:10.788091 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:55:10.787949 5378 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-781058\"/\"kube-root-ca.crt\"" Oct 30 20:55:10.799940 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:55:10.799915 5378 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a9678fc2-1c56-4d45-9007-eee5049e0f48-tls-certs\") pod \"distinct-w0-781058-kserve-8bdf9bfcd-hzdfk\" (UID: \"a9678fc2-1c56-4d45-9007-eee5049e0f48\") " pod="e2e-models-e2e-worker-w0-781058/distinct-w0-781058-kserve-8bdf9bfcd-hzdfk" Oct 30 20:55:10.800065 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:55:10.799962 5378 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a9678fc2-1c56-4d45-9007-eee5049e0f48-home\") pod \"distinct-w0-781058-kserve-8bdf9bfcd-hzdfk\" (UID: \"a9678fc2-1c56-4d45-9007-eee5049e0f48\") " pod="e2e-models-e2e-worker-w0-781058/distinct-w0-781058-kserve-8bdf9bfcd-hzdfk" Oct 30 20:55:10.800065 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:55:10.800052 5378 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a9678fc2-1c56-4d45-9007-eee5049e0f48-model-cache\") pod \"distinct-w0-781058-kserve-8bdf9bfcd-hzdfk\" (UID: \"a9678fc2-1c56-4d45-9007-eee5049e0f48\") " pod="e2e-models-e2e-worker-w0-781058/distinct-w0-781058-kserve-8bdf9bfcd-hzdfk" Oct 30 20:55:10.800161 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:55:10.800130 5378 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a9678fc2-1c56-4d45-9007-eee5049e0f48-dshm\") pod \"distinct-w0-781058-kserve-8bdf9bfcd-hzdfk\" (UID: \"a9678fc2-1c56-4d45-9007-eee5049e0f48\") " pod="e2e-models-e2e-worker-w0-781058/distinct-w0-781058-kserve-8bdf9bfcd-hzdfk" Oct 30 20:55:10.800237 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:55:10.800212 5378 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6v9tk\" (UniqueName: \"kubernetes.io/projected/a9678fc2-1c56-4d45-9007-eee5049e0f48-kube-api-access-6v9tk\") pod \"distinct-w0-781058-kserve-8bdf9bfcd-hzdfk\" (UID: \"a9678fc2-1c56-4d45-9007-eee5049e0f48\") " pod="e2e-models-e2e-worker-w0-781058/distinct-w0-781058-kserve-8bdf9bfcd-hzdfk" Oct 30 20:55:10.800343 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:55:10.800318 5378 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a9678fc2-1c56-4d45-9007-eee5049e0f48-tmp-dir\") pod \"distinct-w0-781058-kserve-8bdf9bfcd-hzdfk\" (UID: \"a9678fc2-1c56-4d45-9007-eee5049e0f48\") " pod="e2e-models-e2e-worker-w0-781058/distinct-w0-781058-kserve-8bdf9bfcd-hzdfk" Oct 30 20:55:10.805833 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:55:10.805814 5378 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-781058\"/\"distinct-w0-781058-kserve-self-signed-certs\"" Oct 30 20:55:10.805921 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:55:10.805843 5378 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-781058\"/\"default-dockercfg-ws5l2\"" Oct 30 20:55:10.901061 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:55:10.901022 5378 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a9678fc2-1c56-4d45-9007-eee5049e0f48-tls-certs\") pod \"distinct-w0-781058-kserve-8bdf9bfcd-hzdfk\" (UID: \"a9678fc2-1c56-4d45-9007-eee5049e0f48\") " pod="e2e-models-e2e-worker-w0-781058/distinct-w0-781058-kserve-8bdf9bfcd-hzdfk" Oct 30 20:55:10.901061 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:55:10.901064 5378 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a9678fc2-1c56-4d45-9007-eee5049e0f48-home\") pod \"distinct-w0-781058-kserve-8bdf9bfcd-hzdfk\" (UID: \"a9678fc2-1c56-4d45-9007-eee5049e0f48\") " pod="e2e-models-e2e-worker-w0-781058/distinct-w0-781058-kserve-8bdf9bfcd-hzdfk" Oct 30 20:55:10.901302 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:55:10.901127 5378 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a9678fc2-1c56-4d45-9007-eee5049e0f48-model-cache\") pod \"distinct-w0-781058-kserve-8bdf9bfcd-hzdfk\" (UID: \"a9678fc2-1c56-4d45-9007-eee5049e0f48\") " pod="e2e-models-e2e-worker-w0-781058/distinct-w0-781058-kserve-8bdf9bfcd-hzdfk" Oct 30 20:55:10.901302 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:55:10.901161 5378 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a9678fc2-1c56-4d45-9007-eee5049e0f48-dshm\") pod \"distinct-w0-781058-kserve-8bdf9bfcd-hzdfk\" (UID: \"a9678fc2-1c56-4d45-9007-eee5049e0f48\") " pod="e2e-models-e2e-worker-w0-781058/distinct-w0-781058-kserve-8bdf9bfcd-hzdfk" Oct 30 20:55:10.901302 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:55:10.901235 5378 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6v9tk\" (UniqueName: \"kubernetes.io/projected/a9678fc2-1c56-4d45-9007-eee5049e0f48-kube-api-access-6v9tk\") pod \"distinct-w0-781058-kserve-8bdf9bfcd-hzdfk\" (UID: \"a9678fc2-1c56-4d45-9007-eee5049e0f48\") " pod="e2e-models-e2e-worker-w0-781058/distinct-w0-781058-kserve-8bdf9bfcd-hzdfk" Oct 30 20:55:10.901446 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:55:10.901310 5378 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a9678fc2-1c56-4d45-9007-eee5049e0f48-tmp-dir\") pod \"distinct-w0-781058-kserve-8bdf9bfcd-hzdfk\" (UID: \"a9678fc2-1c56-4d45-9007-eee5049e0f48\") " pod="e2e-models-e2e-worker-w0-781058/distinct-w0-781058-kserve-8bdf9bfcd-hzdfk" Oct 30 20:55:10.901823 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:55:10.901796 5378 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a9678fc2-1c56-4d45-9007-eee5049e0f48-model-cache\") pod \"distinct-w0-781058-kserve-8bdf9bfcd-hzdfk\" (UID: \"a9678fc2-1c56-4d45-9007-eee5049e0f48\") " pod="e2e-models-e2e-worker-w0-781058/distinct-w0-781058-kserve-8bdf9bfcd-hzdfk" Oct 30 20:55:10.901955 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:55:10.901838 5378 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a9678fc2-1c56-4d45-9007-eee5049e0f48-tmp-dir\") pod \"distinct-w0-781058-kserve-8bdf9bfcd-hzdfk\" (UID: \"a9678fc2-1c56-4d45-9007-eee5049e0f48\") " pod="e2e-models-e2e-worker-w0-781058/distinct-w0-781058-kserve-8bdf9bfcd-hzdfk" Oct 30 20:55:10.901955 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:55:10.901915 5378 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a9678fc2-1c56-4d45-9007-eee5049e0f48-home\") pod \"distinct-w0-781058-kserve-8bdf9bfcd-hzdfk\" (UID: \"a9678fc2-1c56-4d45-9007-eee5049e0f48\") " pod="e2e-models-e2e-worker-w0-781058/distinct-w0-781058-kserve-8bdf9bfcd-hzdfk" Oct 30 20:55:10.903641 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:55:10.903615 5378 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a9678fc2-1c56-4d45-9007-eee5049e0f48-dshm\") pod \"distinct-w0-781058-kserve-8bdf9bfcd-hzdfk\" (UID: \"a9678fc2-1c56-4d45-9007-eee5049e0f48\") " pod="e2e-models-e2e-worker-w0-781058/distinct-w0-781058-kserve-8bdf9bfcd-hzdfk" Oct 30 20:55:10.903949 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:55:10.903932 5378 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a9678fc2-1c56-4d45-9007-eee5049e0f48-tls-certs\") pod \"distinct-w0-781058-kserve-8bdf9bfcd-hzdfk\" (UID: \"a9678fc2-1c56-4d45-9007-eee5049e0f48\") " pod="e2e-models-e2e-worker-w0-781058/distinct-w0-781058-kserve-8bdf9bfcd-hzdfk" Oct 30 20:55:10.920687 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:55:10.920654 5378 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8d9f30fa-h8vv4"] Oct 30 20:55:10.925158 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:55:10.925138 5378 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8d9f30fa-h8vv4" Oct 30 20:55:10.960628 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:55:10.960551 5378 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6v9tk\" (UniqueName: \"kubernetes.io/projected/a9678fc2-1c56-4d45-9007-eee5049e0f48-kube-api-access-6v9tk\") pod \"distinct-w0-781058-kserve-8bdf9bfcd-hzdfk\" (UID: \"a9678fc2-1c56-4d45-9007-eee5049e0f48\") " pod="e2e-models-e2e-worker-w0-781058/distinct-w0-781058-kserve-8bdf9bfcd-hzdfk" Oct 30 20:55:10.981426 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:55:10.978611 5378 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 20:55:10.981426 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:55:10.979369 5378 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-q2mgc\"" Oct 30 20:55:10.981426 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:55:10.981110 5378 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8d9f30fa-h8vv4"] Oct 30 20:55:11.014284 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:55:11.008155 5378 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nqqb7\" (UniqueName: \"kubernetes.io/projected/f05d1764-4d6a-4cc9-915b-0b0ba752091f-kube-api-access-nqqb7\") pod \"maas-api-revoke-sub-8d9f30fa-h8vv4\" (UID: \"f05d1764-4d6a-4cc9-915b-0b0ba752091f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8d9f30fa-h8vv4" Oct 30 20:55:11.014284 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:55:11.008334 5378 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f05d1764-4d6a-4cc9-915b-0b0ba752091f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8d9f30fa-h8vv4\" (UID: \"f05d1764-4d6a-4cc9-915b-0b0ba752091f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8d9f30fa-h8vv4" Oct 30 20:55:11.014284 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:55:11.008381 5378 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f05d1764-4d6a-4cc9-915b-0b0ba752091f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8d9f30fa-h8vv4\" (UID: \"f05d1764-4d6a-4cc9-915b-0b0ba752091f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8d9f30fa-h8vv4" Oct 30 20:55:11.020191 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:55:11.020155 5378 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-781058/distinct-w0-781058-kserve-8bdf9bfcd-hzdfk" Oct 30 20:55:11.112425 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:55:11.111122 5378 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nqqb7\" (UniqueName: \"kubernetes.io/projected/f05d1764-4d6a-4cc9-915b-0b0ba752091f-kube-api-access-nqqb7\") pod \"maas-api-revoke-sub-8d9f30fa-h8vv4\" (UID: \"f05d1764-4d6a-4cc9-915b-0b0ba752091f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8d9f30fa-h8vv4" Oct 30 20:55:11.112425 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:55:11.111288 5378 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f05d1764-4d6a-4cc9-915b-0b0ba752091f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8d9f30fa-h8vv4\" (UID: \"f05d1764-4d6a-4cc9-915b-0b0ba752091f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8d9f30fa-h8vv4" Oct 30 20:55:11.112425 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:55:11.111335 5378 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f05d1764-4d6a-4cc9-915b-0b0ba752091f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8d9f30fa-h8vv4\" (UID: \"f05d1764-4d6a-4cc9-915b-0b0ba752091f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8d9f30fa-h8vv4" Oct 30 20:55:11.112425 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:55:11.112149 5378 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f05d1764-4d6a-4cc9-915b-0b0ba752091f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8d9f30fa-h8vv4\" (UID: \"f05d1764-4d6a-4cc9-915b-0b0ba752091f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8d9f30fa-h8vv4" Oct 30 20:55:11.120298 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:55:11.120271 5378 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f05d1764-4d6a-4cc9-915b-0b0ba752091f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8d9f30fa-h8vv4\" (UID: \"f05d1764-4d6a-4cc9-915b-0b0ba752091f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8d9f30fa-h8vv4" Oct 30 20:55:11.171376 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:55:11.171346 5378 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nqqb7\" (UniqueName: \"kubernetes.io/projected/f05d1764-4d6a-4cc9-915b-0b0ba752091f-kube-api-access-nqqb7\") pod \"maas-api-revoke-sub-8d9f30fa-h8vv4\" (UID: \"f05d1764-4d6a-4cc9-915b-0b0ba752091f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8d9f30fa-h8vv4" Oct 30 20:55:11.239271 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:55:11.239209 5378 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8d9f30fa-h8vv4" Oct 30 20:55:11.430155 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:55:11.430131 5378 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-781058/distinct-w0-781058-kserve-8bdf9bfcd-hzdfk"] Oct 30 20:55:11.431101 ip-10-0-138-149 kubenswrapper[5378]: W1030 20:55:11.431067 5378 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poda9678fc2_1c56_4d45_9007_eee5049e0f48.slice/crio-36d0e4149bbdc27aae8da13c2bbbb753bfbf44b9a837844b6cdee806f3309ad4 WatchSource:0}: Error finding container 36d0e4149bbdc27aae8da13c2bbbb753bfbf44b9a837844b6cdee806f3309ad4: Status 404 returned error can't find the container with id 36d0e4149bbdc27aae8da13c2bbbb753bfbf44b9a837844b6cdee806f3309ad4 Oct 30 20:55:11.616601 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:55:11.612892 5378 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8d9f30fa-h8vv4"] Oct 30 20:55:12.350277 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:55:12.350232 5378 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8d9f30fa-h8vv4" event={"ID":"f05d1764-4d6a-4cc9-915b-0b0ba752091f","Type":"ContainerStarted","Data":"e6eccc026db7218dfcdb1eeb5a21b06fad3ff0ca690afc354045563983f6143a"} Oct 30 20:55:12.350739 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:55:12.350306 5378 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8d9f30fa-h8vv4" event={"ID":"f05d1764-4d6a-4cc9-915b-0b0ba752091f","Type":"ContainerStarted","Data":"e6c4e905b89b1cc747bf943b82cb60a18c1393736e77e84a91c76f96b3980523"} Oct 30 20:55:12.352235 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:55:12.352194 5378 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-781058/distinct-w0-781058-kserve-8bdf9bfcd-hzdfk" event={"ID":"a9678fc2-1c56-4d45-9007-eee5049e0f48","Type":"ContainerStarted","Data":"210dcceffce8b917b42e961a01e6a86c470e71b5535de0b9ccc82c420fc7ee88"} Oct 30 20:55:12.352364 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:55:12.352237 5378 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-781058/distinct-w0-781058-kserve-8bdf9bfcd-hzdfk" event={"ID":"a9678fc2-1c56-4d45-9007-eee5049e0f48","Type":"ContainerStarted","Data":"36d0e4149bbdc27aae8da13c2bbbb753bfbf44b9a837844b6cdee806f3309ad4"} Oct 30 20:55:12.468115 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:55:12.468066 5378 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8d9f30fa-h8vv4" podStartSLOduration=2.468052366 podStartE2EDuration="2.468052366s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:55:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:55:12.465946463 +0000 UTC m=+1002.894060381" watchObservedRunningTime="2026-10-30 20:55:12.468052366 +0000 UTC m=+1002.896166284" Oct 30 20:55:12.571484 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:55:12.571450 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 20:55:12.582418 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:55:12.582369 5378 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w0-781058/distinct-w0-781058-kserve-8bdf9bfcd-hzdfk" podStartSLOduration=2.582355487 podStartE2EDuration="2.582355487s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:55:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:55:12.580332886 +0000 UTC m=+1003.008446805" watchObservedRunningTime="2026-10-30 20:55:12.582355487 +0000 UTC m=+1003.010469443" Oct 30 20:55:13.009290 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:55:13.009254 5378 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0c6ca373-f2mh2"] Oct 30 20:55:13.013062 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:55:13.013044 5378 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0c6ca373-f2mh2" Oct 30 20:55:13.022123 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:55:13.022098 5378 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-781058/distinct-w0-781058-kserve-8bdf9bfcd-hzdfk" Oct 30 20:55:13.072283 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:55:13.072251 5378 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0c6ca373-f2mh2"] Oct 30 20:55:13.147619 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:55:13.147558 5378 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8abc6c54-d10b-4961-811d-f2b7df218958-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0c6ca373-f2mh2\" (UID: \"8abc6c54-d10b-4961-811d-f2b7df218958\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0c6ca373-f2mh2" Oct 30 20:55:13.147778 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:55:13.147633 5378 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6hwxj\" (UniqueName: \"kubernetes.io/projected/8abc6c54-d10b-4961-811d-f2b7df218958-kube-api-access-6hwxj\") pod \"maas-api-revoke-sub-0c6ca373-f2mh2\" (UID: \"8abc6c54-d10b-4961-811d-f2b7df218958\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0c6ca373-f2mh2" Oct 30 20:55:13.147778 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:55:13.147669 5378 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8abc6c54-d10b-4961-811d-f2b7df218958-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0c6ca373-f2mh2\" (UID: \"8abc6c54-d10b-4961-811d-f2b7df218958\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0c6ca373-f2mh2" Oct 30 20:55:13.248607 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:55:13.248552 5378 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8abc6c54-d10b-4961-811d-f2b7df218958-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0c6ca373-f2mh2\" (UID: \"8abc6c54-d10b-4961-811d-f2b7df218958\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0c6ca373-f2mh2" Oct 30 20:55:13.248761 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:55:13.248624 5378 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6hwxj\" (UniqueName: \"kubernetes.io/projected/8abc6c54-d10b-4961-811d-f2b7df218958-kube-api-access-6hwxj\") pod \"maas-api-revoke-sub-0c6ca373-f2mh2\" (UID: \"8abc6c54-d10b-4961-811d-f2b7df218958\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0c6ca373-f2mh2" Oct 30 20:55:13.248761 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:55:13.248677 5378 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8abc6c54-d10b-4961-811d-f2b7df218958-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0c6ca373-f2mh2\" (UID: \"8abc6c54-d10b-4961-811d-f2b7df218958\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0c6ca373-f2mh2" Oct 30 20:55:13.249334 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:55:13.249313 5378 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8abc6c54-d10b-4961-811d-f2b7df218958-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0c6ca373-f2mh2\" (UID: \"8abc6c54-d10b-4961-811d-f2b7df218958\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0c6ca373-f2mh2" Oct 30 20:55:13.250998 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:55:13.250974 5378 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8abc6c54-d10b-4961-811d-f2b7df218958-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0c6ca373-f2mh2\" (UID: \"8abc6c54-d10b-4961-811d-f2b7df218958\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0c6ca373-f2mh2" Oct 30 20:55:13.306390 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:55:13.306329 5378 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6hwxj\" (UniqueName: \"kubernetes.io/projected/8abc6c54-d10b-4961-811d-f2b7df218958-kube-api-access-6hwxj\") pod \"maas-api-revoke-sub-0c6ca373-f2mh2\" (UID: \"8abc6c54-d10b-4961-811d-f2b7df218958\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0c6ca373-f2mh2" Oct 30 20:55:13.326192 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:55:13.326166 5378 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0c6ca373-f2mh2" Oct 30 20:55:13.359738 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:55:13.359705 5378 generic.go:356] "Generic (PLEG): container finished" podID="f05d1764-4d6a-4cc9-915b-0b0ba752091f" containerID="e6eccc026db7218dfcdb1eeb5a21b06fad3ff0ca690afc354045563983f6143a" exitCode=0 Oct 30 20:55:13.360126 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:55:13.359786 5378 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8d9f30fa-h8vv4" event={"ID":"f05d1764-4d6a-4cc9-915b-0b0ba752091f","Type":"ContainerDied","Data":"e6eccc026db7218dfcdb1eeb5a21b06fad3ff0ca690afc354045563983f6143a"} Oct 30 20:55:13.702177 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:55:13.702146 5378 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0c6ca373-f2mh2"] Oct 30 20:55:13.703509 ip-10-0-138-149 kubenswrapper[5378]: W1030 20:55:13.703484 5378 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8abc6c54_d10b_4961_811d_f2b7df218958.slice/crio-246a879232e41a11ba173bb494a16559c290513266bf5b61cb18c4c615cb59ee WatchSource:0}: Error finding container 246a879232e41a11ba173bb494a16559c290513266bf5b61cb18c4c615cb59ee: Status 404 returned error can't find the container with id 246a879232e41a11ba173bb494a16559c290513266bf5b61cb18c4c615cb59ee Oct 30 20:55:14.232270 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:55:14.232186 5378 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-r9g59"] Oct 30 20:55:14.380047 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:55:14.380006 5378 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0c6ca373-f2mh2" event={"ID":"8abc6c54-d10b-4961-811d-f2b7df218958","Type":"ContainerStarted","Data":"e338ba4b17595477700ccc4e9ab9f44cf8bb0b0caeb9770fbb0916196d337360"} Oct 30 20:55:14.380477 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:55:14.380101 5378 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0c6ca373-f2mh2" event={"ID":"8abc6c54-d10b-4961-811d-f2b7df218958","Type":"ContainerStarted","Data":"246a879232e41a11ba173bb494a16559c290513266bf5b61cb18c4c615cb59ee"} Oct 30 20:55:14.567191 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:55:14.567134 5378 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0c6ca373-f2mh2" podStartSLOduration=2.5671187680000003 podStartE2EDuration="2.567118768s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:55:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:55:14.521174424 +0000 UTC m=+1004.949288354" watchObservedRunningTime="2026-10-30 20:55:14.567118768 +0000 UTC m=+1004.995232686" Oct 30 20:55:14.568253 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:55:14.568221 5378 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-682999-openshift-default-54dcb797b5-xhh97"] Oct 30 20:55:14.571866 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:55:14.571846 5378 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-682999-openshift-default-54dcb797b5-xhh97" Oct 30 20:55:14.620674 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:55:14.620647 5378 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8d9f30fa-h8vv4" Oct 30 20:55:14.633207 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:55:14.633177 5378 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-682999-openshift-default-54dcb797b5-xhh97"] Oct 30 20:55:14.665361 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:55:14.665318 5378 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f05d1764-4d6a-4cc9-915b-0b0ba752091f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f05d1764-4d6a-4cc9-915b-0b0ba752091f-maas-api-openshift-ca\") pod \"f05d1764-4d6a-4cc9-915b-0b0ba752091f\" (UID: \"f05d1764-4d6a-4cc9-915b-0b0ba752091f\") " Oct 30 20:55:14.665544 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:55:14.665445 5378 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f05d1764-4d6a-4cc9-915b-0b0ba752091f-kube-api-access-nqqb7\" (UniqueName: \"kubernetes.io/projected/f05d1764-4d6a-4cc9-915b-0b0ba752091f-kube-api-access-nqqb7\") pod \"f05d1764-4d6a-4cc9-915b-0b0ba752091f\" (UID: \"f05d1764-4d6a-4cc9-915b-0b0ba752091f\") " Oct 30 20:55:14.665544 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:55:14.665491 5378 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f05d1764-4d6a-4cc9-915b-0b0ba752091f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f05d1764-4d6a-4cc9-915b-0b0ba752091f-maas-api-xks-ca\") pod \"f05d1764-4d6a-4cc9-915b-0b0ba752091f\" (UID: \"f05d1764-4d6a-4cc9-915b-0b0ba752091f\") " Oct 30 20:55:14.665713 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:55:14.665690 5378 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/95e52f08-4390-4924-89e1-c520ced3743f-workload-socket\") pod \"e2e-worker-w3-682999-openshift-default-54dcb797b5-xhh97\" (UID: \"95e52f08-4390-4924-89e1-c520ced3743f\") " pod="openshift-ingress/e2e-worker-w3-682999-openshift-default-54dcb797b5-xhh97" Oct 30 20:55:14.665806 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:55:14.665791 5378 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/95e52f08-4390-4924-89e1-c520ced3743f-workload-certs\") pod \"e2e-worker-w3-682999-openshift-default-54dcb797b5-xhh97\" (UID: \"95e52f08-4390-4924-89e1-c520ced3743f\") " pod="openshift-ingress/e2e-worker-w3-682999-openshift-default-54dcb797b5-xhh97" Oct 30 20:55:14.665883 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:55:14.665823 5378 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/95e52f08-4390-4924-89e1-c520ced3743f-istio-podinfo\") pod \"e2e-worker-w3-682999-openshift-default-54dcb797b5-xhh97\" (UID: \"95e52f08-4390-4924-89e1-c520ced3743f\") " pod="openshift-ingress/e2e-worker-w3-682999-openshift-default-54dcb797b5-xhh97" Oct 30 20:55:14.665883 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:55:14.665846 5378 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/95e52f08-4390-4924-89e1-c520ced3743f-istio-token\") pod \"e2e-worker-w3-682999-openshift-default-54dcb797b5-xhh97\" (UID: \"95e52f08-4390-4924-89e1-c520ced3743f\") " pod="openshift-ingress/e2e-worker-w3-682999-openshift-default-54dcb797b5-xhh97" Oct 30 20:55:14.666021 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:55:14.665880 5378 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dkfsp\" (UniqueName: \"kubernetes.io/projected/95e52f08-4390-4924-89e1-c520ced3743f-kube-api-access-dkfsp\") pod \"e2e-worker-w3-682999-openshift-default-54dcb797b5-xhh97\" (UID: \"95e52f08-4390-4924-89e1-c520ced3743f\") " pod="openshift-ingress/e2e-worker-w3-682999-openshift-default-54dcb797b5-xhh97" Oct 30 20:55:14.666021 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:55:14.665947 5378 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/95e52f08-4390-4924-89e1-c520ced3743f-istiod-ca-cert\") pod \"e2e-worker-w3-682999-openshift-default-54dcb797b5-xhh97\" (UID: \"95e52f08-4390-4924-89e1-c520ced3743f\") " pod="openshift-ingress/e2e-worker-w3-682999-openshift-default-54dcb797b5-xhh97" Oct 30 20:55:14.666021 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:55:14.666007 5378 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/95e52f08-4390-4924-89e1-c520ced3743f-istio-data\") pod \"e2e-worker-w3-682999-openshift-default-54dcb797b5-xhh97\" (UID: \"95e52f08-4390-4924-89e1-c520ced3743f\") " pod="openshift-ingress/e2e-worker-w3-682999-openshift-default-54dcb797b5-xhh97" Oct 30 20:55:14.666223 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:55:14.666059 5378 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/95e52f08-4390-4924-89e1-c520ced3743f-credential-socket\") pod \"e2e-worker-w3-682999-openshift-default-54dcb797b5-xhh97\" (UID: \"95e52f08-4390-4924-89e1-c520ced3743f\") " pod="openshift-ingress/e2e-worker-w3-682999-openshift-default-54dcb797b5-xhh97" Oct 30 20:55:14.666223 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:55:14.666091 5378 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/95e52f08-4390-4924-89e1-c520ced3743f-istio-envoy\") pod \"e2e-worker-w3-682999-openshift-default-54dcb797b5-xhh97\" (UID: \"95e52f08-4390-4924-89e1-c520ced3743f\") " pod="openshift-ingress/e2e-worker-w3-682999-openshift-default-54dcb797b5-xhh97" Oct 30 20:55:14.680606 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:55:14.677552 5378 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f05d1764-4d6a-4cc9-915b-0b0ba752091f-maas-api-openshift-ca" pod "f05d1764-4d6a-4cc9-915b-0b0ba752091f" (UID: "f05d1764-4d6a-4cc9-915b-0b0ba752091f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:55:14.703720 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:55:14.693944 5378 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f05d1764-4d6a-4cc9-915b-0b0ba752091f-kube-api-access-nqqb7" pod "f05d1764-4d6a-4cc9-915b-0b0ba752091f" (UID: "f05d1764-4d6a-4cc9-915b-0b0ba752091f"). InnerVolumeSpecName "kube-api-access-nqqb7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:55:14.703720 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:55:14.697277 5378 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f05d1764-4d6a-4cc9-915b-0b0ba752091f-maas-api-xks-ca" pod "f05d1764-4d6a-4cc9-915b-0b0ba752091f" (UID: "f05d1764-4d6a-4cc9-915b-0b0ba752091f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 20:55:14.766487 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:55:14.766451 5378 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/95e52f08-4390-4924-89e1-c520ced3743f-istio-envoy\") pod \"e2e-worker-w3-682999-openshift-default-54dcb797b5-xhh97\" (UID: \"95e52f08-4390-4924-89e1-c520ced3743f\") " pod="openshift-ingress/e2e-worker-w3-682999-openshift-default-54dcb797b5-xhh97" Oct 30 20:55:14.766667 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:55:14.766502 5378 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/95e52f08-4390-4924-89e1-c520ced3743f-workload-socket\") pod \"e2e-worker-w3-682999-openshift-default-54dcb797b5-xhh97\" (UID: \"95e52f08-4390-4924-89e1-c520ced3743f\") " pod="openshift-ingress/e2e-worker-w3-682999-openshift-default-54dcb797b5-xhh97" Oct 30 20:55:14.766667 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:55:14.766596 5378 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/95e52f08-4390-4924-89e1-c520ced3743f-workload-certs\") pod \"e2e-worker-w3-682999-openshift-default-54dcb797b5-xhh97\" (UID: \"95e52f08-4390-4924-89e1-c520ced3743f\") " pod="openshift-ingress/e2e-worker-w3-682999-openshift-default-54dcb797b5-xhh97" Oct 30 20:55:14.766667 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:55:14.766642 5378 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/95e52f08-4390-4924-89e1-c520ced3743f-istio-podinfo\") pod \"e2e-worker-w3-682999-openshift-default-54dcb797b5-xhh97\" (UID: \"95e52f08-4390-4924-89e1-c520ced3743f\") " pod="openshift-ingress/e2e-worker-w3-682999-openshift-default-54dcb797b5-xhh97" Oct 30 20:55:14.766667 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:55:14.766664 5378 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/95e52f08-4390-4924-89e1-c520ced3743f-istio-token\") pod \"e2e-worker-w3-682999-openshift-default-54dcb797b5-xhh97\" (UID: \"95e52f08-4390-4924-89e1-c520ced3743f\") " pod="openshift-ingress/e2e-worker-w3-682999-openshift-default-54dcb797b5-xhh97" Oct 30 20:55:14.766916 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:55:14.766694 5378 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dkfsp\" (UniqueName: \"kubernetes.io/projected/95e52f08-4390-4924-89e1-c520ced3743f-kube-api-access-dkfsp\") pod \"e2e-worker-w3-682999-openshift-default-54dcb797b5-xhh97\" (UID: \"95e52f08-4390-4924-89e1-c520ced3743f\") " pod="openshift-ingress/e2e-worker-w3-682999-openshift-default-54dcb797b5-xhh97" Oct 30 20:55:14.766916 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:55:14.766776 5378 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/95e52f08-4390-4924-89e1-c520ced3743f-istiod-ca-cert\") pod \"e2e-worker-w3-682999-openshift-default-54dcb797b5-xhh97\" (UID: \"95e52f08-4390-4924-89e1-c520ced3743f\") " pod="openshift-ingress/e2e-worker-w3-682999-openshift-default-54dcb797b5-xhh97" Oct 30 20:55:14.767043 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:55:14.766997 5378 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/95e52f08-4390-4924-89e1-c520ced3743f-workload-socket\") pod \"e2e-worker-w3-682999-openshift-default-54dcb797b5-xhh97\" (UID: \"95e52f08-4390-4924-89e1-c520ced3743f\") " pod="openshift-ingress/e2e-worker-w3-682999-openshift-default-54dcb797b5-xhh97" Oct 30 20:55:14.767201 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:55:14.767164 5378 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/95e52f08-4390-4924-89e1-c520ced3743f-istio-data\") pod \"e2e-worker-w3-682999-openshift-default-54dcb797b5-xhh97\" (UID: \"95e52f08-4390-4924-89e1-c520ced3743f\") " pod="openshift-ingress/e2e-worker-w3-682999-openshift-default-54dcb797b5-xhh97" Oct 30 20:55:14.767292 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:55:14.767257 5378 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/95e52f08-4390-4924-89e1-c520ced3743f-credential-socket\") pod \"e2e-worker-w3-682999-openshift-default-54dcb797b5-xhh97\" (UID: \"95e52f08-4390-4924-89e1-c520ced3743f\") " pod="openshift-ingress/e2e-worker-w3-682999-openshift-default-54dcb797b5-xhh97" Oct 30 20:55:14.767344 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:55:14.767285 5378 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/95e52f08-4390-4924-89e1-c520ced3743f-workload-certs\") pod \"e2e-worker-w3-682999-openshift-default-54dcb797b5-xhh97\" (UID: \"95e52f08-4390-4924-89e1-c520ced3743f\") " pod="openshift-ingress/e2e-worker-w3-682999-openshift-default-54dcb797b5-xhh97" Oct 30 20:55:14.767488 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:55:14.767465 5378 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/95e52f08-4390-4924-89e1-c520ced3743f-istiod-ca-cert\") pod \"e2e-worker-w3-682999-openshift-default-54dcb797b5-xhh97\" (UID: \"95e52f08-4390-4924-89e1-c520ced3743f\") " pod="openshift-ingress/e2e-worker-w3-682999-openshift-default-54dcb797b5-xhh97" Oct 30 20:55:14.767626 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:55:14.767604 5378 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/95e52f08-4390-4924-89e1-c520ced3743f-credential-socket\") pod \"e2e-worker-w3-682999-openshift-default-54dcb797b5-xhh97\" (UID: \"95e52f08-4390-4924-89e1-c520ced3743f\") " pod="openshift-ingress/e2e-worker-w3-682999-openshift-default-54dcb797b5-xhh97" Oct 30 20:55:14.767720 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:55:14.767664 5378 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nqqb7\" (UniqueName: \"kubernetes.io/projected/f05d1764-4d6a-4cc9-915b-0b0ba752091f-kube-api-access-nqqb7\") on node \"ip-10-0-138-149.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:55:14.767720 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:55:14.767683 5378 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f05d1764-4d6a-4cc9-915b-0b0ba752091f-maas-api-xks-ca\") on node \"ip-10-0-138-149.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:55:14.767720 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:55:14.767699 5378 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f05d1764-4d6a-4cc9-915b-0b0ba752091f-maas-api-openshift-ca\") on node \"ip-10-0-138-149.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:55:14.767866 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:55:14.767771 5378 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/95e52f08-4390-4924-89e1-c520ced3743f-istio-data\") pod \"e2e-worker-w3-682999-openshift-default-54dcb797b5-xhh97\" (UID: \"95e52f08-4390-4924-89e1-c520ced3743f\") " pod="openshift-ingress/e2e-worker-w3-682999-openshift-default-54dcb797b5-xhh97" Oct 30 20:55:14.768992 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:55:14.768971 5378 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/95e52f08-4390-4924-89e1-c520ced3743f-istio-envoy\") pod \"e2e-worker-w3-682999-openshift-default-54dcb797b5-xhh97\" (UID: \"95e52f08-4390-4924-89e1-c520ced3743f\") " pod="openshift-ingress/e2e-worker-w3-682999-openshift-default-54dcb797b5-xhh97" Oct 30 20:55:14.769338 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:55:14.769318 5378 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/95e52f08-4390-4924-89e1-c520ced3743f-istio-podinfo\") pod \"e2e-worker-w3-682999-openshift-default-54dcb797b5-xhh97\" (UID: \"95e52f08-4390-4924-89e1-c520ced3743f\") " pod="openshift-ingress/e2e-worker-w3-682999-openshift-default-54dcb797b5-xhh97" Oct 30 20:55:14.862665 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:55:14.829858 5378 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/95e52f08-4390-4924-89e1-c520ced3743f-istio-token\") pod \"e2e-worker-w3-682999-openshift-default-54dcb797b5-xhh97\" (UID: \"95e52f08-4390-4924-89e1-c520ced3743f\") " pod="openshift-ingress/e2e-worker-w3-682999-openshift-default-54dcb797b5-xhh97" Oct 30 20:55:14.862665 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:55:14.830589 5378 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dkfsp\" (UniqueName: \"kubernetes.io/projected/95e52f08-4390-4924-89e1-c520ced3743f-kube-api-access-dkfsp\") pod \"e2e-worker-w3-682999-openshift-default-54dcb797b5-xhh97\" (UID: \"95e52f08-4390-4924-89e1-c520ced3743f\") " pod="openshift-ingress/e2e-worker-w3-682999-openshift-default-54dcb797b5-xhh97" Oct 30 20:55:14.931192 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:55:14.931152 5378 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-682999-openshift-default-54dcb797b5-xhh97" Oct 30 20:55:15.379531 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:55:15.379467 5378 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-682999-openshift-default-54dcb797b5-xhh97"] Oct 30 20:55:15.390769 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:55:15.390720 5378 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 20:55:15.391162 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:55:15.390838 5378 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 20:55:15.391162 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:55:15.390903 5378 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 20:55:15.392090 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:55:15.392060 5378 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8d9f30fa-h8vv4" event={"ID":"f05d1764-4d6a-4cc9-915b-0b0ba752091f","Type":"ContainerDied","Data":"e6c4e905b89b1cc747bf943b82cb60a18c1393736e77e84a91c76f96b3980523"} Oct 30 20:55:15.392188 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:55:15.392120 5378 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e6c4e905b89b1cc747bf943b82cb60a18c1393736e77e84a91c76f96b3980523" Oct 30 20:55:15.392246 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:55:15.392197 5378 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8d9f30fa-h8vv4" Oct 30 20:55:15.399185 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:55:15.399157 5378 generic.go:356] "Generic (PLEG): container finished" podID="8abc6c54-d10b-4961-811d-f2b7df218958" containerID="e338ba4b17595477700ccc4e9ab9f44cf8bb0b0caeb9770fbb0916196d337360" exitCode=0 Oct 30 20:55:15.399413 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:55:15.399377 5378 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0c6ca373-f2mh2" event={"ID":"8abc6c54-d10b-4961-811d-f2b7df218958","Type":"ContainerDied","Data":"e338ba4b17595477700ccc4e9ab9f44cf8bb0b0caeb9770fbb0916196d337360"} Oct 30 20:55:16.406374 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:55:16.406316 5378 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-682999-openshift-default-54dcb797b5-xhh97" event={"ID":"95e52f08-4390-4924-89e1-c520ced3743f","Type":"ContainerStarted","Data":"4820660cffc38c84c6a5bb8f793ea741daa1d57b1c65c2e8ae8dcc6211617a48"} Oct 30 20:55:16.406931 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:55:16.406394 5378 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-682999-openshift-default-54dcb797b5-xhh97" event={"ID":"95e52f08-4390-4924-89e1-c520ced3743f","Type":"ContainerStarted","Data":"44d7c576d9e5b823fe45952d3aef0333805f0a52f72dd7f8931bec2a5dd48500"} Oct 30 20:55:16.590804 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:55:16.590779 5378 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0c6ca373-f2mh2" Oct 30 20:55:16.592732 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:55:16.592710 5378 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8abc6c54-d10b-4961-811d-f2b7df218958-kube-api-access-6hwxj\" (UniqueName: \"kubernetes.io/projected/8abc6c54-d10b-4961-811d-f2b7df218958-kube-api-access-6hwxj\") pod \"8abc6c54-d10b-4961-811d-f2b7df218958\" (UID: \"8abc6c54-d10b-4961-811d-f2b7df218958\") " Oct 30 20:55:16.592901 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:55:16.592888 5378 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8abc6c54-d10b-4961-811d-f2b7df218958-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8abc6c54-d10b-4961-811d-f2b7df218958-maas-api-openshift-ca\") pod \"8abc6c54-d10b-4961-811d-f2b7df218958\" (UID: \"8abc6c54-d10b-4961-811d-f2b7df218958\") " Oct 30 20:55:16.592948 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:55:16.592936 5378 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8abc6c54-d10b-4961-811d-f2b7df218958-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8abc6c54-d10b-4961-811d-f2b7df218958-maas-api-xks-ca\") pod \"8abc6c54-d10b-4961-811d-f2b7df218958\" (UID: \"8abc6c54-d10b-4961-811d-f2b7df218958\") " Oct 30 20:55:16.593234 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:55:16.593209 5378 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8abc6c54-d10b-4961-811d-f2b7df218958-maas-api-openshift-ca" pod "8abc6c54-d10b-4961-811d-f2b7df218958" (UID: "8abc6c54-d10b-4961-811d-f2b7df218958"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:55:16.594793 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:55:16.594767 5378 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8abc6c54-d10b-4961-811d-f2b7df218958-kube-api-access-6hwxj" pod "8abc6c54-d10b-4961-811d-f2b7df218958" (UID: "8abc6c54-d10b-4961-811d-f2b7df218958"). InnerVolumeSpecName "kube-api-access-6hwxj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:55:16.594877 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:55:16.594858 5378 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8abc6c54-d10b-4961-811d-f2b7df218958-maas-api-xks-ca" pod "8abc6c54-d10b-4961-811d-f2b7df218958" (UID: "8abc6c54-d10b-4961-811d-f2b7df218958"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 20:55:16.632189 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:55:16.632142 5378 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-682999-openshift-default-54dcb797b5-xhh97" podStartSLOduration=2.632129214 podStartE2EDuration="2.632129214s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:55:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:55:16.579932663 +0000 UTC m=+1007.008046582" watchObservedRunningTime="2026-10-30 20:55:16.632129214 +0000 UTC m=+1007.060243132" Oct 30 20:55:16.693956 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:55:16.693865 5378 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6hwxj\" (UniqueName: \"kubernetes.io/projected/8abc6c54-d10b-4961-811d-f2b7df218958-kube-api-access-6hwxj\") on node \"ip-10-0-138-149.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:55:16.693956 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:55:16.693900 5378 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8abc6c54-d10b-4961-811d-f2b7df218958-maas-api-openshift-ca\") on node \"ip-10-0-138-149.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:55:16.693956 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:55:16.693910 5378 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8abc6c54-d10b-4961-811d-f2b7df218958-maas-api-xks-ca\") on node \"ip-10-0-138-149.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:55:16.931825 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:55:16.931784 5378 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-682999-openshift-default-54dcb797b5-xhh97" Oct 30 20:55:16.933336 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:55:16.933315 5378 patch_prober.go:29] interesting pod/e2e-worker-w3-682999-openshift-default-54dcb797b5-xhh97 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused" start-of-body= Oct 30 20:55:16.933399 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:55:16.933366 5378 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-682999-openshift-default-54dcb797b5-xhh97" podUID="95e52f08-4390-4924-89e1-c520ced3743f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused" Oct 30 20:55:17.415485 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:55:17.415463 5378 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0c6ca373-f2mh2" Oct 30 20:55:17.415895 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:55:17.415480 5378 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0c6ca373-f2mh2" event={"ID":"8abc6c54-d10b-4961-811d-f2b7df218958","Type":"ContainerDied","Data":"246a879232e41a11ba173bb494a16559c290513266bf5b61cb18c4c615cb59ee"} Oct 30 20:55:17.415895 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:55:17.415532 5378 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="246a879232e41a11ba173bb494a16559c290513266bf5b61cb18c4c615cb59ee" Oct 30 20:55:17.494885 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:55:17.494851 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 20:55:17.931595 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:55:17.931532 5378 patch_prober.go:29] interesting pod/e2e-worker-w3-682999-openshift-default-54dcb797b5-xhh97 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused" start-of-body= Oct 30 20:55:17.931789 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:55:17.931626 5378 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-682999-openshift-default-54dcb797b5-xhh97" podUID="95e52f08-4390-4924-89e1-c520ced3743f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused" Oct 30 20:55:18.837916 ip-10-0-138-149 kubenswrapper[5378]: E1030 20:55:18.837882 5378 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q6mx7" containerName="extractor" Oct 30 20:55:18.931791 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:55:18.931756 5378 patch_prober.go:29] interesting pod/e2e-worker-w3-682999-openshift-default-54dcb797b5-xhh97 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused" start-of-body= Oct 30 20:55:18.932016 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:55:18.931811 5378 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-682999-openshift-default-54dcb797b5-xhh97" podUID="95e52f08-4390-4924-89e1-c520ced3743f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused" Oct 30 20:55:19.932278 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:55:19.932242 5378 patch_prober.go:29] interesting pod/e2e-worker-w3-682999-openshift-default-54dcb797b5-xhh97 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused" start-of-body= Oct 30 20:55:19.932724 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:55:19.932299 5378 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-682999-openshift-default-54dcb797b5-xhh97" podUID="95e52f08-4390-4924-89e1-c520ced3743f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused" Oct 30 20:55:20.931515 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:55:20.931479 5378 patch_prober.go:29] interesting pod/e2e-worker-w3-682999-openshift-default-54dcb797b5-xhh97 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused" start-of-body= Oct 30 20:55:20.931727 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:55:20.931538 5378 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-682999-openshift-default-54dcb797b5-xhh97" podUID="95e52f08-4390-4924-89e1-c520ced3743f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused" Oct 30 20:55:21.021959 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:55:21.021925 5378 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w0-781058/distinct-w0-781058-kserve-8bdf9bfcd-hzdfk" Oct 30 20:55:21.034297 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:55:21.034275 5378 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w0-781058/distinct-w0-781058-kserve-8bdf9bfcd-hzdfk" Oct 30 20:55:21.455918 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:55:21.455883 5378 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-781058/distinct-w0-781058-kserve-8bdf9bfcd-hzdfk" Oct 30 20:55:21.932010 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:55:21.931969 5378 patch_prober.go:29] interesting pod/e2e-worker-w3-682999-openshift-default-54dcb797b5-xhh97 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused" start-of-body= Oct 30 20:55:21.932173 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:55:21.932037 5378 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-682999-openshift-default-54dcb797b5-xhh97" podUID="95e52f08-4390-4924-89e1-c520ced3743f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused" Oct 30 20:55:22.357050 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:55:22.357013 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 20:55:22.936376 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:55:22.936346 5378 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-682999-openshift-default-54dcb797b5-xhh97" Oct 30 20:55:22.936855 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:55:22.936831 5378 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-682999-openshift-default-54dcb797b5-xhh97" Oct 30 20:55:22.937434 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:55:22.937419 5378 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-682999-openshift-default-54dcb797b5-xhh97" Oct 30 20:55:25.987979 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:55:25.987945 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 20:55:26.134164 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:55:26.134129 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 20:55:27.158423 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:55:27.158386 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 20:55:29.050396 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:55:29.050355 5378 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f3113105-xn7nh"] Oct 30 20:55:29.051094 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:55:29.051071 5378 cpu_manager.go:414] "containerMap: removing container" podUID="f05d1764-4d6a-4cc9-915b-0b0ba752091f" containerName="revoke-keys" Oct 30 20:55:29.051199 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:55:29.051094 5378 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f05d1764-4d6a-4cc9-915b-0b0ba752091f" containerName="revoke-keys" Oct 30 20:55:29.051199 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:55:29.051111 5378 cpu_manager.go:414] "containerMap: removing container" podUID="8abc6c54-d10b-4961-811d-f2b7df218958" containerName="revoke-keys" Oct 30 20:55:29.051199 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:55:29.051118 5378 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8abc6c54-d10b-4961-811d-f2b7df218958" containerName="revoke-keys" Oct 30 20:55:29.051352 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:55:29.051249 5378 memory_manager.go:365] "RemoveStaleState removing state" podUID="f05d1764-4d6a-4cc9-915b-0b0ba752091f" containerName="revoke-keys" Oct 30 20:55:29.051352 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:55:29.051263 5378 memory_manager.go:365] "RemoveStaleState removing state" podUID="8abc6c54-d10b-4961-811d-f2b7df218958" containerName="revoke-keys" Oct 30 20:55:29.054753 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:55:29.054730 5378 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f3113105-xn7nh" Oct 30 20:55:29.111224 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:55:29.111191 5378 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f3113105-xn7nh"] Oct 30 20:55:29.119402 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:55:29.119372 5378 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b831b46a-433f-43a6-837f-dd353948593a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f3113105-xn7nh\" (UID: \"b831b46a-433f-43a6-837f-dd353948593a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f3113105-xn7nh" Oct 30 20:55:29.119555 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:55:29.119531 5378 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b831b46a-433f-43a6-837f-dd353948593a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f3113105-xn7nh\" (UID: \"b831b46a-433f-43a6-837f-dd353948593a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f3113105-xn7nh" Oct 30 20:55:29.119658 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:55:29.119623 5378 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-974rv\" (UniqueName: \"kubernetes.io/projected/b831b46a-433f-43a6-837f-dd353948593a-kube-api-access-974rv\") pod \"maas-api-revoke-sub-f3113105-xn7nh\" (UID: \"b831b46a-433f-43a6-837f-dd353948593a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f3113105-xn7nh" Oct 30 20:55:29.158260 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:55:29.158231 5378 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 20:55:29.158378 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:55:29.158316 5378 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-q2mgc\"" Oct 30 20:55:29.221120 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:55:29.221091 5378 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b831b46a-433f-43a6-837f-dd353948593a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f3113105-xn7nh\" (UID: \"b831b46a-433f-43a6-837f-dd353948593a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f3113105-xn7nh" Oct 30 20:55:29.221267 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:55:29.221128 5378 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-974rv\" (UniqueName: \"kubernetes.io/projected/b831b46a-433f-43a6-837f-dd353948593a-kube-api-access-974rv\") pod \"maas-api-revoke-sub-f3113105-xn7nh\" (UID: \"b831b46a-433f-43a6-837f-dd353948593a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f3113105-xn7nh" Oct 30 20:55:29.221320 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:55:29.221305 5378 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b831b46a-433f-43a6-837f-dd353948593a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f3113105-xn7nh\" (UID: \"b831b46a-433f-43a6-837f-dd353948593a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f3113105-xn7nh" Oct 30 20:55:29.221946 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:55:29.221923 5378 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b831b46a-433f-43a6-837f-dd353948593a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f3113105-xn7nh\" (UID: \"b831b46a-433f-43a6-837f-dd353948593a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f3113105-xn7nh" Oct 30 20:55:29.223381 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:55:29.223361 5378 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b831b46a-433f-43a6-837f-dd353948593a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f3113105-xn7nh\" (UID: \"b831b46a-433f-43a6-837f-dd353948593a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f3113105-xn7nh" Oct 30 20:55:29.278969 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:55:29.278941 5378 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-974rv\" (UniqueName: \"kubernetes.io/projected/b831b46a-433f-43a6-837f-dd353948593a-kube-api-access-974rv\") pod \"maas-api-revoke-sub-f3113105-xn7nh\" (UID: \"b831b46a-433f-43a6-837f-dd353948593a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f3113105-xn7nh" Oct 30 20:55:29.370016 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:55:29.369986 5378 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f3113105-xn7nh" Oct 30 20:55:29.742325 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:55:29.742287 5378 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f3113105-xn7nh"] Oct 30 20:55:29.742450 ip-10-0-138-149 kubenswrapper[5378]: W1030 20:55:29.742329 5378 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb831b46a_433f_43a6_837f_dd353948593a.slice/crio-5a6fd216937b726be4a3d4bb50c5a9a9c8b1d0cc64515146d7182f858ae8e633 WatchSource:0}: Error finding container 5a6fd216937b726be4a3d4bb50c5a9a9c8b1d0cc64515146d7182f858ae8e633: Status 404 returned error can't find the container with id 5a6fd216937b726be4a3d4bb50c5a9a9c8b1d0cc64515146d7182f858ae8e633 Oct 30 20:55:30.120259 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:55:30.120226 5378 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w5-2b425d-openshift-default-f547dbcbf-2dmdq"] Oct 30 20:55:30.124455 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:55:30.124431 5378 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-2b425d-openshift-default-f547dbcbf-2dmdq" Oct 30 20:55:30.182695 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:55:30.182664 5378 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-2b425d-openshift-default-f547dbcbf-2dmdq"] Oct 30 20:55:30.225967 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:55:30.225935 5378 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w5-2b425d-openshift-default-dockercfg-mnjbj\"" Oct 30 20:55:30.232599 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:55:30.232547 5378 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0359a851-6211-4192-9569-255a5b3dc518-istio-token\") pod \"e2e-worker-w5-2b425d-openshift-default-f547dbcbf-2dmdq\" (UID: \"0359a851-6211-4192-9569-255a5b3dc518\") " pod="openshift-ingress/e2e-worker-w5-2b425d-openshift-default-f547dbcbf-2dmdq" Oct 30 20:55:30.232736 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:55:30.232610 5378 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0359a851-6211-4192-9569-255a5b3dc518-istio-envoy\") pod \"e2e-worker-w5-2b425d-openshift-default-f547dbcbf-2dmdq\" (UID: \"0359a851-6211-4192-9569-255a5b3dc518\") " pod="openshift-ingress/e2e-worker-w5-2b425d-openshift-default-f547dbcbf-2dmdq" Oct 30 20:55:30.232736 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:55:30.232631 5378 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m4wv2\" (UniqueName: \"kubernetes.io/projected/0359a851-6211-4192-9569-255a5b3dc518-kube-api-access-m4wv2\") pod \"e2e-worker-w5-2b425d-openshift-default-f547dbcbf-2dmdq\" (UID: \"0359a851-6211-4192-9569-255a5b3dc518\") " pod="openshift-ingress/e2e-worker-w5-2b425d-openshift-default-f547dbcbf-2dmdq" Oct 30 20:55:30.232736 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:55:30.232714 5378 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0359a851-6211-4192-9569-255a5b3dc518-credential-socket\") pod \"e2e-worker-w5-2b425d-openshift-default-f547dbcbf-2dmdq\" (UID: \"0359a851-6211-4192-9569-255a5b3dc518\") " pod="openshift-ingress/e2e-worker-w5-2b425d-openshift-default-f547dbcbf-2dmdq" Oct 30 20:55:30.232885 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:55:30.232821 5378 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0359a851-6211-4192-9569-255a5b3dc518-workload-socket\") pod \"e2e-worker-w5-2b425d-openshift-default-f547dbcbf-2dmdq\" (UID: \"0359a851-6211-4192-9569-255a5b3dc518\") " pod="openshift-ingress/e2e-worker-w5-2b425d-openshift-default-f547dbcbf-2dmdq" Oct 30 20:55:30.232885 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:55:30.232844 5378 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0359a851-6211-4192-9569-255a5b3dc518-istiod-ca-cert\") pod \"e2e-worker-w5-2b425d-openshift-default-f547dbcbf-2dmdq\" (UID: \"0359a851-6211-4192-9569-255a5b3dc518\") " pod="openshift-ingress/e2e-worker-w5-2b425d-openshift-default-f547dbcbf-2dmdq" Oct 30 20:55:30.232885 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:55:30.232862 5378 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0359a851-6211-4192-9569-255a5b3dc518-istio-data\") pod \"e2e-worker-w5-2b425d-openshift-default-f547dbcbf-2dmdq\" (UID: \"0359a851-6211-4192-9569-255a5b3dc518\") " pod="openshift-ingress/e2e-worker-w5-2b425d-openshift-default-f547dbcbf-2dmdq" Oct 30 20:55:30.232988 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:55:30.232960 5378 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0359a851-6211-4192-9569-255a5b3dc518-workload-certs\") pod \"e2e-worker-w5-2b425d-openshift-default-f547dbcbf-2dmdq\" (UID: \"0359a851-6211-4192-9569-255a5b3dc518\") " pod="openshift-ingress/e2e-worker-w5-2b425d-openshift-default-f547dbcbf-2dmdq" Oct 30 20:55:30.233025 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:55:30.233002 5378 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0359a851-6211-4192-9569-255a5b3dc518-istio-podinfo\") pod \"e2e-worker-w5-2b425d-openshift-default-f547dbcbf-2dmdq\" (UID: \"0359a851-6211-4192-9569-255a5b3dc518\") " pod="openshift-ingress/e2e-worker-w5-2b425d-openshift-default-f547dbcbf-2dmdq" Oct 30 20:55:30.334207 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:55:30.334126 5378 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0359a851-6211-4192-9569-255a5b3dc518-istio-token\") pod \"e2e-worker-w5-2b425d-openshift-default-f547dbcbf-2dmdq\" (UID: \"0359a851-6211-4192-9569-255a5b3dc518\") " pod="openshift-ingress/e2e-worker-w5-2b425d-openshift-default-f547dbcbf-2dmdq" Oct 30 20:55:30.334207 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:55:30.334164 5378 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0359a851-6211-4192-9569-255a5b3dc518-istio-envoy\") pod \"e2e-worker-w5-2b425d-openshift-default-f547dbcbf-2dmdq\" (UID: \"0359a851-6211-4192-9569-255a5b3dc518\") " pod="openshift-ingress/e2e-worker-w5-2b425d-openshift-default-f547dbcbf-2dmdq" Oct 30 20:55:30.334207 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:55:30.334182 5378 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m4wv2\" (UniqueName: \"kubernetes.io/projected/0359a851-6211-4192-9569-255a5b3dc518-kube-api-access-m4wv2\") pod \"e2e-worker-w5-2b425d-openshift-default-f547dbcbf-2dmdq\" (UID: \"0359a851-6211-4192-9569-255a5b3dc518\") " pod="openshift-ingress/e2e-worker-w5-2b425d-openshift-default-f547dbcbf-2dmdq" Oct 30 20:55:30.334207 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:55:30.334199 5378 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0359a851-6211-4192-9569-255a5b3dc518-credential-socket\") pod \"e2e-worker-w5-2b425d-openshift-default-f547dbcbf-2dmdq\" (UID: \"0359a851-6211-4192-9569-255a5b3dc518\") " pod="openshift-ingress/e2e-worker-w5-2b425d-openshift-default-f547dbcbf-2dmdq" Oct 30 20:55:30.334560 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:55:30.334238 5378 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0359a851-6211-4192-9569-255a5b3dc518-workload-socket\") pod \"e2e-worker-w5-2b425d-openshift-default-f547dbcbf-2dmdq\" (UID: \"0359a851-6211-4192-9569-255a5b3dc518\") " pod="openshift-ingress/e2e-worker-w5-2b425d-openshift-default-f547dbcbf-2dmdq" Oct 30 20:55:30.334560 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:55:30.334258 5378 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0359a851-6211-4192-9569-255a5b3dc518-istiod-ca-cert\") pod \"e2e-worker-w5-2b425d-openshift-default-f547dbcbf-2dmdq\" (UID: \"0359a851-6211-4192-9569-255a5b3dc518\") " pod="openshift-ingress/e2e-worker-w5-2b425d-openshift-default-f547dbcbf-2dmdq" Oct 30 20:55:30.334560 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:55:30.334275 5378 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0359a851-6211-4192-9569-255a5b3dc518-istio-data\") pod \"e2e-worker-w5-2b425d-openshift-default-f547dbcbf-2dmdq\" (UID: \"0359a851-6211-4192-9569-255a5b3dc518\") " pod="openshift-ingress/e2e-worker-w5-2b425d-openshift-default-f547dbcbf-2dmdq" Oct 30 20:55:30.334560 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:55:30.334334 5378 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0359a851-6211-4192-9569-255a5b3dc518-workload-certs\") pod \"e2e-worker-w5-2b425d-openshift-default-f547dbcbf-2dmdq\" (UID: \"0359a851-6211-4192-9569-255a5b3dc518\") " pod="openshift-ingress/e2e-worker-w5-2b425d-openshift-default-f547dbcbf-2dmdq" Oct 30 20:55:30.334560 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:55:30.334382 5378 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0359a851-6211-4192-9569-255a5b3dc518-istio-podinfo\") pod \"e2e-worker-w5-2b425d-openshift-default-f547dbcbf-2dmdq\" (UID: \"0359a851-6211-4192-9569-255a5b3dc518\") " pod="openshift-ingress/e2e-worker-w5-2b425d-openshift-default-f547dbcbf-2dmdq" Oct 30 20:55:30.334808 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:55:30.334711 5378 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0359a851-6211-4192-9569-255a5b3dc518-workload-socket\") pod \"e2e-worker-w5-2b425d-openshift-default-f547dbcbf-2dmdq\" (UID: \"0359a851-6211-4192-9569-255a5b3dc518\") " pod="openshift-ingress/e2e-worker-w5-2b425d-openshift-default-f547dbcbf-2dmdq" Oct 30 20:55:30.334867 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:55:30.334811 5378 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0359a851-6211-4192-9569-255a5b3dc518-credential-socket\") pod \"e2e-worker-w5-2b425d-openshift-default-f547dbcbf-2dmdq\" (UID: \"0359a851-6211-4192-9569-255a5b3dc518\") " pod="openshift-ingress/e2e-worker-w5-2b425d-openshift-default-f547dbcbf-2dmdq" Oct 30 20:55:30.334921 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:55:30.334885 5378 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0359a851-6211-4192-9569-255a5b3dc518-istio-data\") pod \"e2e-worker-w5-2b425d-openshift-default-f547dbcbf-2dmdq\" (UID: \"0359a851-6211-4192-9569-255a5b3dc518\") " pod="openshift-ingress/e2e-worker-w5-2b425d-openshift-default-f547dbcbf-2dmdq" Oct 30 20:55:30.334981 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:55:30.334933 5378 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0359a851-6211-4192-9569-255a5b3dc518-workload-certs\") pod \"e2e-worker-w5-2b425d-openshift-default-f547dbcbf-2dmdq\" (UID: \"0359a851-6211-4192-9569-255a5b3dc518\") " pod="openshift-ingress/e2e-worker-w5-2b425d-openshift-default-f547dbcbf-2dmdq" Oct 30 20:55:30.335058 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:55:30.335038 5378 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0359a851-6211-4192-9569-255a5b3dc518-istiod-ca-cert\") pod \"e2e-worker-w5-2b425d-openshift-default-f547dbcbf-2dmdq\" (UID: \"0359a851-6211-4192-9569-255a5b3dc518\") " pod="openshift-ingress/e2e-worker-w5-2b425d-openshift-default-f547dbcbf-2dmdq" Oct 30 20:55:30.336430 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:55:30.336410 5378 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0359a851-6211-4192-9569-255a5b3dc518-istio-envoy\") pod \"e2e-worker-w5-2b425d-openshift-default-f547dbcbf-2dmdq\" (UID: \"0359a851-6211-4192-9569-255a5b3dc518\") " pod="openshift-ingress/e2e-worker-w5-2b425d-openshift-default-f547dbcbf-2dmdq" Oct 30 20:55:30.336862 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:55:30.336845 5378 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0359a851-6211-4192-9569-255a5b3dc518-istio-podinfo\") pod \"e2e-worker-w5-2b425d-openshift-default-f547dbcbf-2dmdq\" (UID: \"0359a851-6211-4192-9569-255a5b3dc518\") " pod="openshift-ingress/e2e-worker-w5-2b425d-openshift-default-f547dbcbf-2dmdq" Oct 30 20:55:30.391979 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:55:30.391947 5378 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0359a851-6211-4192-9569-255a5b3dc518-istio-token\") pod \"e2e-worker-w5-2b425d-openshift-default-f547dbcbf-2dmdq\" (UID: \"0359a851-6211-4192-9569-255a5b3dc518\") " pod="openshift-ingress/e2e-worker-w5-2b425d-openshift-default-f547dbcbf-2dmdq" Oct 30 20:55:30.392135 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:55:30.392114 5378 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m4wv2\" (UniqueName: \"kubernetes.io/projected/0359a851-6211-4192-9569-255a5b3dc518-kube-api-access-m4wv2\") pod \"e2e-worker-w5-2b425d-openshift-default-f547dbcbf-2dmdq\" (UID: \"0359a851-6211-4192-9569-255a5b3dc518\") " pod="openshift-ingress/e2e-worker-w5-2b425d-openshift-default-f547dbcbf-2dmdq" Oct 30 20:55:30.439004 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:55:30.438972 5378 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-2b425d-openshift-default-f547dbcbf-2dmdq" Oct 30 20:55:30.498647 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:55:30.497744 5378 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f3113105-xn7nh" event={"ID":"b831b46a-433f-43a6-837f-dd353948593a","Type":"ContainerStarted","Data":"6a666916611bf7e2a5928fd7680f91ad2a0830d8df9a435ed444316e1cb957ce"} Oct 30 20:55:30.498647 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:55:30.497814 5378 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f3113105-xn7nh" event={"ID":"b831b46a-433f-43a6-837f-dd353948593a","Type":"ContainerStarted","Data":"5a6fd216937b726be4a3d4bb50c5a9a9c8b1d0cc64515146d7182f858ae8e633"} Oct 30 20:55:30.617418 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:55:30.617371 5378 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f3113105-xn7nh" podStartSLOduration=1.6173577049999999 podStartE2EDuration="1.617357705s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:55:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:55:30.615468973 +0000 UTC m=+1021.043582892" watchObservedRunningTime="2026-10-30 20:55:30.617357705 +0000 UTC m=+1021.045471668" Oct 30 20:55:30.868771 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:55:30.868741 5378 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-2b425d-openshift-default-f547dbcbf-2dmdq"] Oct 30 20:55:30.869837 ip-10-0-138-149 kubenswrapper[5378]: W1030 20:55:30.869808 5378 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0359a851_6211_4192_9569_255a5b3dc518.slice/crio-148e8811b1cbdf509d183a5f30c129066a57ea490904c4ff483622d0111fddad WatchSource:0}: Error finding container 148e8811b1cbdf509d183a5f30c129066a57ea490904c4ff483622d0111fddad: Status 404 returned error can't find the container with id 148e8811b1cbdf509d183a5f30c129066a57ea490904c4ff483622d0111fddad Oct 30 20:55:30.872179 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:55:30.872150 5378 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 20:55:30.872257 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:55:30.872207 5378 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 20:55:30.872257 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:55:30.872239 5378 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 20:55:31.504952 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:55:31.504867 5378 generic.go:356] "Generic (PLEG): container finished" podID="b831b46a-433f-43a6-837f-dd353948593a" containerID="6a666916611bf7e2a5928fd7680f91ad2a0830d8df9a435ed444316e1cb957ce" exitCode=0 Oct 30 20:55:31.505309 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:55:31.504952 5378 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f3113105-xn7nh" event={"ID":"b831b46a-433f-43a6-837f-dd353948593a","Type":"ContainerDied","Data":"6a666916611bf7e2a5928fd7680f91ad2a0830d8df9a435ed444316e1cb957ce"} Oct 30 20:55:31.506603 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:55:31.506578 5378 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-2b425d-openshift-default-f547dbcbf-2dmdq" event={"ID":"0359a851-6211-4192-9569-255a5b3dc518","Type":"ContainerStarted","Data":"e9fd800b4f60a2b6e715d2da7ae20fc8df38a0a3782046ca707f2856ac9d78f0"} Oct 30 20:55:31.506656 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:55:31.506629 5378 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-2b425d-openshift-default-f547dbcbf-2dmdq" event={"ID":"0359a851-6211-4192-9569-255a5b3dc518","Type":"ContainerStarted","Data":"148e8811b1cbdf509d183a5f30c129066a57ea490904c4ff483622d0111fddad"} Oct 30 20:55:31.752827 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:55:31.752761 5378 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w5-2b425d-openshift-default-f547dbcbf-2dmdq" podStartSLOduration=1.752744633 podStartE2EDuration="1.752744633s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:55:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:55:31.752183503 +0000 UTC m=+1022.180297433" watchObservedRunningTime="2026-10-30 20:55:31.752744633 +0000 UTC m=+1022.180858555" Oct 30 20:55:32.011863 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:55:32.011813 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 20:55:32.439376 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:55:32.439345 5378 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w5-2b425d-openshift-default-f547dbcbf-2dmdq" Oct 30 20:55:32.440785 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:55:32.440761 5378 patch_prober.go:29] interesting pod/e2e-worker-w5-2b425d-openshift-default-f547dbcbf-2dmdq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused" start-of-body= Oct 30 20:55:32.440905 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:55:32.440807 5378 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-2b425d-openshift-default-f547dbcbf-2dmdq" podUID="0359a851-6211-4192-9569-255a5b3dc518" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused" Oct 30 20:55:32.691673 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:55:32.691605 5378 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f3113105-xn7nh" Oct 30 20:55:32.781710 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:55:32.781679 5378 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b831b46a-433f-43a6-837f-dd353948593a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b831b46a-433f-43a6-837f-dd353948593a-maas-api-openshift-ca\") pod \"b831b46a-433f-43a6-837f-dd353948593a\" (UID: \"b831b46a-433f-43a6-837f-dd353948593a\") " Oct 30 20:55:32.781887 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:55:32.781744 5378 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b831b46a-433f-43a6-837f-dd353948593a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b831b46a-433f-43a6-837f-dd353948593a-maas-api-xks-ca\") pod \"b831b46a-433f-43a6-837f-dd353948593a\" (UID: \"b831b46a-433f-43a6-837f-dd353948593a\") " Oct 30 20:55:32.781887 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:55:32.781816 5378 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b831b46a-433f-43a6-837f-dd353948593a-kube-api-access-974rv\" (UniqueName: \"kubernetes.io/projected/b831b46a-433f-43a6-837f-dd353948593a-kube-api-access-974rv\") pod \"b831b46a-433f-43a6-837f-dd353948593a\" (UID: \"b831b46a-433f-43a6-837f-dd353948593a\") " Oct 30 20:55:32.782038 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:55:32.782012 5378 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b831b46a-433f-43a6-837f-dd353948593a-maas-api-openshift-ca" pod "b831b46a-433f-43a6-837f-dd353948593a" (UID: "b831b46a-433f-43a6-837f-dd353948593a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:55:32.782134 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:55:32.782118 5378 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b831b46a-433f-43a6-837f-dd353948593a-maas-api-openshift-ca\") on node \"ip-10-0-138-149.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:55:32.783911 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:55:32.783884 5378 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b831b46a-433f-43a6-837f-dd353948593a-kube-api-access-974rv" pod "b831b46a-433f-43a6-837f-dd353948593a" (UID: "b831b46a-433f-43a6-837f-dd353948593a"). InnerVolumeSpecName "kube-api-access-974rv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:55:32.783911 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:55:32.783880 5378 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b831b46a-433f-43a6-837f-dd353948593a-maas-api-xks-ca" pod "b831b46a-433f-43a6-837f-dd353948593a" (UID: "b831b46a-433f-43a6-837f-dd353948593a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 20:55:32.883787 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:55:32.883758 5378 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b831b46a-433f-43a6-837f-dd353948593a-maas-api-xks-ca\") on node \"ip-10-0-138-149.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:55:32.883787 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:55:32.883784 5378 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-974rv\" (UniqueName: \"kubernetes.io/projected/b831b46a-433f-43a6-837f-dd353948593a-kube-api-access-974rv\") on node \"ip-10-0-138-149.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:55:33.439687 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:55:33.439648 5378 patch_prober.go:29] interesting pod/e2e-worker-w5-2b425d-openshift-default-f547dbcbf-2dmdq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused" start-of-body= Oct 30 20:55:33.439854 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:55:33.439707 5378 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-2b425d-openshift-default-f547dbcbf-2dmdq" podUID="0359a851-6211-4192-9569-255a5b3dc518" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused" Oct 30 20:55:33.518687 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:55:33.518654 5378 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f3113105-xn7nh" event={"ID":"b831b46a-433f-43a6-837f-dd353948593a","Type":"ContainerDied","Data":"5a6fd216937b726be4a3d4bb50c5a9a9c8b1d0cc64515146d7182f858ae8e633"} Oct 30 20:55:33.518687 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:55:33.518685 5378 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f3113105-xn7nh" Oct 30 20:55:33.518894 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:55:33.518727 5378 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5a6fd216937b726be4a3d4bb50c5a9a9c8b1d0cc64515146d7182f858ae8e633" Oct 30 20:55:34.440038 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:55:34.440009 5378 patch_prober.go:29] interesting pod/e2e-worker-w5-2b425d-openshift-default-f547dbcbf-2dmdq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused" start-of-body= Oct 30 20:55:34.440423 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:55:34.440057 5378 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-2b425d-openshift-default-f547dbcbf-2dmdq" podUID="0359a851-6211-4192-9569-255a5b3dc518" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused" Oct 30 20:55:35.441037 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:55:35.441000 5378 patch_prober.go:29] interesting pod/e2e-worker-w5-2b425d-openshift-default-f547dbcbf-2dmdq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused" start-of-body= Oct 30 20:55:35.441645 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:55:35.441613 5378 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-2b425d-openshift-default-f547dbcbf-2dmdq" podUID="0359a851-6211-4192-9569-255a5b3dc518" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused" Oct 30 20:55:36.443008 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:55:36.442971 5378 patch_prober.go:29] interesting pod/e2e-worker-w5-2b425d-openshift-default-f547dbcbf-2dmdq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused" start-of-body= Oct 30 20:55:36.443657 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:55:36.443615 5378 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-2b425d-openshift-default-f547dbcbf-2dmdq" podUID="0359a851-6211-4192-9569-255a5b3dc518" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused" Oct 30 20:55:37.177735 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:55:37.177697 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 20:55:37.440130 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:55:37.440044 5378 patch_prober.go:29] interesting pod/e2e-worker-w5-2b425d-openshift-default-f547dbcbf-2dmdq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused" start-of-body= Oct 30 20:55:37.440130 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:55:37.440097 5378 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-2b425d-openshift-default-f547dbcbf-2dmdq" podUID="0359a851-6211-4192-9569-255a5b3dc518" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused" Oct 30 20:55:38.444317 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:55:38.444289 5378 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w5-2b425d-openshift-default-f547dbcbf-2dmdq" Oct 30 20:55:38.444729 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:55:38.444671 5378 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w5-2b425d-openshift-default-f547dbcbf-2dmdq" Oct 30 20:55:38.445346 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:55:38.445326 5378 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w5-2b425d-openshift-default-f547dbcbf-2dmdq" Oct 30 20:55:41.501378 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:55:41.501350 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 20:55:42.116854 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:55:42.116815 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 20:55:46.848589 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:55:46.848535 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 20:55:48.837337 ip-10-0-138-149 kubenswrapper[5378]: E1030 20:55:48.837292 5378 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q6mx7" containerName="extractor" Oct 30 20:55:51.649592 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:55:51.649547 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 20:55:53.916159 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:55:53.916122 5378 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-r9g59"] Oct 30 20:55:56.550266 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:55:56.550232 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 20:56:01.345145 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:56:01.345113 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 20:56:03.798667 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:56:03.798633 5378 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w2-d061d0/distinct-w2-d061d0-kserve-679c7cd7c4-8czrv"] Oct 30 20:56:03.799174 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:56:03.799159 5378 cpu_manager.go:414] "containerMap: removing container" podUID="b831b46a-433f-43a6-837f-dd353948593a" containerName="revoke-keys" Oct 30 20:56:03.799241 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:56:03.799175 5378 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b831b46a-433f-43a6-837f-dd353948593a" containerName="revoke-keys" Oct 30 20:56:03.799297 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:56:03.799282 5378 memory_manager.go:365] "RemoveStaleState removing state" podUID="b831b46a-433f-43a6-837f-dd353948593a" containerName="revoke-keys" Oct 30 20:56:03.803816 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:56:03.803794 5378 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-d061d0/distinct-w2-d061d0-kserve-679c7cd7c4-8czrv" Oct 30 20:56:03.876420 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:56:03.876387 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 20:56:03.907406 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:56:03.907381 5378 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-d061d0\"/\"distinct-w2-d061d0-kserve-self-signed-certs\"" Oct 30 20:56:03.909806 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:56:03.909784 5378 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-d061d0/distinct-w2-d061d0-kserve-679c7cd7c4-8czrv"] Oct 30 20:56:03.951100 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:56:03.950994 5378 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2c139b9f-c7df-4675-b431-95d4136095e5-tls-certs\") pod \"distinct-w2-d061d0-kserve-679c7cd7c4-8czrv\" (UID: \"2c139b9f-c7df-4675-b431-95d4136095e5\") " pod="e2e-models-e2e-worker-w2-d061d0/distinct-w2-d061d0-kserve-679c7cd7c4-8czrv" Oct 30 20:56:03.951500 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:56:03.951465 5378 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2c139b9f-c7df-4675-b431-95d4136095e5-home\") pod \"distinct-w2-d061d0-kserve-679c7cd7c4-8czrv\" (UID: \"2c139b9f-c7df-4675-b431-95d4136095e5\") " pod="e2e-models-e2e-worker-w2-d061d0/distinct-w2-d061d0-kserve-679c7cd7c4-8czrv" Oct 30 20:56:03.951500 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:56:03.951515 5378 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2c139b9f-c7df-4675-b431-95d4136095e5-tmp-dir\") pod \"distinct-w2-d061d0-kserve-679c7cd7c4-8czrv\" (UID: \"2c139b9f-c7df-4675-b431-95d4136095e5\") " pod="e2e-models-e2e-worker-w2-d061d0/distinct-w2-d061d0-kserve-679c7cd7c4-8czrv" Oct 30 20:56:03.951783 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:56:03.951690 5378 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b5tz4\" (UniqueName: \"kubernetes.io/projected/2c139b9f-c7df-4675-b431-95d4136095e5-kube-api-access-b5tz4\") pod \"distinct-w2-d061d0-kserve-679c7cd7c4-8czrv\" (UID: \"2c139b9f-c7df-4675-b431-95d4136095e5\") " pod="e2e-models-e2e-worker-w2-d061d0/distinct-w2-d061d0-kserve-679c7cd7c4-8czrv" Oct 30 20:56:03.951783 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:56:03.951753 5378 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2c139b9f-c7df-4675-b431-95d4136095e5-model-cache\") pod \"distinct-w2-d061d0-kserve-679c7cd7c4-8czrv\" (UID: \"2c139b9f-c7df-4675-b431-95d4136095e5\") " pod="e2e-models-e2e-worker-w2-d061d0/distinct-w2-d061d0-kserve-679c7cd7c4-8czrv" Oct 30 20:56:03.951935 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:56:03.951790 5378 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2c139b9f-c7df-4675-b431-95d4136095e5-dshm\") pod \"distinct-w2-d061d0-kserve-679c7cd7c4-8czrv\" (UID: \"2c139b9f-c7df-4675-b431-95d4136095e5\") " pod="e2e-models-e2e-worker-w2-d061d0/distinct-w2-d061d0-kserve-679c7cd7c4-8czrv" Oct 30 20:56:04.052899 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:56:04.052797 5378 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2c139b9f-c7df-4675-b431-95d4136095e5-home\") pod \"distinct-w2-d061d0-kserve-679c7cd7c4-8czrv\" (UID: \"2c139b9f-c7df-4675-b431-95d4136095e5\") " pod="e2e-models-e2e-worker-w2-d061d0/distinct-w2-d061d0-kserve-679c7cd7c4-8czrv" Oct 30 20:56:04.052899 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:56:04.052844 5378 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2c139b9f-c7df-4675-b431-95d4136095e5-tmp-dir\") pod \"distinct-w2-d061d0-kserve-679c7cd7c4-8czrv\" (UID: \"2c139b9f-c7df-4675-b431-95d4136095e5\") " pod="e2e-models-e2e-worker-w2-d061d0/distinct-w2-d061d0-kserve-679c7cd7c4-8czrv" Oct 30 20:56:04.053169 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:56:04.052970 5378 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b5tz4\" (UniqueName: \"kubernetes.io/projected/2c139b9f-c7df-4675-b431-95d4136095e5-kube-api-access-b5tz4\") pod \"distinct-w2-d061d0-kserve-679c7cd7c4-8czrv\" (UID: \"2c139b9f-c7df-4675-b431-95d4136095e5\") " pod="e2e-models-e2e-worker-w2-d061d0/distinct-w2-d061d0-kserve-679c7cd7c4-8czrv" Oct 30 20:56:04.053169 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:56:04.053030 5378 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2c139b9f-c7df-4675-b431-95d4136095e5-model-cache\") pod \"distinct-w2-d061d0-kserve-679c7cd7c4-8czrv\" (UID: \"2c139b9f-c7df-4675-b431-95d4136095e5\") " pod="e2e-models-e2e-worker-w2-d061d0/distinct-w2-d061d0-kserve-679c7cd7c4-8czrv" Oct 30 20:56:04.053169 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:56:04.053071 5378 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2c139b9f-c7df-4675-b431-95d4136095e5-dshm\") pod \"distinct-w2-d061d0-kserve-679c7cd7c4-8czrv\" (UID: \"2c139b9f-c7df-4675-b431-95d4136095e5\") " pod="e2e-models-e2e-worker-w2-d061d0/distinct-w2-d061d0-kserve-679c7cd7c4-8czrv" Oct 30 20:56:04.053536 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:56:04.053499 5378 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2c139b9f-c7df-4675-b431-95d4136095e5-tmp-dir\") pod \"distinct-w2-d061d0-kserve-679c7cd7c4-8czrv\" (UID: \"2c139b9f-c7df-4675-b431-95d4136095e5\") " pod="e2e-models-e2e-worker-w2-d061d0/distinct-w2-d061d0-kserve-679c7cd7c4-8czrv" Oct 30 20:56:04.053536 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:56:04.053516 5378 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2c139b9f-c7df-4675-b431-95d4136095e5-tls-certs\") pod \"distinct-w2-d061d0-kserve-679c7cd7c4-8czrv\" (UID: \"2c139b9f-c7df-4675-b431-95d4136095e5\") " pod="e2e-models-e2e-worker-w2-d061d0/distinct-w2-d061d0-kserve-679c7cd7c4-8czrv" Oct 30 20:56:04.053749 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:56:04.053593 5378 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2c139b9f-c7df-4675-b431-95d4136095e5-home\") pod \"distinct-w2-d061d0-kserve-679c7cd7c4-8czrv\" (UID: \"2c139b9f-c7df-4675-b431-95d4136095e5\") " pod="e2e-models-e2e-worker-w2-d061d0/distinct-w2-d061d0-kserve-679c7cd7c4-8czrv" Oct 30 20:56:04.053749 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:56:04.053666 5378 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2c139b9f-c7df-4675-b431-95d4136095e5-model-cache\") pod \"distinct-w2-d061d0-kserve-679c7cd7c4-8czrv\" (UID: \"2c139b9f-c7df-4675-b431-95d4136095e5\") " pod="e2e-models-e2e-worker-w2-d061d0/distinct-w2-d061d0-kserve-679c7cd7c4-8czrv" Oct 30 20:56:04.055636 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:56:04.055610 5378 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2c139b9f-c7df-4675-b431-95d4136095e5-dshm\") pod \"distinct-w2-d061d0-kserve-679c7cd7c4-8czrv\" (UID: \"2c139b9f-c7df-4675-b431-95d4136095e5\") " pod="e2e-models-e2e-worker-w2-d061d0/distinct-w2-d061d0-kserve-679c7cd7c4-8czrv" Oct 30 20:56:04.056032 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:56:04.056016 5378 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2c139b9f-c7df-4675-b431-95d4136095e5-tls-certs\") pod \"distinct-w2-d061d0-kserve-679c7cd7c4-8czrv\" (UID: \"2c139b9f-c7df-4675-b431-95d4136095e5\") " pod="e2e-models-e2e-worker-w2-d061d0/distinct-w2-d061d0-kserve-679c7cd7c4-8czrv" Oct 30 20:56:04.111285 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:56:04.111252 5378 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b5tz4\" (UniqueName: \"kubernetes.io/projected/2c139b9f-c7df-4675-b431-95d4136095e5-kube-api-access-b5tz4\") pod \"distinct-w2-d061d0-kserve-679c7cd7c4-8czrv\" (UID: \"2c139b9f-c7df-4675-b431-95d4136095e5\") " pod="e2e-models-e2e-worker-w2-d061d0/distinct-w2-d061d0-kserve-679c7cd7c4-8czrv" Oct 30 20:56:04.116099 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:56:04.116071 5378 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-d061d0/distinct-w2-d061d0-kserve-679c7cd7c4-8czrv" Oct 30 20:56:04.538643 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:56:04.538615 5378 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-d061d0/distinct-w2-d061d0-kserve-679c7cd7c4-8czrv"] Oct 30 20:56:04.538766 ip-10-0-138-149 kubenswrapper[5378]: W1030 20:56:04.538722 5378 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod2c139b9f_c7df_4675_b431_95d4136095e5.slice/crio-3c41ff3842d6b2c8544d235e3321d4bb093903d35c776332295c8263f40d4606 WatchSource:0}: Error finding container 3c41ff3842d6b2c8544d235e3321d4bb093903d35c776332295c8263f40d4606: Status 404 returned error can't find the container with id 3c41ff3842d6b2c8544d235e3321d4bb093903d35c776332295c8263f40d4606 Oct 30 20:56:04.733683 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:56:04.733646 5378 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-d061d0/distinct-w2-d061d0-kserve-679c7cd7c4-8czrv" event={"ID":"2c139b9f-c7df-4675-b431-95d4136095e5","Type":"ContainerStarted","Data":"c7558118e9ce72affc672e1d18263ca42d790131c00a7dac78fd384bcd70f1aa"} Oct 30 20:56:04.733683 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:56:04.733687 5378 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-d061d0/distinct-w2-d061d0-kserve-679c7cd7c4-8czrv" event={"ID":"2c139b9f-c7df-4675-b431-95d4136095e5","Type":"ContainerStarted","Data":"3c41ff3842d6b2c8544d235e3321d4bb093903d35c776332295c8263f40d4606"} Oct 30 20:56:04.850217 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:56:04.850165 5378 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w2-d061d0/distinct-w2-d061d0-kserve-679c7cd7c4-8czrv" podStartSLOduration=1.8501518099999998 podStartE2EDuration="1.85015181s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:56:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:56:04.849022982 +0000 UTC m=+1055.277136901" watchObservedRunningTime="2026-10-30 20:56:04.85015181 +0000 UTC m=+1055.278265729" Oct 30 20:56:05.116691 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:56:05.116619 5378 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-d061d0/distinct-w2-d061d0-kserve-679c7cd7c4-8czrv" Oct 30 20:56:06.249400 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:56:06.249368 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 20:56:11.000165 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:56:11.000132 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 20:56:13.630957 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:56:13.630922 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 20:56:14.116665 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:56:14.116627 5378 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w2-d061d0/distinct-w2-d061d0-kserve-679c7cd7c4-8czrv" Oct 30 20:56:14.136939 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:56:14.136913 5378 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w2-d061d0/distinct-w2-d061d0-kserve-679c7cd7c4-8czrv" Oct 30 20:56:14.803135 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:56:14.803104 5378 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-d061d0/distinct-w2-d061d0-kserve-679c7cd7c4-8czrv" Oct 30 20:56:15.856878 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:56:15.856848 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 20:56:17.616842 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:56:17.616806 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 20:56:18.838343 ip-10-0-138-149 kubenswrapper[5378]: E1030 20:56:18.838297 5378 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q6mx7" containerName="extractor" Oct 30 20:56:20.747978 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:56:20.747944 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 20:56:21.775451 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:56:21.775416 5378 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-r9g59"] Oct 30 20:56:23.451047 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:56:23.451012 5378 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w4-1fdaa7/distinct-w4-1fdaa7-kserve-86dc9774b9-6x2gf"] Oct 30 20:56:23.455083 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:56:23.455054 5378 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-1fdaa7/distinct-w4-1fdaa7-kserve-86dc9774b9-6x2gf" Oct 30 20:56:23.511704 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:56:23.511670 5378 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-1fdaa7/distinct-w4-1fdaa7-kserve-86dc9774b9-6x2gf"] Oct 30 20:56:23.528992 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:56:23.528960 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 20:56:23.557965 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:56:23.557934 5378 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-1fdaa7\"/\"distinct-w4-1fdaa7-kserve-self-signed-certs\"" Oct 30 20:56:23.575430 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:56:23.575397 5378 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2e162c95-0527-4cb3-85a3-795aae212d29-dshm\") pod \"distinct-w4-1fdaa7-kserve-86dc9774b9-6x2gf\" (UID: \"2e162c95-0527-4cb3-85a3-795aae212d29\") " pod="e2e-models-e2e-worker-w4-1fdaa7/distinct-w4-1fdaa7-kserve-86dc9774b9-6x2gf" Oct 30 20:56:23.575635 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:56:23.575444 5378 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fq5tf\" (UniqueName: \"kubernetes.io/projected/2e162c95-0527-4cb3-85a3-795aae212d29-kube-api-access-fq5tf\") pod \"distinct-w4-1fdaa7-kserve-86dc9774b9-6x2gf\" (UID: \"2e162c95-0527-4cb3-85a3-795aae212d29\") " pod="e2e-models-e2e-worker-w4-1fdaa7/distinct-w4-1fdaa7-kserve-86dc9774b9-6x2gf" Oct 30 20:56:23.575635 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:56:23.575503 5378 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2e162c95-0527-4cb3-85a3-795aae212d29-home\") pod \"distinct-w4-1fdaa7-kserve-86dc9774b9-6x2gf\" (UID: \"2e162c95-0527-4cb3-85a3-795aae212d29\") " pod="e2e-models-e2e-worker-w4-1fdaa7/distinct-w4-1fdaa7-kserve-86dc9774b9-6x2gf" Oct 30 20:56:23.575772 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:56:23.575696 5378 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2e162c95-0527-4cb3-85a3-795aae212d29-tmp-dir\") pod \"distinct-w4-1fdaa7-kserve-86dc9774b9-6x2gf\" (UID: \"2e162c95-0527-4cb3-85a3-795aae212d29\") " pod="e2e-models-e2e-worker-w4-1fdaa7/distinct-w4-1fdaa7-kserve-86dc9774b9-6x2gf" Oct 30 20:56:23.575839 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:56:23.575822 5378 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2e162c95-0527-4cb3-85a3-795aae212d29-tls-certs\") pod \"distinct-w4-1fdaa7-kserve-86dc9774b9-6x2gf\" (UID: \"2e162c95-0527-4cb3-85a3-795aae212d29\") " pod="e2e-models-e2e-worker-w4-1fdaa7/distinct-w4-1fdaa7-kserve-86dc9774b9-6x2gf" Oct 30 20:56:23.575900 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:56:23.575848 5378 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2e162c95-0527-4cb3-85a3-795aae212d29-model-cache\") pod \"distinct-w4-1fdaa7-kserve-86dc9774b9-6x2gf\" (UID: \"2e162c95-0527-4cb3-85a3-795aae212d29\") " pod="e2e-models-e2e-worker-w4-1fdaa7/distinct-w4-1fdaa7-kserve-86dc9774b9-6x2gf" Oct 30 20:56:23.677209 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:56:23.677177 5378 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2e162c95-0527-4cb3-85a3-795aae212d29-tls-certs\") pod \"distinct-w4-1fdaa7-kserve-86dc9774b9-6x2gf\" (UID: \"2e162c95-0527-4cb3-85a3-795aae212d29\") " pod="e2e-models-e2e-worker-w4-1fdaa7/distinct-w4-1fdaa7-kserve-86dc9774b9-6x2gf" Oct 30 20:56:23.677209 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:56:23.677211 5378 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2e162c95-0527-4cb3-85a3-795aae212d29-model-cache\") pod \"distinct-w4-1fdaa7-kserve-86dc9774b9-6x2gf\" (UID: \"2e162c95-0527-4cb3-85a3-795aae212d29\") " pod="e2e-models-e2e-worker-w4-1fdaa7/distinct-w4-1fdaa7-kserve-86dc9774b9-6x2gf" Oct 30 20:56:23.677448 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:56:23.677233 5378 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2e162c95-0527-4cb3-85a3-795aae212d29-dshm\") pod \"distinct-w4-1fdaa7-kserve-86dc9774b9-6x2gf\" (UID: \"2e162c95-0527-4cb3-85a3-795aae212d29\") " pod="e2e-models-e2e-worker-w4-1fdaa7/distinct-w4-1fdaa7-kserve-86dc9774b9-6x2gf" Oct 30 20:56:23.677448 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:56:23.677268 5378 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fq5tf\" (UniqueName: \"kubernetes.io/projected/2e162c95-0527-4cb3-85a3-795aae212d29-kube-api-access-fq5tf\") pod \"distinct-w4-1fdaa7-kserve-86dc9774b9-6x2gf\" (UID: \"2e162c95-0527-4cb3-85a3-795aae212d29\") " pod="e2e-models-e2e-worker-w4-1fdaa7/distinct-w4-1fdaa7-kserve-86dc9774b9-6x2gf" Oct 30 20:56:23.677448 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:56:23.677394 5378 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2e162c95-0527-4cb3-85a3-795aae212d29-home\") pod \"distinct-w4-1fdaa7-kserve-86dc9774b9-6x2gf\" (UID: \"2e162c95-0527-4cb3-85a3-795aae212d29\") " pod="e2e-models-e2e-worker-w4-1fdaa7/distinct-w4-1fdaa7-kserve-86dc9774b9-6x2gf" Oct 30 20:56:23.677657 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:56:23.677550 5378 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2e162c95-0527-4cb3-85a3-795aae212d29-tmp-dir\") pod \"distinct-w4-1fdaa7-kserve-86dc9774b9-6x2gf\" (UID: \"2e162c95-0527-4cb3-85a3-795aae212d29\") " pod="e2e-models-e2e-worker-w4-1fdaa7/distinct-w4-1fdaa7-kserve-86dc9774b9-6x2gf" Oct 30 20:56:23.677715 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:56:23.677670 5378 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2e162c95-0527-4cb3-85a3-795aae212d29-model-cache\") pod \"distinct-w4-1fdaa7-kserve-86dc9774b9-6x2gf\" (UID: \"2e162c95-0527-4cb3-85a3-795aae212d29\") " pod="e2e-models-e2e-worker-w4-1fdaa7/distinct-w4-1fdaa7-kserve-86dc9774b9-6x2gf" Oct 30 20:56:23.677789 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:56:23.677766 5378 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2e162c95-0527-4cb3-85a3-795aae212d29-home\") pod \"distinct-w4-1fdaa7-kserve-86dc9774b9-6x2gf\" (UID: \"2e162c95-0527-4cb3-85a3-795aae212d29\") " pod="e2e-models-e2e-worker-w4-1fdaa7/distinct-w4-1fdaa7-kserve-86dc9774b9-6x2gf" Oct 30 20:56:23.677855 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:56:23.677839 5378 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2e162c95-0527-4cb3-85a3-795aae212d29-tmp-dir\") pod \"distinct-w4-1fdaa7-kserve-86dc9774b9-6x2gf\" (UID: \"2e162c95-0527-4cb3-85a3-795aae212d29\") " pod="e2e-models-e2e-worker-w4-1fdaa7/distinct-w4-1fdaa7-kserve-86dc9774b9-6x2gf" Oct 30 20:56:23.679712 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:56:23.679680 5378 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2e162c95-0527-4cb3-85a3-795aae212d29-dshm\") pod \"distinct-w4-1fdaa7-kserve-86dc9774b9-6x2gf\" (UID: \"2e162c95-0527-4cb3-85a3-795aae212d29\") " pod="e2e-models-e2e-worker-w4-1fdaa7/distinct-w4-1fdaa7-kserve-86dc9774b9-6x2gf" Oct 30 20:56:23.679857 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:56:23.679841 5378 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2e162c95-0527-4cb3-85a3-795aae212d29-tls-certs\") pod \"distinct-w4-1fdaa7-kserve-86dc9774b9-6x2gf\" (UID: \"2e162c95-0527-4cb3-85a3-795aae212d29\") " pod="e2e-models-e2e-worker-w4-1fdaa7/distinct-w4-1fdaa7-kserve-86dc9774b9-6x2gf" Oct 30 20:56:23.734901 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:56:23.734827 5378 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fq5tf\" (UniqueName: \"kubernetes.io/projected/2e162c95-0527-4cb3-85a3-795aae212d29-kube-api-access-fq5tf\") pod \"distinct-w4-1fdaa7-kserve-86dc9774b9-6x2gf\" (UID: \"2e162c95-0527-4cb3-85a3-795aae212d29\") " pod="e2e-models-e2e-worker-w4-1fdaa7/distinct-w4-1fdaa7-kserve-86dc9774b9-6x2gf" Oct 30 20:56:23.768397 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:56:23.768370 5378 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-1fdaa7/distinct-w4-1fdaa7-kserve-86dc9774b9-6x2gf" Oct 30 20:56:24.180227 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:56:24.180198 5378 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-1fdaa7/distinct-w4-1fdaa7-kserve-86dc9774b9-6x2gf"] Oct 30 20:56:24.180960 ip-10-0-138-149 kubenswrapper[5378]: W1030 20:56:24.180929 5378 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod2e162c95_0527_4cb3_85a3_795aae212d29.slice/crio-3bfb60bdda754fc1a736253ae1a6ec39cbb9a14594806ba8447c256d96e2e264 WatchSource:0}: Error finding container 3bfb60bdda754fc1a736253ae1a6ec39cbb9a14594806ba8447c256d96e2e264: Status 404 returned error can't find the container with id 3bfb60bdda754fc1a736253ae1a6ec39cbb9a14594806ba8447c256d96e2e264 Oct 30 20:56:24.899386 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:56:24.899352 5378 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-1fdaa7/distinct-w4-1fdaa7-kserve-86dc9774b9-6x2gf" event={"ID":"2e162c95-0527-4cb3-85a3-795aae212d29","Type":"ContainerStarted","Data":"1af171b9a3e007ba20860d03d532aace90ca2406d47479ca5ab185b23f57d478"} Oct 30 20:56:24.899386 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:56:24.899392 5378 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-1fdaa7/distinct-w4-1fdaa7-kserve-86dc9774b9-6x2gf" event={"ID":"2e162c95-0527-4cb3-85a3-795aae212d29","Type":"ContainerStarted","Data":"3bfb60bdda754fc1a736253ae1a6ec39cbb9a14594806ba8447c256d96e2e264"} Oct 30 20:56:25.017480 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:56:25.017430 5378 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w4-1fdaa7/distinct-w4-1fdaa7-kserve-86dc9774b9-6x2gf" podStartSLOduration=2.017416678 podStartE2EDuration="2.017416678s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:56:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:56:25.016173185 +0000 UTC m=+1075.444287128" watchObservedRunningTime="2026-10-30 20:56:25.017416678 +0000 UTC m=+1075.445530597" Oct 30 20:56:25.672367 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:56:25.672334 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 20:56:25.768788 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:56:25.768758 5378 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-1fdaa7/distinct-w4-1fdaa7-kserve-86dc9774b9-6x2gf" Oct 30 20:56:29.307067 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:56:29.307028 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 20:56:30.607811 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:56:30.607780 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 20:56:33.768965 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:56:33.768932 5378 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w4-1fdaa7/distinct-w4-1fdaa7-kserve-86dc9774b9-6x2gf" Oct 30 20:56:33.781723 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:56:33.781699 5378 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w4-1fdaa7/distinct-w4-1fdaa7-kserve-86dc9774b9-6x2gf" Oct 30 20:56:33.958770 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:56:33.958740 5378 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-1fdaa7/distinct-w4-1fdaa7-kserve-86dc9774b9-6x2gf" Oct 30 20:56:35.465027 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:56:35.464994 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 20:56:40.275810 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:56:40.275721 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 20:56:41.834925 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:56:41.834893 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 20:56:45.301257 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:56:45.301223 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 20:56:48.836575 ip-10-0-138-149 kubenswrapper[5378]: E1030 20:56:48.836523 5378 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q6mx7" containerName="extractor" Oct 30 20:56:49.357503 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:56:49.357470 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 20:56:50.159190 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:56:50.159156 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 20:56:54.972345 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:56:54.972312 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 20:56:58.427841 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:56:58.427808 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 20:56:59.137144 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:56:59.137110 5378 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-r9g59"] Oct 30 20:56:59.670363 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:56:59.670326 5378 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-f9f277-openshift-default-869d58fb77-wwkxn"] Oct 30 20:56:59.674533 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:56:59.674513 5378 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-f9f277-openshift-default-869d58fb77-wwkxn" Oct 30 20:56:59.764586 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:56:59.763490 5378 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-f9f277-openshift-default-869d58fb77-wwkxn"] Oct 30 20:56:59.785971 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:56:59.785936 5378 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3c98266e-ebfe-4c44-ae6e-079e5109bf9c-istio-data\") pod \"e2e-worker-w1-f9f277-openshift-default-869d58fb77-wwkxn\" (UID: \"3c98266e-ebfe-4c44-ae6e-079e5109bf9c\") " pod="openshift-ingress/e2e-worker-w1-f9f277-openshift-default-869d58fb77-wwkxn" Oct 30 20:56:59.786191 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:56:59.786166 5378 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3c98266e-ebfe-4c44-ae6e-079e5109bf9c-workload-socket\") pod \"e2e-worker-w1-f9f277-openshift-default-869d58fb77-wwkxn\" (UID: \"3c98266e-ebfe-4c44-ae6e-079e5109bf9c\") " pod="openshift-ingress/e2e-worker-w1-f9f277-openshift-default-869d58fb77-wwkxn" Oct 30 20:56:59.786276 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:56:59.786205 5378 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3c98266e-ebfe-4c44-ae6e-079e5109bf9c-istio-envoy\") pod \"e2e-worker-w1-f9f277-openshift-default-869d58fb77-wwkxn\" (UID: \"3c98266e-ebfe-4c44-ae6e-079e5109bf9c\") " pod="openshift-ingress/e2e-worker-w1-f9f277-openshift-default-869d58fb77-wwkxn" Oct 30 20:56:59.786412 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:56:59.786395 5378 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3c98266e-ebfe-4c44-ae6e-079e5109bf9c-istio-podinfo\") pod \"e2e-worker-w1-f9f277-openshift-default-869d58fb77-wwkxn\" (UID: \"3c98266e-ebfe-4c44-ae6e-079e5109bf9c\") " pod="openshift-ingress/e2e-worker-w1-f9f277-openshift-default-869d58fb77-wwkxn" Oct 30 20:56:59.786742 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:56:59.786704 5378 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3c98266e-ebfe-4c44-ae6e-079e5109bf9c-istiod-ca-cert\") pod \"e2e-worker-w1-f9f277-openshift-default-869d58fb77-wwkxn\" (UID: \"3c98266e-ebfe-4c44-ae6e-079e5109bf9c\") " pod="openshift-ingress/e2e-worker-w1-f9f277-openshift-default-869d58fb77-wwkxn" Oct 30 20:56:59.786911 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:56:59.786773 5378 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3c98266e-ebfe-4c44-ae6e-079e5109bf9c-workload-certs\") pod \"e2e-worker-w1-f9f277-openshift-default-869d58fb77-wwkxn\" (UID: \"3c98266e-ebfe-4c44-ae6e-079e5109bf9c\") " pod="openshift-ingress/e2e-worker-w1-f9f277-openshift-default-869d58fb77-wwkxn" Oct 30 20:56:59.787042 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:56:59.786961 5378 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3c98266e-ebfe-4c44-ae6e-079e5109bf9c-istio-token\") pod \"e2e-worker-w1-f9f277-openshift-default-869d58fb77-wwkxn\" (UID: \"3c98266e-ebfe-4c44-ae6e-079e5109bf9c\") " pod="openshift-ingress/e2e-worker-w1-f9f277-openshift-default-869d58fb77-wwkxn" Oct 30 20:56:59.787042 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:56:59.787016 5378 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3c98266e-ebfe-4c44-ae6e-079e5109bf9c-credential-socket\") pod \"e2e-worker-w1-f9f277-openshift-default-869d58fb77-wwkxn\" (UID: \"3c98266e-ebfe-4c44-ae6e-079e5109bf9c\") " pod="openshift-ingress/e2e-worker-w1-f9f277-openshift-default-869d58fb77-wwkxn" Oct 30 20:56:59.787195 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:56:59.787115 5378 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v9nct\" (UniqueName: \"kubernetes.io/projected/3c98266e-ebfe-4c44-ae6e-079e5109bf9c-kube-api-access-v9nct\") pod \"e2e-worker-w1-f9f277-openshift-default-869d58fb77-wwkxn\" (UID: \"3c98266e-ebfe-4c44-ae6e-079e5109bf9c\") " pod="openshift-ingress/e2e-worker-w1-f9f277-openshift-default-869d58fb77-wwkxn" Oct 30 20:56:59.869772 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:56:59.869734 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 20:56:59.888116 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:56:59.888067 5378 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3c98266e-ebfe-4c44-ae6e-079e5109bf9c-istio-podinfo\") pod \"e2e-worker-w1-f9f277-openshift-default-869d58fb77-wwkxn\" (UID: \"3c98266e-ebfe-4c44-ae6e-079e5109bf9c\") " pod="openshift-ingress/e2e-worker-w1-f9f277-openshift-default-869d58fb77-wwkxn" Oct 30 20:56:59.888291 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:56:59.888166 5378 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3c98266e-ebfe-4c44-ae6e-079e5109bf9c-istiod-ca-cert\") pod \"e2e-worker-w1-f9f277-openshift-default-869d58fb77-wwkxn\" (UID: \"3c98266e-ebfe-4c44-ae6e-079e5109bf9c\") " pod="openshift-ingress/e2e-worker-w1-f9f277-openshift-default-869d58fb77-wwkxn" Oct 30 20:56:59.888291 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:56:59.888206 5378 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3c98266e-ebfe-4c44-ae6e-079e5109bf9c-workload-certs\") pod \"e2e-worker-w1-f9f277-openshift-default-869d58fb77-wwkxn\" (UID: \"3c98266e-ebfe-4c44-ae6e-079e5109bf9c\") " pod="openshift-ingress/e2e-worker-w1-f9f277-openshift-default-869d58fb77-wwkxn" Oct 30 20:56:59.888291 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:56:59.888254 5378 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3c98266e-ebfe-4c44-ae6e-079e5109bf9c-istio-token\") pod \"e2e-worker-w1-f9f277-openshift-default-869d58fb77-wwkxn\" (UID: \"3c98266e-ebfe-4c44-ae6e-079e5109bf9c\") " pod="openshift-ingress/e2e-worker-w1-f9f277-openshift-default-869d58fb77-wwkxn" Oct 30 20:56:59.888479 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:56:59.888295 5378 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3c98266e-ebfe-4c44-ae6e-079e5109bf9c-credential-socket\") pod \"e2e-worker-w1-f9f277-openshift-default-869d58fb77-wwkxn\" (UID: \"3c98266e-ebfe-4c44-ae6e-079e5109bf9c\") " pod="openshift-ingress/e2e-worker-w1-f9f277-openshift-default-869d58fb77-wwkxn" Oct 30 20:56:59.888479 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:56:59.888355 5378 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v9nct\" (UniqueName: \"kubernetes.io/projected/3c98266e-ebfe-4c44-ae6e-079e5109bf9c-kube-api-access-v9nct\") pod \"e2e-worker-w1-f9f277-openshift-default-869d58fb77-wwkxn\" (UID: \"3c98266e-ebfe-4c44-ae6e-079e5109bf9c\") " pod="openshift-ingress/e2e-worker-w1-f9f277-openshift-default-869d58fb77-wwkxn" Oct 30 20:56:59.888479 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:56:59.888412 5378 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3c98266e-ebfe-4c44-ae6e-079e5109bf9c-istio-data\") pod \"e2e-worker-w1-f9f277-openshift-default-869d58fb77-wwkxn\" (UID: \"3c98266e-ebfe-4c44-ae6e-079e5109bf9c\") " pod="openshift-ingress/e2e-worker-w1-f9f277-openshift-default-869d58fb77-wwkxn" Oct 30 20:56:59.888479 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:56:59.888444 5378 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3c98266e-ebfe-4c44-ae6e-079e5109bf9c-workload-socket\") pod \"e2e-worker-w1-f9f277-openshift-default-869d58fb77-wwkxn\" (UID: \"3c98266e-ebfe-4c44-ae6e-079e5109bf9c\") " pod="openshift-ingress/e2e-worker-w1-f9f277-openshift-default-869d58fb77-wwkxn" Oct 30 20:56:59.888479 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:56:59.888471 5378 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3c98266e-ebfe-4c44-ae6e-079e5109bf9c-istio-envoy\") pod \"e2e-worker-w1-f9f277-openshift-default-869d58fb77-wwkxn\" (UID: \"3c98266e-ebfe-4c44-ae6e-079e5109bf9c\") " pod="openshift-ingress/e2e-worker-w1-f9f277-openshift-default-869d58fb77-wwkxn" Oct 30 20:56:59.888772 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:56:59.888642 5378 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3c98266e-ebfe-4c44-ae6e-079e5109bf9c-workload-certs\") pod \"e2e-worker-w1-f9f277-openshift-default-869d58fb77-wwkxn\" (UID: \"3c98266e-ebfe-4c44-ae6e-079e5109bf9c\") " pod="openshift-ingress/e2e-worker-w1-f9f277-openshift-default-869d58fb77-wwkxn" Oct 30 20:56:59.889252 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:56:59.889032 5378 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3c98266e-ebfe-4c44-ae6e-079e5109bf9c-credential-socket\") pod \"e2e-worker-w1-f9f277-openshift-default-869d58fb77-wwkxn\" (UID: \"3c98266e-ebfe-4c44-ae6e-079e5109bf9c\") " pod="openshift-ingress/e2e-worker-w1-f9f277-openshift-default-869d58fb77-wwkxn" Oct 30 20:56:59.889252 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:56:59.889110 5378 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3c98266e-ebfe-4c44-ae6e-079e5109bf9c-istiod-ca-cert\") pod \"e2e-worker-w1-f9f277-openshift-default-869d58fb77-wwkxn\" (UID: \"3c98266e-ebfe-4c44-ae6e-079e5109bf9c\") " pod="openshift-ingress/e2e-worker-w1-f9f277-openshift-default-869d58fb77-wwkxn" Oct 30 20:56:59.889252 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:56:59.889206 5378 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3c98266e-ebfe-4c44-ae6e-079e5109bf9c-istio-data\") pod \"e2e-worker-w1-f9f277-openshift-default-869d58fb77-wwkxn\" (UID: \"3c98266e-ebfe-4c44-ae6e-079e5109bf9c\") " pod="openshift-ingress/e2e-worker-w1-f9f277-openshift-default-869d58fb77-wwkxn" Oct 30 20:56:59.889477 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:56:59.889289 5378 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3c98266e-ebfe-4c44-ae6e-079e5109bf9c-workload-socket\") pod \"e2e-worker-w1-f9f277-openshift-default-869d58fb77-wwkxn\" (UID: \"3c98266e-ebfe-4c44-ae6e-079e5109bf9c\") " pod="openshift-ingress/e2e-worker-w1-f9f277-openshift-default-869d58fb77-wwkxn" Oct 30 20:56:59.890968 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:56:59.890945 5378 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3c98266e-ebfe-4c44-ae6e-079e5109bf9c-istio-envoy\") pod \"e2e-worker-w1-f9f277-openshift-default-869d58fb77-wwkxn\" (UID: \"3c98266e-ebfe-4c44-ae6e-079e5109bf9c\") " pod="openshift-ingress/e2e-worker-w1-f9f277-openshift-default-869d58fb77-wwkxn" Oct 30 20:56:59.891931 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:56:59.891912 5378 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3c98266e-ebfe-4c44-ae6e-079e5109bf9c-istio-podinfo\") pod \"e2e-worker-w1-f9f277-openshift-default-869d58fb77-wwkxn\" (UID: \"3c98266e-ebfe-4c44-ae6e-079e5109bf9c\") " pod="openshift-ingress/e2e-worker-w1-f9f277-openshift-default-869d58fb77-wwkxn" Oct 30 20:56:59.965414 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:56:59.965375 5378 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v9nct\" (UniqueName: \"kubernetes.io/projected/3c98266e-ebfe-4c44-ae6e-079e5109bf9c-kube-api-access-v9nct\") pod \"e2e-worker-w1-f9f277-openshift-default-869d58fb77-wwkxn\" (UID: \"3c98266e-ebfe-4c44-ae6e-079e5109bf9c\") " pod="openshift-ingress/e2e-worker-w1-f9f277-openshift-default-869d58fb77-wwkxn" Oct 30 20:56:59.977590 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:56:59.973712 5378 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3c98266e-ebfe-4c44-ae6e-079e5109bf9c-istio-token\") pod \"e2e-worker-w1-f9f277-openshift-default-869d58fb77-wwkxn\" (UID: \"3c98266e-ebfe-4c44-ae6e-079e5109bf9c\") " pod="openshift-ingress/e2e-worker-w1-f9f277-openshift-default-869d58fb77-wwkxn" Oct 30 20:56:59.989815 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:56:59.989787 5378 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-f9f277-openshift-default-869d58fb77-wwkxn" Oct 30 20:57:00.508315 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:57:00.508268 5378 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 20:57:00.508441 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:57:00.508362 5378 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 20:57:00.508441 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:57:00.508415 5378 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 20:57:00.510925 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:57:00.510880 5378 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-f9f277-openshift-default-869d58fb77-wwkxn"] Oct 30 20:57:01.119635 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:57:01.119592 5378 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-f9f277-openshift-default-869d58fb77-wwkxn" event={"ID":"3c98266e-ebfe-4c44-ae6e-079e5109bf9c","Type":"ContainerStarted","Data":"5b9e127f42ee4f6169792ae410b55f7f06a5197c2448b74af8fc3d46d48ceed2"} Oct 30 20:57:01.120038 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:57:01.119649 5378 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-f9f277-openshift-default-869d58fb77-wwkxn" event={"ID":"3c98266e-ebfe-4c44-ae6e-079e5109bf9c","Type":"ContainerStarted","Data":"e0afb729e20be762809cc3a816360d11af514657d991c81038b1447fc0e13870"} Oct 30 20:57:01.247582 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:57:01.247499 5378 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-f9f277-openshift-default-869d58fb77-wwkxn" podStartSLOduration=2.247486483 podStartE2EDuration="2.247486483s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:56:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:57:01.245424928 +0000 UTC m=+1111.673538884" watchObservedRunningTime="2026-10-30 20:57:01.247486483 +0000 UTC m=+1111.675600402" Oct 30 20:57:01.990628 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:57:01.990598 5378 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-f9f277-openshift-default-869d58fb77-wwkxn" Oct 30 20:57:01.992272 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:57:01.992248 5378 patch_prober.go:29] interesting pod/e2e-worker-w1-f9f277-openshift-default-869d58fb77-wwkxn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused" start-of-body= Oct 30 20:57:01.992387 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:57:01.992294 5378 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-f9f277-openshift-default-869d58fb77-wwkxn" podUID="3c98266e-ebfe-4c44-ae6e-079e5109bf9c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused" Oct 30 20:57:02.990058 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:57:02.990025 5378 patch_prober.go:29] interesting pod/e2e-worker-w1-f9f277-openshift-default-869d58fb77-wwkxn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused" start-of-body= Oct 30 20:57:02.990458 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:57:02.990087 5378 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-f9f277-openshift-default-869d58fb77-wwkxn" podUID="3c98266e-ebfe-4c44-ae6e-079e5109bf9c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused" Oct 30 20:57:03.991070 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:57:03.991035 5378 patch_prober.go:29] interesting pod/e2e-worker-w1-f9f277-openshift-default-869d58fb77-wwkxn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused" start-of-body= Oct 30 20:57:03.991519 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:57:03.991097 5378 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-f9f277-openshift-default-869d58fb77-wwkxn" podUID="3c98266e-ebfe-4c44-ae6e-079e5109bf9c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused" Oct 30 20:57:04.766589 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:57:04.762416 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 20:57:04.990060 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:57:04.990027 5378 patch_prober.go:29] interesting pod/e2e-worker-w1-f9f277-openshift-default-869d58fb77-wwkxn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused" start-of-body= Oct 30 20:57:04.990236 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:57:04.990090 5378 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-f9f277-openshift-default-869d58fb77-wwkxn" podUID="3c98266e-ebfe-4c44-ae6e-079e5109bf9c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused" Oct 30 20:57:05.990638 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:57:05.990605 5378 patch_prober.go:29] interesting pod/e2e-worker-w1-f9f277-openshift-default-869d58fb77-wwkxn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused" start-of-body= Oct 30 20:57:05.991081 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:57:05.990666 5378 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-f9f277-openshift-default-869d58fb77-wwkxn" podUID="3c98266e-ebfe-4c44-ae6e-079e5109bf9c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused" Oct 30 20:57:06.990898 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:57:06.990863 5378 patch_prober.go:29] interesting pod/e2e-worker-w1-f9f277-openshift-default-869d58fb77-wwkxn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused" start-of-body= Oct 30 20:57:06.991308 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:57:06.990940 5378 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-f9f277-openshift-default-869d58fb77-wwkxn" podUID="3c98266e-ebfe-4c44-ae6e-079e5109bf9c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused" Oct 30 20:57:07.995063 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:57:07.995032 5378 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-f9f277-openshift-default-869d58fb77-wwkxn" Oct 30 20:57:07.995469 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:57:07.995326 5378 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-f9f277-openshift-default-869d58fb77-wwkxn" Oct 30 20:57:07.996097 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:57:07.996079 5378 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-f9f277-openshift-default-869d58fb77-wwkxn" Oct 30 20:57:09.571786 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:57:09.571745 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 20:57:14.371778 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:57:14.371720 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 20:57:15.530822 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:57:15.530787 5378 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-682999-openshift-default-54dcb797b5-zhslb"] Oct 30 20:57:15.534918 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:57:15.534893 5378 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-682999-openshift-default-54dcb797b5-zhslb" Oct 30 20:57:15.595185 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:57:15.595157 5378 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/afdb629d-0850-4d80-a6b9-3556bd0d706e-workload-socket\") pod \"e2e-worker-w3-682999-openshift-default-54dcb797b5-zhslb\" (UID: \"afdb629d-0850-4d80-a6b9-3556bd0d706e\") " pod="openshift-ingress/e2e-worker-w3-682999-openshift-default-54dcb797b5-zhslb" Oct 30 20:57:15.595331 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:57:15.595194 5378 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zngmr\" (UniqueName: \"kubernetes.io/projected/afdb629d-0850-4d80-a6b9-3556bd0d706e-kube-api-access-zngmr\") pod \"e2e-worker-w3-682999-openshift-default-54dcb797b5-zhslb\" (UID: \"afdb629d-0850-4d80-a6b9-3556bd0d706e\") " pod="openshift-ingress/e2e-worker-w3-682999-openshift-default-54dcb797b5-zhslb" Oct 30 20:57:15.595331 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:57:15.595220 5378 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/afdb629d-0850-4d80-a6b9-3556bd0d706e-workload-certs\") pod \"e2e-worker-w3-682999-openshift-default-54dcb797b5-zhslb\" (UID: \"afdb629d-0850-4d80-a6b9-3556bd0d706e\") " pod="openshift-ingress/e2e-worker-w3-682999-openshift-default-54dcb797b5-zhslb" Oct 30 20:57:15.595331 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:57:15.595254 5378 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/afdb629d-0850-4d80-a6b9-3556bd0d706e-istio-data\") pod \"e2e-worker-w3-682999-openshift-default-54dcb797b5-zhslb\" (UID: \"afdb629d-0850-4d80-a6b9-3556bd0d706e\") " pod="openshift-ingress/e2e-worker-w3-682999-openshift-default-54dcb797b5-zhslb" Oct 30 20:57:15.595454 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:57:15.595406 5378 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/afdb629d-0850-4d80-a6b9-3556bd0d706e-istio-podinfo\") pod \"e2e-worker-w3-682999-openshift-default-54dcb797b5-zhslb\" (UID: \"afdb629d-0850-4d80-a6b9-3556bd0d706e\") " pod="openshift-ingress/e2e-worker-w3-682999-openshift-default-54dcb797b5-zhslb" Oct 30 20:57:15.595454 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:57:15.595445 5378 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/afdb629d-0850-4d80-a6b9-3556bd0d706e-istio-token\") pod \"e2e-worker-w3-682999-openshift-default-54dcb797b5-zhslb\" (UID: \"afdb629d-0850-4d80-a6b9-3556bd0d706e\") " pod="openshift-ingress/e2e-worker-w3-682999-openshift-default-54dcb797b5-zhslb" Oct 30 20:57:15.595540 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:57:15.595522 5378 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/afdb629d-0850-4d80-a6b9-3556bd0d706e-istio-envoy\") pod \"e2e-worker-w3-682999-openshift-default-54dcb797b5-zhslb\" (UID: \"afdb629d-0850-4d80-a6b9-3556bd0d706e\") " pod="openshift-ingress/e2e-worker-w3-682999-openshift-default-54dcb797b5-zhslb" Oct 30 20:57:15.595614 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:57:15.595543 5378 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/afdb629d-0850-4d80-a6b9-3556bd0d706e-credential-socket\") pod \"e2e-worker-w3-682999-openshift-default-54dcb797b5-zhslb\" (UID: \"afdb629d-0850-4d80-a6b9-3556bd0d706e\") " pod="openshift-ingress/e2e-worker-w3-682999-openshift-default-54dcb797b5-zhslb" Oct 30 20:57:15.595695 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:57:15.595669 5378 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/afdb629d-0850-4d80-a6b9-3556bd0d706e-istiod-ca-cert\") pod \"e2e-worker-w3-682999-openshift-default-54dcb797b5-zhslb\" (UID: \"afdb629d-0850-4d80-a6b9-3556bd0d706e\") " pod="openshift-ingress/e2e-worker-w3-682999-openshift-default-54dcb797b5-zhslb" Oct 30 20:57:15.629887 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:57:15.629821 5378 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-682999-openshift-default-54dcb797b5-zhslb"] Oct 30 20:57:15.697165 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:57:15.697134 5378 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/afdb629d-0850-4d80-a6b9-3556bd0d706e-istiod-ca-cert\") pod \"e2e-worker-w3-682999-openshift-default-54dcb797b5-zhslb\" (UID: \"afdb629d-0850-4d80-a6b9-3556bd0d706e\") " pod="openshift-ingress/e2e-worker-w3-682999-openshift-default-54dcb797b5-zhslb" Oct 30 20:57:15.697342 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:57:15.697192 5378 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/afdb629d-0850-4d80-a6b9-3556bd0d706e-workload-socket\") pod \"e2e-worker-w3-682999-openshift-default-54dcb797b5-zhslb\" (UID: \"afdb629d-0850-4d80-a6b9-3556bd0d706e\") " pod="openshift-ingress/e2e-worker-w3-682999-openshift-default-54dcb797b5-zhslb" Oct 30 20:57:15.697342 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:57:15.697223 5378 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zngmr\" (UniqueName: \"kubernetes.io/projected/afdb629d-0850-4d80-a6b9-3556bd0d706e-kube-api-access-zngmr\") pod \"e2e-worker-w3-682999-openshift-default-54dcb797b5-zhslb\" (UID: \"afdb629d-0850-4d80-a6b9-3556bd0d706e\") " pod="openshift-ingress/e2e-worker-w3-682999-openshift-default-54dcb797b5-zhslb" Oct 30 20:57:15.697342 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:57:15.697250 5378 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/afdb629d-0850-4d80-a6b9-3556bd0d706e-workload-certs\") pod \"e2e-worker-w3-682999-openshift-default-54dcb797b5-zhslb\" (UID: \"afdb629d-0850-4d80-a6b9-3556bd0d706e\") " pod="openshift-ingress/e2e-worker-w3-682999-openshift-default-54dcb797b5-zhslb" Oct 30 20:57:15.697342 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:57:15.697268 5378 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/afdb629d-0850-4d80-a6b9-3556bd0d706e-istio-data\") pod \"e2e-worker-w3-682999-openshift-default-54dcb797b5-zhslb\" (UID: \"afdb629d-0850-4d80-a6b9-3556bd0d706e\") " pod="openshift-ingress/e2e-worker-w3-682999-openshift-default-54dcb797b5-zhslb" Oct 30 20:57:15.697495 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:57:15.697341 5378 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/afdb629d-0850-4d80-a6b9-3556bd0d706e-istio-podinfo\") pod \"e2e-worker-w3-682999-openshift-default-54dcb797b5-zhslb\" (UID: \"afdb629d-0850-4d80-a6b9-3556bd0d706e\") " pod="openshift-ingress/e2e-worker-w3-682999-openshift-default-54dcb797b5-zhslb" Oct 30 20:57:15.697495 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:57:15.697398 5378 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/afdb629d-0850-4d80-a6b9-3556bd0d706e-istio-token\") pod \"e2e-worker-w3-682999-openshift-default-54dcb797b5-zhslb\" (UID: \"afdb629d-0850-4d80-a6b9-3556bd0d706e\") " pod="openshift-ingress/e2e-worker-w3-682999-openshift-default-54dcb797b5-zhslb" Oct 30 20:57:15.697495 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:57:15.697462 5378 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/afdb629d-0850-4d80-a6b9-3556bd0d706e-istio-envoy\") pod \"e2e-worker-w3-682999-openshift-default-54dcb797b5-zhslb\" (UID: \"afdb629d-0850-4d80-a6b9-3556bd0d706e\") " pod="openshift-ingress/e2e-worker-w3-682999-openshift-default-54dcb797b5-zhslb" Oct 30 20:57:15.697495 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:57:15.697491 5378 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/afdb629d-0850-4d80-a6b9-3556bd0d706e-credential-socket\") pod \"e2e-worker-w3-682999-openshift-default-54dcb797b5-zhslb\" (UID: \"afdb629d-0850-4d80-a6b9-3556bd0d706e\") " pod="openshift-ingress/e2e-worker-w3-682999-openshift-default-54dcb797b5-zhslb" Oct 30 20:57:15.697696 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:57:15.697675 5378 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/afdb629d-0850-4d80-a6b9-3556bd0d706e-workload-socket\") pod \"e2e-worker-w3-682999-openshift-default-54dcb797b5-zhslb\" (UID: \"afdb629d-0850-4d80-a6b9-3556bd0d706e\") " pod="openshift-ingress/e2e-worker-w3-682999-openshift-default-54dcb797b5-zhslb" Oct 30 20:57:15.697966 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:57:15.697936 5378 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/afdb629d-0850-4d80-a6b9-3556bd0d706e-credential-socket\") pod \"e2e-worker-w3-682999-openshift-default-54dcb797b5-zhslb\" (UID: \"afdb629d-0850-4d80-a6b9-3556bd0d706e\") " pod="openshift-ingress/e2e-worker-w3-682999-openshift-default-54dcb797b5-zhslb" Oct 30 20:57:15.698107 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:57:15.697985 5378 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/afdb629d-0850-4d80-a6b9-3556bd0d706e-workload-certs\") pod \"e2e-worker-w3-682999-openshift-default-54dcb797b5-zhslb\" (UID: \"afdb629d-0850-4d80-a6b9-3556bd0d706e\") " pod="openshift-ingress/e2e-worker-w3-682999-openshift-default-54dcb797b5-zhslb" Oct 30 20:57:15.698107 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:57:15.698004 5378 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/afdb629d-0850-4d80-a6b9-3556bd0d706e-istiod-ca-cert\") pod \"e2e-worker-w3-682999-openshift-default-54dcb797b5-zhslb\" (UID: \"afdb629d-0850-4d80-a6b9-3556bd0d706e\") " pod="openshift-ingress/e2e-worker-w3-682999-openshift-default-54dcb797b5-zhslb" Oct 30 20:57:15.698107 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:57:15.697993 5378 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/afdb629d-0850-4d80-a6b9-3556bd0d706e-istio-data\") pod \"e2e-worker-w3-682999-openshift-default-54dcb797b5-zhslb\" (UID: \"afdb629d-0850-4d80-a6b9-3556bd0d706e\") " pod="openshift-ingress/e2e-worker-w3-682999-openshift-default-54dcb797b5-zhslb" Oct 30 20:57:15.699838 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:57:15.699817 5378 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/afdb629d-0850-4d80-a6b9-3556bd0d706e-istio-envoy\") pod \"e2e-worker-w3-682999-openshift-default-54dcb797b5-zhslb\" (UID: \"afdb629d-0850-4d80-a6b9-3556bd0d706e\") " pod="openshift-ingress/e2e-worker-w3-682999-openshift-default-54dcb797b5-zhslb" Oct 30 20:57:15.699960 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:57:15.699937 5378 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/afdb629d-0850-4d80-a6b9-3556bd0d706e-istio-podinfo\") pod \"e2e-worker-w3-682999-openshift-default-54dcb797b5-zhslb\" (UID: \"afdb629d-0850-4d80-a6b9-3556bd0d706e\") " pod="openshift-ingress/e2e-worker-w3-682999-openshift-default-54dcb797b5-zhslb" Oct 30 20:57:15.755175 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:57:15.755150 5378 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/afdb629d-0850-4d80-a6b9-3556bd0d706e-istio-token\") pod \"e2e-worker-w3-682999-openshift-default-54dcb797b5-zhslb\" (UID: \"afdb629d-0850-4d80-a6b9-3556bd0d706e\") " pod="openshift-ingress/e2e-worker-w3-682999-openshift-default-54dcb797b5-zhslb" Oct 30 20:57:15.755341 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:57:15.755321 5378 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zngmr\" (UniqueName: \"kubernetes.io/projected/afdb629d-0850-4d80-a6b9-3556bd0d706e-kube-api-access-zngmr\") pod \"e2e-worker-w3-682999-openshift-default-54dcb797b5-zhslb\" (UID: \"afdb629d-0850-4d80-a6b9-3556bd0d706e\") " pod="openshift-ingress/e2e-worker-w3-682999-openshift-default-54dcb797b5-zhslb" Oct 30 20:57:15.850673 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:57:15.850644 5378 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-682999-openshift-default-54dcb797b5-zhslb" Oct 30 20:57:16.278811 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:57:16.278781 5378 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-682999-openshift-default-54dcb797b5-zhslb"] Oct 30 20:57:16.279028 ip-10-0-138-149 kubenswrapper[5378]: W1030 20:57:16.278999 5378 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podafdb629d_0850_4d80_a6b9_3556bd0d706e.slice/crio-7894447d60effb1a8aed8bdef86167c1fdb9ef27d8619a9bf5699c67e59ceb3c WatchSource:0}: Error finding container 7894447d60effb1a8aed8bdef86167c1fdb9ef27d8619a9bf5699c67e59ceb3c: Status 404 returned error can't find the container with id 7894447d60effb1a8aed8bdef86167c1fdb9ef27d8619a9bf5699c67e59ceb3c Oct 30 20:57:16.281379 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:57:16.281348 5378 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 20:57:16.281443 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:57:16.281414 5378 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 20:57:16.281479 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:57:16.281444 5378 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 20:57:17.214234 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:57:17.214195 5378 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-682999-openshift-default-54dcb797b5-zhslb" event={"ID":"afdb629d-0850-4d80-a6b9-3556bd0d706e","Type":"ContainerStarted","Data":"1b8acf4813b04532a32225114288d63035b23d2beca277eb36e69cec3a33c818"} Oct 30 20:57:17.214716 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:57:17.214249 5378 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-682999-openshift-default-54dcb797b5-zhslb" event={"ID":"afdb629d-0850-4d80-a6b9-3556bd0d706e","Type":"ContainerStarted","Data":"7894447d60effb1a8aed8bdef86167c1fdb9ef27d8619a9bf5699c67e59ceb3c"} Oct 30 20:57:17.436167 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:57:17.436110 5378 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-682999-openshift-default-54dcb797b5-zhslb" podStartSLOduration=2.436096071 podStartE2EDuration="2.436096071s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:57:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:57:17.384557638 +0000 UTC m=+1127.812671558" watchObservedRunningTime="2026-10-30 20:57:17.436096071 +0000 UTC m=+1127.864209989" Oct 30 20:57:17.851408 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:57:17.851376 5378 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-682999-openshift-default-54dcb797b5-zhslb" Oct 30 20:57:17.852598 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:57:17.852553 5378 patch_prober.go:29] interesting pod/e2e-worker-w3-682999-openshift-default-54dcb797b5-zhslb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused" start-of-body= Oct 30 20:57:17.852821 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:57:17.852789 5378 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-682999-openshift-default-54dcb797b5-zhslb" podUID="afdb629d-0850-4d80-a6b9-3556bd0d706e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused" Oct 30 20:57:18.836934 ip-10-0-138-149 kubenswrapper[5378]: E1030 20:57:18.836899 5378 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q6mx7" containerName="extractor" Oct 30 20:57:18.851141 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:57:18.851110 5378 patch_prober.go:29] interesting pod/e2e-worker-w3-682999-openshift-default-54dcb797b5-zhslb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused" start-of-body= Oct 30 20:57:18.851267 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:57:18.851166 5378 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-682999-openshift-default-54dcb797b5-zhslb" podUID="afdb629d-0850-4d80-a6b9-3556bd0d706e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused" Oct 30 20:57:19.215469 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:57:19.215437 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 20:57:19.851296 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:57:19.851264 5378 patch_prober.go:29] interesting pod/e2e-worker-w3-682999-openshift-default-54dcb797b5-zhslb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused" start-of-body= Oct 30 20:57:19.851701 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:57:19.851318 5378 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-682999-openshift-default-54dcb797b5-zhslb" podUID="afdb629d-0850-4d80-a6b9-3556bd0d706e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused" Oct 30 20:57:20.851895 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:57:20.851862 5378 patch_prober.go:29] interesting pod/e2e-worker-w3-682999-openshift-default-54dcb797b5-zhslb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused" start-of-body= Oct 30 20:57:20.852291 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:57:20.851939 5378 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-682999-openshift-default-54dcb797b5-zhslb" podUID="afdb629d-0850-4d80-a6b9-3556bd0d706e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused" Oct 30 20:57:21.851742 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:57:21.851705 5378 patch_prober.go:29] interesting pod/e2e-worker-w3-682999-openshift-default-54dcb797b5-zhslb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused" start-of-body= Oct 30 20:57:21.851940 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:57:21.851761 5378 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-682999-openshift-default-54dcb797b5-zhslb" podUID="afdb629d-0850-4d80-a6b9-3556bd0d706e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused" Oct 30 20:57:22.851421 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:57:22.851381 5378 patch_prober.go:29] interesting pod/e2e-worker-w3-682999-openshift-default-54dcb797b5-zhslb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused" start-of-body= Oct 30 20:57:22.851607 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:57:22.851458 5378 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-682999-openshift-default-54dcb797b5-zhslb" podUID="afdb629d-0850-4d80-a6b9-3556bd0d706e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused" Oct 30 20:57:23.856053 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:57:23.856021 5378 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-682999-openshift-default-54dcb797b5-zhslb" Oct 30 20:57:23.856417 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:57:23.856374 5378 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-682999-openshift-default-54dcb797b5-zhslb" Oct 30 20:57:23.856977 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:57:23.856959 5378 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-682999-openshift-default-54dcb797b5-zhslb" Oct 30 20:57:24.106654 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:57:24.106529 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 20:57:28.961388 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:57:28.961356 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 20:57:33.773233 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:57:33.773195 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 20:57:36.398965 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:57:36.398930 5378 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-r9g59"] Oct 30 20:57:38.665960 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:57:38.665927 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 20:57:42.201720 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:57:42.201662 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 20:57:43.462341 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:57:43.462298 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 20:57:48.342294 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:57:48.342262 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 20:57:48.837122 ip-10-0-138-149 kubenswrapper[5378]: E1030 20:57:48.837086 5378 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q6mx7" containerName="extractor" Oct 30 20:57:53.252022 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:57:53.251990 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 20:57:54.317891 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:57:54.317859 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 20:57:58.159091 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:57:58.159056 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 20:58:02.970391 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:58:02.970360 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 20:58:05.088758 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:58:05.088730 5378 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-r9g59"] Oct 30 20:58:07.874697 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:58:07.874654 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 20:58:12.673223 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:58:12.673143 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 20:58:17.599746 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:58:17.599712 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 20:58:18.837077 ip-10-0-138-149 kubenswrapper[5378]: E1030 20:58:18.837040 5378 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q6mx7" containerName="extractor" Oct 30 20:58:19.255843 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:58:19.255794 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 20:58:22.924981 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:58:22.922524 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 20:58:27.734303 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:58:27.734266 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 20:58:29.861994 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:58:29.861965 5378 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-7x9jg_c2ff359c-cf92-4a5b-8610-f6eb41acbe73/ovn-acl-logging/0.log" Oct 30 20:58:29.865352 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:58:29.865325 5378 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-7x9jg_c2ff359c-cf92-4a5b-8610-f6eb41acbe73/ovn-acl-logging/0.log" Oct 30 20:58:32.341126 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:58:32.341094 5378 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-781058-openshift-default-55fb495b7-5zx7k"] Oct 30 20:58:32.345025 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:58:32.345007 5378 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-781058-openshift-default-55fb495b7-5zx7k" Oct 30 20:58:32.391125 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:58:32.391099 5378 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-781058-openshift-default-55fb495b7-khvpl"] Oct 30 20:58:32.394842 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:58:32.394826 5378 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-781058-openshift-default-55fb495b7-khvpl" Oct 30 20:58:32.405703 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:58:32.405680 5378 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w0-781058-openshift-default-dockercfg-mkvr5\"" Oct 30 20:58:32.422034 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:58:32.422010 5378 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/48a0901f-8b2b-4278-8422-226bb6bc6dd3-istio-podinfo\") pod \"e2e-worker-w0-781058-openshift-default-55fb495b7-5zx7k\" (UID: \"48a0901f-8b2b-4278-8422-226bb6bc6dd3\") " pod="openshift-ingress/e2e-worker-w0-781058-openshift-default-55fb495b7-5zx7k" Oct 30 20:58:32.422132 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:58:32.422040 5378 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/48a0901f-8b2b-4278-8422-226bb6bc6dd3-istio-envoy\") pod \"e2e-worker-w0-781058-openshift-default-55fb495b7-5zx7k\" (UID: \"48a0901f-8b2b-4278-8422-226bb6bc6dd3\") " pod="openshift-ingress/e2e-worker-w0-781058-openshift-default-55fb495b7-5zx7k" Oct 30 20:58:32.422132 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:58:32.422059 5378 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/afcb5db1-70aa-4758-90e6-e21b112d48f4-credential-socket\") pod \"e2e-worker-w0-781058-openshift-default-55fb495b7-khvpl\" (UID: \"afcb5db1-70aa-4758-90e6-e21b112d48f4\") " pod="openshift-ingress/e2e-worker-w0-781058-openshift-default-55fb495b7-khvpl" Oct 30 20:58:32.422208 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:58:32.422134 5378 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/48a0901f-8b2b-4278-8422-226bb6bc6dd3-istiod-ca-cert\") pod \"e2e-worker-w0-781058-openshift-default-55fb495b7-5zx7k\" (UID: \"48a0901f-8b2b-4278-8422-226bb6bc6dd3\") " pod="openshift-ingress/e2e-worker-w0-781058-openshift-default-55fb495b7-5zx7k" Oct 30 20:58:32.422208 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:58:32.422159 5378 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/48a0901f-8b2b-4278-8422-226bb6bc6dd3-credential-socket\") pod \"e2e-worker-w0-781058-openshift-default-55fb495b7-5zx7k\" (UID: \"48a0901f-8b2b-4278-8422-226bb6bc6dd3\") " pod="openshift-ingress/e2e-worker-w0-781058-openshift-default-55fb495b7-5zx7k" Oct 30 20:58:32.422208 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:58:32.422176 5378 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/afcb5db1-70aa-4758-90e6-e21b112d48f4-istio-envoy\") pod \"e2e-worker-w0-781058-openshift-default-55fb495b7-khvpl\" (UID: \"afcb5db1-70aa-4758-90e6-e21b112d48f4\") " pod="openshift-ingress/e2e-worker-w0-781058-openshift-default-55fb495b7-khvpl" Oct 30 20:58:32.422208 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:58:32.422197 5378 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/48a0901f-8b2b-4278-8422-226bb6bc6dd3-workload-socket\") pod \"e2e-worker-w0-781058-openshift-default-55fb495b7-5zx7k\" (UID: \"48a0901f-8b2b-4278-8422-226bb6bc6dd3\") " pod="openshift-ingress/e2e-worker-w0-781058-openshift-default-55fb495b7-5zx7k" Oct 30 20:58:32.422336 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:58:32.422268 5378 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/afcb5db1-70aa-4758-90e6-e21b112d48f4-istio-podinfo\") pod \"e2e-worker-w0-781058-openshift-default-55fb495b7-khvpl\" (UID: \"afcb5db1-70aa-4758-90e6-e21b112d48f4\") " pod="openshift-ingress/e2e-worker-w0-781058-openshift-default-55fb495b7-khvpl" Oct 30 20:58:32.422399 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:58:32.422382 5378 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/afcb5db1-70aa-4758-90e6-e21b112d48f4-istio-data\") pod \"e2e-worker-w0-781058-openshift-default-55fb495b7-khvpl\" (UID: \"afcb5db1-70aa-4758-90e6-e21b112d48f4\") " pod="openshift-ingress/e2e-worker-w0-781058-openshift-default-55fb495b7-khvpl" Oct 30 20:58:32.422439 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:58:32.422424 5378 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/48a0901f-8b2b-4278-8422-226bb6bc6dd3-istio-token\") pod \"e2e-worker-w0-781058-openshift-default-55fb495b7-5zx7k\" (UID: \"48a0901f-8b2b-4278-8422-226bb6bc6dd3\") " pod="openshift-ingress/e2e-worker-w0-781058-openshift-default-55fb495b7-5zx7k" Oct 30 20:58:32.422536 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:58:32.422524 5378 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/afcb5db1-70aa-4758-90e6-e21b112d48f4-istio-token\") pod \"e2e-worker-w0-781058-openshift-default-55fb495b7-khvpl\" (UID: \"afcb5db1-70aa-4758-90e6-e21b112d48f4\") " pod="openshift-ingress/e2e-worker-w0-781058-openshift-default-55fb495b7-khvpl" Oct 30 20:58:32.422620 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:58:32.422599 5378 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/afcb5db1-70aa-4758-90e6-e21b112d48f4-workload-socket\") pod \"e2e-worker-w0-781058-openshift-default-55fb495b7-khvpl\" (UID: \"afcb5db1-70aa-4758-90e6-e21b112d48f4\") " pod="openshift-ingress/e2e-worker-w0-781058-openshift-default-55fb495b7-khvpl" Oct 30 20:58:32.422681 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:58:32.422668 5378 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/afcb5db1-70aa-4758-90e6-e21b112d48f4-istiod-ca-cert\") pod \"e2e-worker-w0-781058-openshift-default-55fb495b7-khvpl\" (UID: \"afcb5db1-70aa-4758-90e6-e21b112d48f4\") " pod="openshift-ingress/e2e-worker-w0-781058-openshift-default-55fb495b7-khvpl" Oct 30 20:58:32.422717 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:58:32.422701 5378 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/afcb5db1-70aa-4758-90e6-e21b112d48f4-workload-certs\") pod \"e2e-worker-w0-781058-openshift-default-55fb495b7-khvpl\" (UID: \"afcb5db1-70aa-4758-90e6-e21b112d48f4\") " pod="openshift-ingress/e2e-worker-w0-781058-openshift-default-55fb495b7-khvpl" Oct 30 20:58:32.422758 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:58:32.422743 5378 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/48a0901f-8b2b-4278-8422-226bb6bc6dd3-istio-data\") pod \"e2e-worker-w0-781058-openshift-default-55fb495b7-5zx7k\" (UID: \"48a0901f-8b2b-4278-8422-226bb6bc6dd3\") " pod="openshift-ingress/e2e-worker-w0-781058-openshift-default-55fb495b7-5zx7k" Oct 30 20:58:32.422806 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:58:32.422782 5378 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/48a0901f-8b2b-4278-8422-226bb6bc6dd3-workload-certs\") pod \"e2e-worker-w0-781058-openshift-default-55fb495b7-5zx7k\" (UID: \"48a0901f-8b2b-4278-8422-226bb6bc6dd3\") " pod="openshift-ingress/e2e-worker-w0-781058-openshift-default-55fb495b7-5zx7k" Oct 30 20:58:32.422863 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:58:32.422829 5378 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c9xkz\" (UniqueName: \"kubernetes.io/projected/48a0901f-8b2b-4278-8422-226bb6bc6dd3-kube-api-access-c9xkz\") pod \"e2e-worker-w0-781058-openshift-default-55fb495b7-5zx7k\" (UID: \"48a0901f-8b2b-4278-8422-226bb6bc6dd3\") " pod="openshift-ingress/e2e-worker-w0-781058-openshift-default-55fb495b7-5zx7k" Oct 30 20:58:32.422922 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:58:32.422900 5378 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x6cdp\" (UniqueName: \"kubernetes.io/projected/afcb5db1-70aa-4758-90e6-e21b112d48f4-kube-api-access-x6cdp\") pod \"e2e-worker-w0-781058-openshift-default-55fb495b7-khvpl\" (UID: \"afcb5db1-70aa-4758-90e6-e21b112d48f4\") " pod="openshift-ingress/e2e-worker-w0-781058-openshift-default-55fb495b7-khvpl" Oct 30 20:58:32.445228 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:58:32.445200 5378 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-781058-openshift-default-55fb495b7-5zx7k"] Oct 30 20:58:32.455689 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:58:32.455636 5378 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-781058-openshift-default-55fb495b7-khvpl"] Oct 30 20:58:32.524090 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:58:32.524057 5378 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/48a0901f-8b2b-4278-8422-226bb6bc6dd3-istio-podinfo\") pod \"e2e-worker-w0-781058-openshift-default-55fb495b7-5zx7k\" (UID: \"48a0901f-8b2b-4278-8422-226bb6bc6dd3\") " pod="openshift-ingress/e2e-worker-w0-781058-openshift-default-55fb495b7-5zx7k" Oct 30 20:58:32.524240 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:58:32.524091 5378 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/48a0901f-8b2b-4278-8422-226bb6bc6dd3-istio-envoy\") pod \"e2e-worker-w0-781058-openshift-default-55fb495b7-5zx7k\" (UID: \"48a0901f-8b2b-4278-8422-226bb6bc6dd3\") " pod="openshift-ingress/e2e-worker-w0-781058-openshift-default-55fb495b7-5zx7k" Oct 30 20:58:32.524240 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:58:32.524125 5378 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/afcb5db1-70aa-4758-90e6-e21b112d48f4-credential-socket\") pod \"e2e-worker-w0-781058-openshift-default-55fb495b7-khvpl\" (UID: \"afcb5db1-70aa-4758-90e6-e21b112d48f4\") " pod="openshift-ingress/e2e-worker-w0-781058-openshift-default-55fb495b7-khvpl" Oct 30 20:58:32.524240 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:58:32.524154 5378 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/48a0901f-8b2b-4278-8422-226bb6bc6dd3-istiod-ca-cert\") pod \"e2e-worker-w0-781058-openshift-default-55fb495b7-5zx7k\" (UID: \"48a0901f-8b2b-4278-8422-226bb6bc6dd3\") " pod="openshift-ingress/e2e-worker-w0-781058-openshift-default-55fb495b7-5zx7k" Oct 30 20:58:32.524240 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:58:32.524185 5378 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/48a0901f-8b2b-4278-8422-226bb6bc6dd3-credential-socket\") pod \"e2e-worker-w0-781058-openshift-default-55fb495b7-5zx7k\" (UID: \"48a0901f-8b2b-4278-8422-226bb6bc6dd3\") " pod="openshift-ingress/e2e-worker-w0-781058-openshift-default-55fb495b7-5zx7k" Oct 30 20:58:32.524240 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:58:32.524212 5378 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/afcb5db1-70aa-4758-90e6-e21b112d48f4-istio-envoy\") pod \"e2e-worker-w0-781058-openshift-default-55fb495b7-khvpl\" (UID: \"afcb5db1-70aa-4758-90e6-e21b112d48f4\") " pod="openshift-ingress/e2e-worker-w0-781058-openshift-default-55fb495b7-khvpl" Oct 30 20:58:32.524240 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:58:32.524240 5378 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/48a0901f-8b2b-4278-8422-226bb6bc6dd3-workload-socket\") pod \"e2e-worker-w0-781058-openshift-default-55fb495b7-5zx7k\" (UID: \"48a0901f-8b2b-4278-8422-226bb6bc6dd3\") " pod="openshift-ingress/e2e-worker-w0-781058-openshift-default-55fb495b7-5zx7k" Oct 30 20:58:32.524602 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:58:32.524264 5378 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/afcb5db1-70aa-4758-90e6-e21b112d48f4-istio-podinfo\") pod \"e2e-worker-w0-781058-openshift-default-55fb495b7-khvpl\" (UID: \"afcb5db1-70aa-4758-90e6-e21b112d48f4\") " pod="openshift-ingress/e2e-worker-w0-781058-openshift-default-55fb495b7-khvpl" Oct 30 20:58:32.524602 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:58:32.524293 5378 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/afcb5db1-70aa-4758-90e6-e21b112d48f4-istio-data\") pod \"e2e-worker-w0-781058-openshift-default-55fb495b7-khvpl\" (UID: \"afcb5db1-70aa-4758-90e6-e21b112d48f4\") " pod="openshift-ingress/e2e-worker-w0-781058-openshift-default-55fb495b7-khvpl" Oct 30 20:58:32.524602 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:58:32.524350 5378 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/48a0901f-8b2b-4278-8422-226bb6bc6dd3-istio-token\") pod \"e2e-worker-w0-781058-openshift-default-55fb495b7-5zx7k\" (UID: \"48a0901f-8b2b-4278-8422-226bb6bc6dd3\") " pod="openshift-ingress/e2e-worker-w0-781058-openshift-default-55fb495b7-5zx7k" Oct 30 20:58:32.524602 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:58:32.524450 5378 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/afcb5db1-70aa-4758-90e6-e21b112d48f4-istio-token\") pod \"e2e-worker-w0-781058-openshift-default-55fb495b7-khvpl\" (UID: \"afcb5db1-70aa-4758-90e6-e21b112d48f4\") " pod="openshift-ingress/e2e-worker-w0-781058-openshift-default-55fb495b7-khvpl" Oct 30 20:58:32.524602 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:58:32.524477 5378 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/afcb5db1-70aa-4758-90e6-e21b112d48f4-workload-socket\") pod \"e2e-worker-w0-781058-openshift-default-55fb495b7-khvpl\" (UID: \"afcb5db1-70aa-4758-90e6-e21b112d48f4\") " pod="openshift-ingress/e2e-worker-w0-781058-openshift-default-55fb495b7-khvpl" Oct 30 20:58:32.524602 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:58:32.524514 5378 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/afcb5db1-70aa-4758-90e6-e21b112d48f4-istiod-ca-cert\") pod \"e2e-worker-w0-781058-openshift-default-55fb495b7-khvpl\" (UID: \"afcb5db1-70aa-4758-90e6-e21b112d48f4\") " pod="openshift-ingress/e2e-worker-w0-781058-openshift-default-55fb495b7-khvpl" Oct 30 20:58:32.524602 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:58:32.524545 5378 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/afcb5db1-70aa-4758-90e6-e21b112d48f4-workload-certs\") pod \"e2e-worker-w0-781058-openshift-default-55fb495b7-khvpl\" (UID: \"afcb5db1-70aa-4758-90e6-e21b112d48f4\") " pod="openshift-ingress/e2e-worker-w0-781058-openshift-default-55fb495b7-khvpl" Oct 30 20:58:32.524602 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:58:32.524554 5378 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/afcb5db1-70aa-4758-90e6-e21b112d48f4-credential-socket\") pod \"e2e-worker-w0-781058-openshift-default-55fb495b7-khvpl\" (UID: \"afcb5db1-70aa-4758-90e6-e21b112d48f4\") " pod="openshift-ingress/e2e-worker-w0-781058-openshift-default-55fb495b7-khvpl" Oct 30 20:58:32.525023 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:58:32.524609 5378 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/48a0901f-8b2b-4278-8422-226bb6bc6dd3-istio-data\") pod \"e2e-worker-w0-781058-openshift-default-55fb495b7-5zx7k\" (UID: \"48a0901f-8b2b-4278-8422-226bb6bc6dd3\") " pod="openshift-ingress/e2e-worker-w0-781058-openshift-default-55fb495b7-5zx7k" Oct 30 20:58:32.525023 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:58:32.524621 5378 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/48a0901f-8b2b-4278-8422-226bb6bc6dd3-credential-socket\") pod \"e2e-worker-w0-781058-openshift-default-55fb495b7-5zx7k\" (UID: \"48a0901f-8b2b-4278-8422-226bb6bc6dd3\") " pod="openshift-ingress/e2e-worker-w0-781058-openshift-default-55fb495b7-5zx7k" Oct 30 20:58:32.525023 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:58:32.524665 5378 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/48a0901f-8b2b-4278-8422-226bb6bc6dd3-workload-certs\") pod \"e2e-worker-w0-781058-openshift-default-55fb495b7-5zx7k\" (UID: \"48a0901f-8b2b-4278-8422-226bb6bc6dd3\") " pod="openshift-ingress/e2e-worker-w0-781058-openshift-default-55fb495b7-5zx7k" Oct 30 20:58:32.525023 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:58:32.524668 5378 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/48a0901f-8b2b-4278-8422-226bb6bc6dd3-workload-socket\") pod \"e2e-worker-w0-781058-openshift-default-55fb495b7-5zx7k\" (UID: \"48a0901f-8b2b-4278-8422-226bb6bc6dd3\") " pod="openshift-ingress/e2e-worker-w0-781058-openshift-default-55fb495b7-5zx7k" Oct 30 20:58:32.525023 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:58:32.524693 5378 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c9xkz\" (UniqueName: \"kubernetes.io/projected/48a0901f-8b2b-4278-8422-226bb6bc6dd3-kube-api-access-c9xkz\") pod \"e2e-worker-w0-781058-openshift-default-55fb495b7-5zx7k\" (UID: \"48a0901f-8b2b-4278-8422-226bb6bc6dd3\") " pod="openshift-ingress/e2e-worker-w0-781058-openshift-default-55fb495b7-5zx7k" Oct 30 20:58:32.525023 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:58:32.524736 5378 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x6cdp\" (UniqueName: \"kubernetes.io/projected/afcb5db1-70aa-4758-90e6-e21b112d48f4-kube-api-access-x6cdp\") pod \"e2e-worker-w0-781058-openshift-default-55fb495b7-khvpl\" (UID: \"afcb5db1-70aa-4758-90e6-e21b112d48f4\") " pod="openshift-ingress/e2e-worker-w0-781058-openshift-default-55fb495b7-khvpl" Oct 30 20:58:32.525023 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:58:32.524922 5378 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/48a0901f-8b2b-4278-8422-226bb6bc6dd3-istio-data\") pod \"e2e-worker-w0-781058-openshift-default-55fb495b7-5zx7k\" (UID: \"48a0901f-8b2b-4278-8422-226bb6bc6dd3\") " pod="openshift-ingress/e2e-worker-w0-781058-openshift-default-55fb495b7-5zx7k" Oct 30 20:58:32.525023 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:58:32.524954 5378 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/48a0901f-8b2b-4278-8422-226bb6bc6dd3-workload-certs\") pod \"e2e-worker-w0-781058-openshift-default-55fb495b7-5zx7k\" (UID: \"48a0901f-8b2b-4278-8422-226bb6bc6dd3\") " pod="openshift-ingress/e2e-worker-w0-781058-openshift-default-55fb495b7-5zx7k" Oct 30 20:58:32.525440 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:58:32.525189 5378 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/afcb5db1-70aa-4758-90e6-e21b112d48f4-workload-certs\") pod \"e2e-worker-w0-781058-openshift-default-55fb495b7-khvpl\" (UID: \"afcb5db1-70aa-4758-90e6-e21b112d48f4\") " pod="openshift-ingress/e2e-worker-w0-781058-openshift-default-55fb495b7-khvpl" Oct 30 20:58:32.525440 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:58:32.525349 5378 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/afcb5db1-70aa-4758-90e6-e21b112d48f4-workload-socket\") pod \"e2e-worker-w0-781058-openshift-default-55fb495b7-khvpl\" (UID: \"afcb5db1-70aa-4758-90e6-e21b112d48f4\") " pod="openshift-ingress/e2e-worker-w0-781058-openshift-default-55fb495b7-khvpl" Oct 30 20:58:32.525440 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:58:32.525410 5378 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/afcb5db1-70aa-4758-90e6-e21b112d48f4-istio-data\") pod \"e2e-worker-w0-781058-openshift-default-55fb495b7-khvpl\" (UID: \"afcb5db1-70aa-4758-90e6-e21b112d48f4\") " pod="openshift-ingress/e2e-worker-w0-781058-openshift-default-55fb495b7-khvpl" Oct 30 20:58:32.525551 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:58:32.525475 5378 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/afcb5db1-70aa-4758-90e6-e21b112d48f4-istiod-ca-cert\") pod \"e2e-worker-w0-781058-openshift-default-55fb495b7-khvpl\" (UID: \"afcb5db1-70aa-4758-90e6-e21b112d48f4\") " pod="openshift-ingress/e2e-worker-w0-781058-openshift-default-55fb495b7-khvpl" Oct 30 20:58:32.525872 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:58:32.525848 5378 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/48a0901f-8b2b-4278-8422-226bb6bc6dd3-istiod-ca-cert\") pod \"e2e-worker-w0-781058-openshift-default-55fb495b7-5zx7k\" (UID: \"48a0901f-8b2b-4278-8422-226bb6bc6dd3\") " pod="openshift-ingress/e2e-worker-w0-781058-openshift-default-55fb495b7-5zx7k" Oct 30 20:58:32.527084 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:58:32.527063 5378 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/afcb5db1-70aa-4758-90e6-e21b112d48f4-istio-envoy\") pod \"e2e-worker-w0-781058-openshift-default-55fb495b7-khvpl\" (UID: \"afcb5db1-70aa-4758-90e6-e21b112d48f4\") " pod="openshift-ingress/e2e-worker-w0-781058-openshift-default-55fb495b7-khvpl" Oct 30 20:58:32.527282 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:58:32.527263 5378 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/48a0901f-8b2b-4278-8422-226bb6bc6dd3-istio-podinfo\") pod \"e2e-worker-w0-781058-openshift-default-55fb495b7-5zx7k\" (UID: \"48a0901f-8b2b-4278-8422-226bb6bc6dd3\") " pod="openshift-ingress/e2e-worker-w0-781058-openshift-default-55fb495b7-5zx7k" Oct 30 20:58:32.527555 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:58:32.527535 5378 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/afcb5db1-70aa-4758-90e6-e21b112d48f4-istio-podinfo\") pod \"e2e-worker-w0-781058-openshift-default-55fb495b7-khvpl\" (UID: \"afcb5db1-70aa-4758-90e6-e21b112d48f4\") " pod="openshift-ingress/e2e-worker-w0-781058-openshift-default-55fb495b7-khvpl" Oct 30 20:58:32.527641 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:58:32.527542 5378 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/48a0901f-8b2b-4278-8422-226bb6bc6dd3-istio-envoy\") pod \"e2e-worker-w0-781058-openshift-default-55fb495b7-5zx7k\" (UID: \"48a0901f-8b2b-4278-8422-226bb6bc6dd3\") " pod="openshift-ingress/e2e-worker-w0-781058-openshift-default-55fb495b7-5zx7k" Oct 30 20:58:32.560985 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:58:32.560959 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 20:58:32.585005 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:58:32.584973 5378 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/48a0901f-8b2b-4278-8422-226bb6bc6dd3-istio-token\") pod \"e2e-worker-w0-781058-openshift-default-55fb495b7-5zx7k\" (UID: \"48a0901f-8b2b-4278-8422-226bb6bc6dd3\") " pod="openshift-ingress/e2e-worker-w0-781058-openshift-default-55fb495b7-5zx7k" Oct 30 20:58:32.585117 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:58:32.585077 5378 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c9xkz\" (UniqueName: \"kubernetes.io/projected/48a0901f-8b2b-4278-8422-226bb6bc6dd3-kube-api-access-c9xkz\") pod \"e2e-worker-w0-781058-openshift-default-55fb495b7-5zx7k\" (UID: \"48a0901f-8b2b-4278-8422-226bb6bc6dd3\") " pod="openshift-ingress/e2e-worker-w0-781058-openshift-default-55fb495b7-5zx7k" Oct 30 20:58:32.585365 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:58:32.585344 5378 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/afcb5db1-70aa-4758-90e6-e21b112d48f4-istio-token\") pod \"e2e-worker-w0-781058-openshift-default-55fb495b7-khvpl\" (UID: \"afcb5db1-70aa-4758-90e6-e21b112d48f4\") " pod="openshift-ingress/e2e-worker-w0-781058-openshift-default-55fb495b7-khvpl" Oct 30 20:58:32.585432 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:58:32.585391 5378 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x6cdp\" (UniqueName: \"kubernetes.io/projected/afcb5db1-70aa-4758-90e6-e21b112d48f4-kube-api-access-x6cdp\") pod \"e2e-worker-w0-781058-openshift-default-55fb495b7-khvpl\" (UID: \"afcb5db1-70aa-4758-90e6-e21b112d48f4\") " pod="openshift-ingress/e2e-worker-w0-781058-openshift-default-55fb495b7-khvpl" Oct 30 20:58:32.651054 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:58:32.651015 5378 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-d061d0-openshift-default-86b87f495-rqk9z"] Oct 30 20:58:32.656922 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:58:32.656898 5378 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-d061d0-openshift-default-86b87f495-rqk9z" Oct 30 20:58:32.658331 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:58:32.658299 5378 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-781058-openshift-default-55fb495b7-5zx7k" Oct 30 20:58:32.713524 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:58:32.712763 5378 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w2-d061d0-openshift-default-dockercfg-cwlxq\"" Oct 30 20:58:32.713524 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:58:32.712962 5378 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-781058-openshift-default-55fb495b7-khvpl" Oct 30 20:58:32.718680 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:58:32.716699 5378 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-d061d0-openshift-default-86b87f495-rqk9z"] Oct 30 20:58:32.742609 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:58:32.742228 5378 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5aec1868-e81e-40f2-b955-29d704ba75b9-istio-envoy\") pod \"e2e-worker-w2-d061d0-openshift-default-86b87f495-rqk9z\" (UID: \"5aec1868-e81e-40f2-b955-29d704ba75b9\") " pod="openshift-ingress/e2e-worker-w2-d061d0-openshift-default-86b87f495-rqk9z" Oct 30 20:58:32.742609 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:58:32.742272 5378 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5aec1868-e81e-40f2-b955-29d704ba75b9-istio-token\") pod \"e2e-worker-w2-d061d0-openshift-default-86b87f495-rqk9z\" (UID: \"5aec1868-e81e-40f2-b955-29d704ba75b9\") " pod="openshift-ingress/e2e-worker-w2-d061d0-openshift-default-86b87f495-rqk9z" Oct 30 20:58:32.742609 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:58:32.742306 5378 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5aec1868-e81e-40f2-b955-29d704ba75b9-credential-socket\") pod \"e2e-worker-w2-d061d0-openshift-default-86b87f495-rqk9z\" (UID: \"5aec1868-e81e-40f2-b955-29d704ba75b9\") " pod="openshift-ingress/e2e-worker-w2-d061d0-openshift-default-86b87f495-rqk9z" Oct 30 20:58:32.742609 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:58:32.742381 5378 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5aec1868-e81e-40f2-b955-29d704ba75b9-workload-certs\") pod \"e2e-worker-w2-d061d0-openshift-default-86b87f495-rqk9z\" (UID: \"5aec1868-e81e-40f2-b955-29d704ba75b9\") " pod="openshift-ingress/e2e-worker-w2-d061d0-openshift-default-86b87f495-rqk9z" Oct 30 20:58:32.742609 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:58:32.742399 5378 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5aec1868-e81e-40f2-b955-29d704ba75b9-istio-data\") pod \"e2e-worker-w2-d061d0-openshift-default-86b87f495-rqk9z\" (UID: \"5aec1868-e81e-40f2-b955-29d704ba75b9\") " pod="openshift-ingress/e2e-worker-w2-d061d0-openshift-default-86b87f495-rqk9z" Oct 30 20:58:32.742609 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:58:32.742414 5378 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5aec1868-e81e-40f2-b955-29d704ba75b9-istio-podinfo\") pod \"e2e-worker-w2-d061d0-openshift-default-86b87f495-rqk9z\" (UID: \"5aec1868-e81e-40f2-b955-29d704ba75b9\") " pod="openshift-ingress/e2e-worker-w2-d061d0-openshift-default-86b87f495-rqk9z" Oct 30 20:58:32.742609 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:58:32.742431 5378 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5aec1868-e81e-40f2-b955-29d704ba75b9-istiod-ca-cert\") pod \"e2e-worker-w2-d061d0-openshift-default-86b87f495-rqk9z\" (UID: \"5aec1868-e81e-40f2-b955-29d704ba75b9\") " pod="openshift-ingress/e2e-worker-w2-d061d0-openshift-default-86b87f495-rqk9z" Oct 30 20:58:32.742609 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:58:32.742510 5378 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dhc57\" (UniqueName: \"kubernetes.io/projected/5aec1868-e81e-40f2-b955-29d704ba75b9-kube-api-access-dhc57\") pod \"e2e-worker-w2-d061d0-openshift-default-86b87f495-rqk9z\" (UID: \"5aec1868-e81e-40f2-b955-29d704ba75b9\") " pod="openshift-ingress/e2e-worker-w2-d061d0-openshift-default-86b87f495-rqk9z" Oct 30 20:58:32.742609 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:58:32.742535 5378 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5aec1868-e81e-40f2-b955-29d704ba75b9-workload-socket\") pod \"e2e-worker-w2-d061d0-openshift-default-86b87f495-rqk9z\" (UID: \"5aec1868-e81e-40f2-b955-29d704ba75b9\") " pod="openshift-ingress/e2e-worker-w2-d061d0-openshift-default-86b87f495-rqk9z" Oct 30 20:58:32.843378 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:58:32.843339 5378 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5aec1868-e81e-40f2-b955-29d704ba75b9-credential-socket\") pod \"e2e-worker-w2-d061d0-openshift-default-86b87f495-rqk9z\" (UID: \"5aec1868-e81e-40f2-b955-29d704ba75b9\") " pod="openshift-ingress/e2e-worker-w2-d061d0-openshift-default-86b87f495-rqk9z" Oct 30 20:58:32.843526 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:58:32.843485 5378 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5aec1868-e81e-40f2-b955-29d704ba75b9-workload-certs\") pod \"e2e-worker-w2-d061d0-openshift-default-86b87f495-rqk9z\" (UID: \"5aec1868-e81e-40f2-b955-29d704ba75b9\") " pod="openshift-ingress/e2e-worker-w2-d061d0-openshift-default-86b87f495-rqk9z" Oct 30 20:58:32.843526 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:58:32.843520 5378 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5aec1868-e81e-40f2-b955-29d704ba75b9-istio-data\") pod \"e2e-worker-w2-d061d0-openshift-default-86b87f495-rqk9z\" (UID: \"5aec1868-e81e-40f2-b955-29d704ba75b9\") " pod="openshift-ingress/e2e-worker-w2-d061d0-openshift-default-86b87f495-rqk9z" Oct 30 20:58:32.843674 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:58:32.843545 5378 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5aec1868-e81e-40f2-b955-29d704ba75b9-istio-podinfo\") pod \"e2e-worker-w2-d061d0-openshift-default-86b87f495-rqk9z\" (UID: \"5aec1868-e81e-40f2-b955-29d704ba75b9\") " pod="openshift-ingress/e2e-worker-w2-d061d0-openshift-default-86b87f495-rqk9z" Oct 30 20:58:32.843674 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:58:32.843607 5378 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5aec1868-e81e-40f2-b955-29d704ba75b9-istiod-ca-cert\") pod \"e2e-worker-w2-d061d0-openshift-default-86b87f495-rqk9z\" (UID: \"5aec1868-e81e-40f2-b955-29d704ba75b9\") " pod="openshift-ingress/e2e-worker-w2-d061d0-openshift-default-86b87f495-rqk9z" Oct 30 20:58:32.843817 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:58:32.843727 5378 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dhc57\" (UniqueName: \"kubernetes.io/projected/5aec1868-e81e-40f2-b955-29d704ba75b9-kube-api-access-dhc57\") pod \"e2e-worker-w2-d061d0-openshift-default-86b87f495-rqk9z\" (UID: \"5aec1868-e81e-40f2-b955-29d704ba75b9\") " pod="openshift-ingress/e2e-worker-w2-d061d0-openshift-default-86b87f495-rqk9z" Oct 30 20:58:32.843817 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:58:32.843763 5378 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5aec1868-e81e-40f2-b955-29d704ba75b9-credential-socket\") pod \"e2e-worker-w2-d061d0-openshift-default-86b87f495-rqk9z\" (UID: \"5aec1868-e81e-40f2-b955-29d704ba75b9\") " pod="openshift-ingress/e2e-worker-w2-d061d0-openshift-default-86b87f495-rqk9z" Oct 30 20:58:32.843817 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:58:32.843777 5378 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5aec1868-e81e-40f2-b955-29d704ba75b9-workload-socket\") pod \"e2e-worker-w2-d061d0-openshift-default-86b87f495-rqk9z\" (UID: \"5aec1868-e81e-40f2-b955-29d704ba75b9\") " pod="openshift-ingress/e2e-worker-w2-d061d0-openshift-default-86b87f495-rqk9z" Oct 30 20:58:32.843817 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:58:32.843794 5378 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5aec1868-e81e-40f2-b955-29d704ba75b9-workload-certs\") pod \"e2e-worker-w2-d061d0-openshift-default-86b87f495-rqk9z\" (UID: \"5aec1868-e81e-40f2-b955-29d704ba75b9\") " pod="openshift-ingress/e2e-worker-w2-d061d0-openshift-default-86b87f495-rqk9z" Oct 30 20:58:32.844024 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:58:32.843855 5378 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5aec1868-e81e-40f2-b955-29d704ba75b9-istio-envoy\") pod \"e2e-worker-w2-d061d0-openshift-default-86b87f495-rqk9z\" (UID: \"5aec1868-e81e-40f2-b955-29d704ba75b9\") " pod="openshift-ingress/e2e-worker-w2-d061d0-openshift-default-86b87f495-rqk9z" Oct 30 20:58:32.844024 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:58:32.843882 5378 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5aec1868-e81e-40f2-b955-29d704ba75b9-istio-token\") pod \"e2e-worker-w2-d061d0-openshift-default-86b87f495-rqk9z\" (UID: \"5aec1868-e81e-40f2-b955-29d704ba75b9\") " pod="openshift-ingress/e2e-worker-w2-d061d0-openshift-default-86b87f495-rqk9z" Oct 30 20:58:32.844384 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:58:32.844322 5378 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5aec1868-e81e-40f2-b955-29d704ba75b9-workload-socket\") pod \"e2e-worker-w2-d061d0-openshift-default-86b87f495-rqk9z\" (UID: \"5aec1868-e81e-40f2-b955-29d704ba75b9\") " pod="openshift-ingress/e2e-worker-w2-d061d0-openshift-default-86b87f495-rqk9z" Oct 30 20:58:32.844384 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:58:32.844350 5378 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5aec1868-e81e-40f2-b955-29d704ba75b9-istio-data\") pod \"e2e-worker-w2-d061d0-openshift-default-86b87f495-rqk9z\" (UID: \"5aec1868-e81e-40f2-b955-29d704ba75b9\") " pod="openshift-ingress/e2e-worker-w2-d061d0-openshift-default-86b87f495-rqk9z" Oct 30 20:58:32.844819 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:58:32.844795 5378 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5aec1868-e81e-40f2-b955-29d704ba75b9-istiod-ca-cert\") pod \"e2e-worker-w2-d061d0-openshift-default-86b87f495-rqk9z\" (UID: \"5aec1868-e81e-40f2-b955-29d704ba75b9\") " pod="openshift-ingress/e2e-worker-w2-d061d0-openshift-default-86b87f495-rqk9z" Oct 30 20:58:32.846216 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:58:32.846194 5378 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5aec1868-e81e-40f2-b955-29d704ba75b9-istio-podinfo\") pod \"e2e-worker-w2-d061d0-openshift-default-86b87f495-rqk9z\" (UID: \"5aec1868-e81e-40f2-b955-29d704ba75b9\") " pod="openshift-ingress/e2e-worker-w2-d061d0-openshift-default-86b87f495-rqk9z" Oct 30 20:58:32.846577 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:58:32.846545 5378 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5aec1868-e81e-40f2-b955-29d704ba75b9-istio-envoy\") pod \"e2e-worker-w2-d061d0-openshift-default-86b87f495-rqk9z\" (UID: \"5aec1868-e81e-40f2-b955-29d704ba75b9\") " pod="openshift-ingress/e2e-worker-w2-d061d0-openshift-default-86b87f495-rqk9z" Oct 30 20:58:32.901934 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:58:32.901872 5378 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5aec1868-e81e-40f2-b955-29d704ba75b9-istio-token\") pod \"e2e-worker-w2-d061d0-openshift-default-86b87f495-rqk9z\" (UID: \"5aec1868-e81e-40f2-b955-29d704ba75b9\") " pod="openshift-ingress/e2e-worker-w2-d061d0-openshift-default-86b87f495-rqk9z" Oct 30 20:58:32.902043 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:58:32.901937 5378 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dhc57\" (UniqueName: \"kubernetes.io/projected/5aec1868-e81e-40f2-b955-29d704ba75b9-kube-api-access-dhc57\") pod \"e2e-worker-w2-d061d0-openshift-default-86b87f495-rqk9z\" (UID: \"5aec1868-e81e-40f2-b955-29d704ba75b9\") " pod="openshift-ingress/e2e-worker-w2-d061d0-openshift-default-86b87f495-rqk9z" Oct 30 20:58:32.971651 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:58:32.971618 5378 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-d061d0-openshift-default-86b87f495-rqk9z" Oct 30 20:58:33.037786 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:58:33.037749 5378 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-781058-openshift-default-55fb495b7-5zx7k"] Oct 30 20:58:33.048268 ip-10-0-138-149 kubenswrapper[5378]: W1030 20:58:33.048236 5378 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod48a0901f_8b2b_4278_8422_226bb6bc6dd3.slice/crio-68447e73ada5c838ae757208013800e46def98e0e5e8ce7d87383a3cd27279b3 WatchSource:0}: Error finding container 68447e73ada5c838ae757208013800e46def98e0e5e8ce7d87383a3cd27279b3: Status 404 returned error can't find the container with id 68447e73ada5c838ae757208013800e46def98e0e5e8ce7d87383a3cd27279b3 Oct 30 20:58:33.050720 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:58:33.050686 5378 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 20:58:33.050809 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:58:33.050758 5378 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 20:58:33.050809 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:58:33.050790 5378 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 20:58:33.134874 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:58:33.134847 5378 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-781058-openshift-default-55fb495b7-khvpl"] Oct 30 20:58:33.135137 ip-10-0-138-149 kubenswrapper[5378]: W1030 20:58:33.135115 5378 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podafcb5db1_70aa_4758_90e6_e21b112d48f4.slice/crio-7904f41e16b8f8abff2b5e054eb6e4b1147c84c9ff11305442c79027fbf7d00e WatchSource:0}: Error finding container 7904f41e16b8f8abff2b5e054eb6e4b1147c84c9ff11305442c79027fbf7d00e: Status 404 returned error can't find the container with id 7904f41e16b8f8abff2b5e054eb6e4b1147c84c9ff11305442c79027fbf7d00e Oct 30 20:58:33.137804 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:58:33.137766 5378 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 20:58:33.137897 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:58:33.137849 5378 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 20:58:33.137937 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:58:33.137917 5378 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 20:58:33.347844 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:58:33.347819 5378 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-d061d0-openshift-default-86b87f495-rqk9z"] Oct 30 20:58:33.348267 ip-10-0-138-149 kubenswrapper[5378]: W1030 20:58:33.348240 5378 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5aec1868_e81e_40f2_b955_29d704ba75b9.slice/crio-643be13243800a8a1084cbdb7d831abb05933c451fde5442955265f0b1b839bd WatchSource:0}: Error finding container 643be13243800a8a1084cbdb7d831abb05933c451fde5442955265f0b1b839bd: Status 404 returned error can't find the container with id 643be13243800a8a1084cbdb7d831abb05933c451fde5442955265f0b1b839bd Oct 30 20:58:33.350831 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:58:33.350803 5378 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 20:58:33.350906 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:58:33.350867 5378 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 20:58:33.350906 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:58:33.350897 5378 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 20:58:33.745128 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:58:33.745025 5378 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-781058-openshift-default-55fb495b7-khvpl" event={"ID":"afcb5db1-70aa-4758-90e6-e21b112d48f4","Type":"ContainerStarted","Data":"c682db6caa42bbefaa74a7509ac7c21897323acff7831c3caf7369bba15a262f"} Oct 30 20:58:33.745128 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:58:33.745084 5378 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-781058-openshift-default-55fb495b7-khvpl" event={"ID":"afcb5db1-70aa-4758-90e6-e21b112d48f4","Type":"ContainerStarted","Data":"7904f41e16b8f8abff2b5e054eb6e4b1147c84c9ff11305442c79027fbf7d00e"} Oct 30 20:58:33.745759 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:58:33.745733 5378 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-781058-openshift-default-55fb495b7-5zx7k" event={"ID":"48a0901f-8b2b-4278-8422-226bb6bc6dd3","Type":"ContainerStarted","Data":"e636fc8115299f0916b8822ff29832fec0ea4a10f340562fe04ee4014ab985c5"} Oct 30 20:58:33.745863 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:58:33.745790 5378 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-781058-openshift-default-55fb495b7-5zx7k" event={"ID":"48a0901f-8b2b-4278-8422-226bb6bc6dd3","Type":"ContainerStarted","Data":"68447e73ada5c838ae757208013800e46def98e0e5e8ce7d87383a3cd27279b3"} Oct 30 20:58:33.747485 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:58:33.747445 5378 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-d061d0-openshift-default-86b87f495-rqk9z" event={"ID":"5aec1868-e81e-40f2-b955-29d704ba75b9","Type":"ContainerStarted","Data":"d802741d82742e989d600f7d722a185de016263597496297fa00dc928f183dff"} Oct 30 20:58:33.747614 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:58:33.747518 5378 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-d061d0-openshift-default-86b87f495-rqk9z" event={"ID":"5aec1868-e81e-40f2-b955-29d704ba75b9","Type":"ContainerStarted","Data":"643be13243800a8a1084cbdb7d831abb05933c451fde5442955265f0b1b839bd"} Oct 30 20:58:33.880590 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:58:33.877159 5378 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-781058-openshift-default-55fb495b7-khvpl" podStartSLOduration=1.8771365850000001 podStartE2EDuration="1.877136585s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:58:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:58:33.863798643 +0000 UTC m=+1204.291912575" watchObservedRunningTime="2026-10-30 20:58:33.877136585 +0000 UTC m=+1204.305250547" Oct 30 20:58:33.972588 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:58:33.972221 5378 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-d061d0-openshift-default-86b87f495-rqk9z" Oct 30 20:58:33.997651 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:58:33.985344 5378 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-781058-openshift-default-55fb495b7-5zx7k" podStartSLOduration=1.985325413 podStartE2EDuration="1.985325413s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:58:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:58:33.983091176 +0000 UTC m=+1204.411205115" watchObservedRunningTime="2026-10-30 20:58:33.985325413 +0000 UTC m=+1204.413439334" Oct 30 20:58:34.108614 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:58:34.106001 5378 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-d061d0-openshift-default-86b87f495-rqk9z" podStartSLOduration=2.105981736 podStartE2EDuration="2.105981736s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:58:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:58:34.105330914 +0000 UTC m=+1204.533444850" watchObservedRunningTime="2026-10-30 20:58:34.105981736 +0000 UTC m=+1204.534095663" Oct 30 20:58:34.665590 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:58:34.661765 5378 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-781058-openshift-default-55fb495b7-5zx7k" Oct 30 20:58:34.665590 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:58:34.661919 5378 patch_prober.go:29] interesting pod/e2e-worker-w0-781058-openshift-default-55fb495b7-5zx7k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" start-of-body= Oct 30 20:58:34.665590 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:58:34.661976 5378 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-781058-openshift-default-55fb495b7-5zx7k" podUID="48a0901f-8b2b-4278-8422-226bb6bc6dd3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" Oct 30 20:58:34.729071 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:58:34.729037 5378 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-781058-openshift-default-55fb495b7-khvpl" Oct 30 20:58:34.729865 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:58:34.729836 5378 patch_prober.go:29] interesting pod/e2e-worker-w0-781058-openshift-default-55fb495b7-khvpl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused" start-of-body= Oct 30 20:58:34.730002 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:58:34.729911 5378 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-781058-openshift-default-55fb495b7-khvpl" podUID="afcb5db1-70aa-4758-90e6-e21b112d48f4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused" Oct 30 20:58:34.975651 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:58:34.975252 5378 patch_prober.go:29] interesting pod/e2e-worker-w2-d061d0-openshift-default-86b87f495-rqk9z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused" start-of-body= Oct 30 20:58:34.975651 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:58:34.975314 5378 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-d061d0-openshift-default-86b87f495-rqk9z" podUID="5aec1868-e81e-40f2-b955-29d704ba75b9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused" Oct 30 20:58:35.664590 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:58:35.660663 5378 patch_prober.go:29] interesting pod/e2e-worker-w0-781058-openshift-default-55fb495b7-5zx7k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" start-of-body= Oct 30 20:58:35.664590 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:58:35.660728 5378 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-781058-openshift-default-55fb495b7-5zx7k" podUID="48a0901f-8b2b-4278-8422-226bb6bc6dd3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" Oct 30 20:58:35.717588 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:58:35.717513 5378 patch_prober.go:29] interesting pod/e2e-worker-w0-781058-openshift-default-55fb495b7-khvpl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused" start-of-body= Oct 30 20:58:35.718098 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:58:35.717602 5378 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-781058-openshift-default-55fb495b7-khvpl" podUID="afcb5db1-70aa-4758-90e6-e21b112d48f4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused" Oct 30 20:58:35.974647 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:58:35.973899 5378 patch_prober.go:29] interesting pod/e2e-worker-w2-d061d0-openshift-default-86b87f495-rqk9z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused" start-of-body= Oct 30 20:58:35.974647 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:58:35.973961 5378 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-d061d0-openshift-default-86b87f495-rqk9z" podUID="5aec1868-e81e-40f2-b955-29d704ba75b9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused" Oct 30 20:58:36.119744 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:58:36.117934 5378 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-r9g59"] Oct 30 20:58:36.665212 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:58:36.665173 5378 patch_prober.go:29] interesting pod/e2e-worker-w0-781058-openshift-default-55fb495b7-5zx7k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" start-of-body= Oct 30 20:58:36.665525 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:58:36.665493 5378 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-781058-openshift-default-55fb495b7-5zx7k" podUID="48a0901f-8b2b-4278-8422-226bb6bc6dd3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" Oct 30 20:58:36.717999 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:58:36.717961 5378 patch_prober.go:29] interesting pod/e2e-worker-w0-781058-openshift-default-55fb495b7-khvpl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused" start-of-body= Oct 30 20:58:36.718640 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:58:36.718593 5378 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-781058-openshift-default-55fb495b7-khvpl" podUID="afcb5db1-70aa-4758-90e6-e21b112d48f4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused" Oct 30 20:58:36.973647 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:58:36.972930 5378 patch_prober.go:29] interesting pod/e2e-worker-w2-d061d0-openshift-default-86b87f495-rqk9z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused" start-of-body= Oct 30 20:58:36.973647 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:58:36.972997 5378 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-d061d0-openshift-default-86b87f495-rqk9z" podUID="5aec1868-e81e-40f2-b955-29d704ba75b9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused" Oct 30 20:58:37.665590 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:58:37.660074 5378 patch_prober.go:29] interesting pod/e2e-worker-w0-781058-openshift-default-55fb495b7-5zx7k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" start-of-body= Oct 30 20:58:37.665590 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:58:37.660149 5378 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-781058-openshift-default-55fb495b7-5zx7k" podUID="48a0901f-8b2b-4278-8422-226bb6bc6dd3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" Oct 30 20:58:37.724585 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:58:37.718600 5378 patch_prober.go:29] interesting pod/e2e-worker-w0-781058-openshift-default-55fb495b7-khvpl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused" start-of-body= Oct 30 20:58:37.724585 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:58:37.718663 5378 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-781058-openshift-default-55fb495b7-khvpl" podUID="afcb5db1-70aa-4758-90e6-e21b112d48f4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused" Oct 30 20:58:37.991659 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:58:37.990652 5378 patch_prober.go:29] interesting pod/e2e-worker-w2-d061d0-openshift-default-86b87f495-rqk9z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused" start-of-body= Oct 30 20:58:37.991659 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:58:37.990717 5378 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-d061d0-openshift-default-86b87f495-rqk9z" podUID="5aec1868-e81e-40f2-b955-29d704ba75b9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused" Oct 30 20:58:38.039110 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:58:38.039061 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 20:58:38.661590 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:58:38.660955 5378 patch_prober.go:29] interesting pod/e2e-worker-w0-781058-openshift-default-55fb495b7-5zx7k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" start-of-body= Oct 30 20:58:38.661590 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:58:38.661021 5378 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-781058-openshift-default-55fb495b7-5zx7k" podUID="48a0901f-8b2b-4278-8422-226bb6bc6dd3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" Oct 30 20:58:38.717589 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:58:38.716933 5378 patch_prober.go:29] interesting pod/e2e-worker-w0-781058-openshift-default-55fb495b7-khvpl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused" start-of-body= Oct 30 20:58:38.717589 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:58:38.716994 5378 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-781058-openshift-default-55fb495b7-khvpl" podUID="afcb5db1-70aa-4758-90e6-e21b112d48f4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused" Oct 30 20:58:38.986645 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:58:38.973957 5378 patch_prober.go:29] interesting pod/e2e-worker-w2-d061d0-openshift-default-86b87f495-rqk9z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused" start-of-body= Oct 30 20:58:38.986645 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:58:38.974024 5378 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-d061d0-openshift-default-86b87f495-rqk9z" podUID="5aec1868-e81e-40f2-b955-29d704ba75b9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused" Oct 30 20:58:39.662589 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:58:39.661965 5378 patch_prober.go:29] interesting pod/e2e-worker-w0-781058-openshift-default-55fb495b7-5zx7k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" start-of-body= Oct 30 20:58:39.662589 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:58:39.662026 5378 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-781058-openshift-default-55fb495b7-5zx7k" podUID="48a0901f-8b2b-4278-8422-226bb6bc6dd3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" Oct 30 20:58:39.684530 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:58:39.684486 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 20:58:39.713919 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:58:39.713894 5378 patch_prober.go:29] interesting pod/e2e-worker-w0-781058-openshift-default-55fb495b7-khvpl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused" start-of-body= Oct 30 20:58:39.714050 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:58:39.713948 5378 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-781058-openshift-default-55fb495b7-khvpl" podUID="afcb5db1-70aa-4758-90e6-e21b112d48f4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused" Oct 30 20:58:39.972329 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:58:39.972232 5378 patch_prober.go:29] interesting pod/e2e-worker-w2-d061d0-openshift-default-86b87f495-rqk9z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused" start-of-body= Oct 30 20:58:39.972505 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:58:39.972318 5378 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-d061d0-openshift-default-86b87f495-rqk9z" podUID="5aec1868-e81e-40f2-b955-29d704ba75b9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused" Oct 30 20:58:41.238395 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:58:41.238356 5378 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-d061d0-openshift-default-86b87f495-rqk9z" Oct 30 20:58:41.238922 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:58:41.238820 5378 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-d061d0-openshift-default-86b87f495-rqk9z" Oct 30 20:58:41.239761 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:58:41.239737 5378 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-d061d0-openshift-default-86b87f495-rqk9z" Oct 30 20:58:41.710287 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:58:41.710219 5378 patch_prober.go:29] interesting pod/e2e-worker-w0-781058-openshift-default-55fb495b7-5zx7k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.73:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 20:58:41.710500 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:58:41.710345 5378 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-781058-openshift-default-55fb495b7-5zx7k" podUID="48a0901f-8b2b-4278-8422-226bb6bc6dd3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.73:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 20:58:41.755307 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:58:41.754701 5378 patch_prober.go:29] interesting pod/e2e-worker-w0-781058-openshift-default-55fb495b7-khvpl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.74:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 20:58:41.755307 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:58:41.754761 5378 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-781058-openshift-default-55fb495b7-khvpl" podUID="afcb5db1-70aa-4758-90e6-e21b112d48f4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.74:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 20:58:42.403310 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:58:42.403272 5378 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-781058-openshift-default-55fb495b7-5zx7k" Oct 30 20:58:42.403839 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:58:42.403691 5378 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-781058-openshift-default-55fb495b7-5zx7k" Oct 30 20:58:42.404611 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:58:42.404552 5378 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-781058-openshift-default-55fb495b7-5zx7k" Oct 30 20:58:42.450585 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:58:42.450531 5378 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-781058-openshift-default-55fb495b7-khvpl" Oct 30 20:58:42.450986 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:58:42.450952 5378 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-781058-openshift-default-55fb495b7-khvpl" Oct 30 20:58:42.451711 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:58:42.451688 5378 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-781058-openshift-default-55fb495b7-khvpl" Oct 30 20:58:42.515823 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:58:42.515796 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 20:58:43.014073 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:58:43.014030 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 20:58:43.049903 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:58:43.049873 5378 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4939c02d-fm4s4"] Oct 30 20:58:43.054687 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:58:43.054664 5378 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4939c02d-fm4s4"] Oct 30 20:58:43.741914 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:58:43.741883 5378 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="563c5c7c-a42c-417c-b883-98efbbef92ca" path="/var/lib/kubelet/pods/563c5c7c-a42c-417c-b883-98efbbef92ca/volumes" Oct 30 20:58:44.549472 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:58:44.549440 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 20:58:47.885967 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:58:47.885929 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 20:58:48.837773 ip-10-0-138-149 kubenswrapper[5378]: E1030 20:58:48.837731 5378 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q6mx7" containerName="extractor" Oct 30 20:58:52.672774 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:58:52.672741 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 20:58:56.337652 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:58:56.337616 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 20:58:57.871943 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:58:57.871902 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 20:58:59.412933 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:58:59.412902 5378 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w1-f9f277/distinct-2-w1-f9f277-kserve-6c49cf5dcb-h4x74"] Oct 30 20:58:59.416974 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:58:59.416949 5378 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-f9f277/distinct-2-w1-f9f277-kserve-6c49cf5dcb-h4x74" Oct 30 20:58:59.422422 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:58:59.422393 5378 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9eb6ff54-b77b-453c-a166-988b98cf8f49-tmp-dir\") pod \"distinct-2-w1-f9f277-kserve-6c49cf5dcb-h4x74\" (UID: \"9eb6ff54-b77b-453c-a166-988b98cf8f49\") " pod="e2e-models-e2e-worker-w1-f9f277/distinct-2-w1-f9f277-kserve-6c49cf5dcb-h4x74" Oct 30 20:58:59.422587 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:58:59.422465 5378 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9eb6ff54-b77b-453c-a166-988b98cf8f49-dshm\") pod \"distinct-2-w1-f9f277-kserve-6c49cf5dcb-h4x74\" (UID: \"9eb6ff54-b77b-453c-a166-988b98cf8f49\") " pod="e2e-models-e2e-worker-w1-f9f277/distinct-2-w1-f9f277-kserve-6c49cf5dcb-h4x74" Oct 30 20:58:59.422656 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:58:59.422560 5378 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9eb6ff54-b77b-453c-a166-988b98cf8f49-home\") pod \"distinct-2-w1-f9f277-kserve-6c49cf5dcb-h4x74\" (UID: \"9eb6ff54-b77b-453c-a166-988b98cf8f49\") " pod="e2e-models-e2e-worker-w1-f9f277/distinct-2-w1-f9f277-kserve-6c49cf5dcb-h4x74" Oct 30 20:58:59.422725 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:58:59.422705 5378 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9eb6ff54-b77b-453c-a166-988b98cf8f49-tls-certs\") pod \"distinct-2-w1-f9f277-kserve-6c49cf5dcb-h4x74\" (UID: \"9eb6ff54-b77b-453c-a166-988b98cf8f49\") " pod="e2e-models-e2e-worker-w1-f9f277/distinct-2-w1-f9f277-kserve-6c49cf5dcb-h4x74" Oct 30 20:58:59.422781 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:58:59.422764 5378 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fbz2k\" (UniqueName: \"kubernetes.io/projected/9eb6ff54-b77b-453c-a166-988b98cf8f49-kube-api-access-fbz2k\") pod \"distinct-2-w1-f9f277-kserve-6c49cf5dcb-h4x74\" (UID: \"9eb6ff54-b77b-453c-a166-988b98cf8f49\") " pod="e2e-models-e2e-worker-w1-f9f277/distinct-2-w1-f9f277-kserve-6c49cf5dcb-h4x74" Oct 30 20:58:59.422856 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:58:59.422839 5378 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9eb6ff54-b77b-453c-a166-988b98cf8f49-model-cache\") pod \"distinct-2-w1-f9f277-kserve-6c49cf5dcb-h4x74\" (UID: \"9eb6ff54-b77b-453c-a166-988b98cf8f49\") " pod="e2e-models-e2e-worker-w1-f9f277/distinct-2-w1-f9f277-kserve-6c49cf5dcb-h4x74" Oct 30 20:58:59.480940 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:58:59.480909 5378 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-f9f277/distinct-2-w1-f9f277-kserve-6c49cf5dcb-h4x74"] Oct 30 20:58:59.520774 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:58:59.520747 5378 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w1-f9f277\"/\"distinct-2-w1-f9f277-kserve-self-signed-certs\"" Oct 30 20:58:59.524274 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:58:59.524248 5378 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9eb6ff54-b77b-453c-a166-988b98cf8f49-tls-certs\") pod \"distinct-2-w1-f9f277-kserve-6c49cf5dcb-h4x74\" (UID: \"9eb6ff54-b77b-453c-a166-988b98cf8f49\") " pod="e2e-models-e2e-worker-w1-f9f277/distinct-2-w1-f9f277-kserve-6c49cf5dcb-h4x74" Oct 30 20:58:59.524388 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:58:59.524291 5378 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fbz2k\" (UniqueName: \"kubernetes.io/projected/9eb6ff54-b77b-453c-a166-988b98cf8f49-kube-api-access-fbz2k\") pod \"distinct-2-w1-f9f277-kserve-6c49cf5dcb-h4x74\" (UID: \"9eb6ff54-b77b-453c-a166-988b98cf8f49\") " pod="e2e-models-e2e-worker-w1-f9f277/distinct-2-w1-f9f277-kserve-6c49cf5dcb-h4x74" Oct 30 20:58:59.524388 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:58:59.524324 5378 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9eb6ff54-b77b-453c-a166-988b98cf8f49-model-cache\") pod \"distinct-2-w1-f9f277-kserve-6c49cf5dcb-h4x74\" (UID: \"9eb6ff54-b77b-453c-a166-988b98cf8f49\") " pod="e2e-models-e2e-worker-w1-f9f277/distinct-2-w1-f9f277-kserve-6c49cf5dcb-h4x74" Oct 30 20:58:59.524527 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:58:59.524440 5378 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9eb6ff54-b77b-453c-a166-988b98cf8f49-tmp-dir\") pod \"distinct-2-w1-f9f277-kserve-6c49cf5dcb-h4x74\" (UID: \"9eb6ff54-b77b-453c-a166-988b98cf8f49\") " pod="e2e-models-e2e-worker-w1-f9f277/distinct-2-w1-f9f277-kserve-6c49cf5dcb-h4x74" Oct 30 20:58:59.524527 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:58:59.524522 5378 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9eb6ff54-b77b-453c-a166-988b98cf8f49-dshm\") pod \"distinct-2-w1-f9f277-kserve-6c49cf5dcb-h4x74\" (UID: \"9eb6ff54-b77b-453c-a166-988b98cf8f49\") " pod="e2e-models-e2e-worker-w1-f9f277/distinct-2-w1-f9f277-kserve-6c49cf5dcb-h4x74" Oct 30 20:58:59.524675 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:58:59.524581 5378 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9eb6ff54-b77b-453c-a166-988b98cf8f49-home\") pod \"distinct-2-w1-f9f277-kserve-6c49cf5dcb-h4x74\" (UID: \"9eb6ff54-b77b-453c-a166-988b98cf8f49\") " pod="e2e-models-e2e-worker-w1-f9f277/distinct-2-w1-f9f277-kserve-6c49cf5dcb-h4x74" Oct 30 20:58:59.524778 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:58:59.524761 5378 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9eb6ff54-b77b-453c-a166-988b98cf8f49-model-cache\") pod \"distinct-2-w1-f9f277-kserve-6c49cf5dcb-h4x74\" (UID: \"9eb6ff54-b77b-453c-a166-988b98cf8f49\") " pod="e2e-models-e2e-worker-w1-f9f277/distinct-2-w1-f9f277-kserve-6c49cf5dcb-h4x74" Oct 30 20:58:59.524848 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:58:59.524820 5378 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9eb6ff54-b77b-453c-a166-988b98cf8f49-tmp-dir\") pod \"distinct-2-w1-f9f277-kserve-6c49cf5dcb-h4x74\" (UID: \"9eb6ff54-b77b-453c-a166-988b98cf8f49\") " pod="e2e-models-e2e-worker-w1-f9f277/distinct-2-w1-f9f277-kserve-6c49cf5dcb-h4x74" Oct 30 20:58:59.524943 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:58:59.524927 5378 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9eb6ff54-b77b-453c-a166-988b98cf8f49-home\") pod \"distinct-2-w1-f9f277-kserve-6c49cf5dcb-h4x74\" (UID: \"9eb6ff54-b77b-453c-a166-988b98cf8f49\") " pod="e2e-models-e2e-worker-w1-f9f277/distinct-2-w1-f9f277-kserve-6c49cf5dcb-h4x74" Oct 30 20:58:59.526625 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:58:59.526604 5378 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9eb6ff54-b77b-453c-a166-988b98cf8f49-dshm\") pod \"distinct-2-w1-f9f277-kserve-6c49cf5dcb-h4x74\" (UID: \"9eb6ff54-b77b-453c-a166-988b98cf8f49\") " pod="e2e-models-e2e-worker-w1-f9f277/distinct-2-w1-f9f277-kserve-6c49cf5dcb-h4x74" Oct 30 20:58:59.526879 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:58:59.526862 5378 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9eb6ff54-b77b-453c-a166-988b98cf8f49-tls-certs\") pod \"distinct-2-w1-f9f277-kserve-6c49cf5dcb-h4x74\" (UID: \"9eb6ff54-b77b-453c-a166-988b98cf8f49\") " pod="e2e-models-e2e-worker-w1-f9f277/distinct-2-w1-f9f277-kserve-6c49cf5dcb-h4x74" Oct 30 20:58:59.602718 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:58:59.602644 5378 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fbz2k\" (UniqueName: \"kubernetes.io/projected/9eb6ff54-b77b-453c-a166-988b98cf8f49-kube-api-access-fbz2k\") pod \"distinct-2-w1-f9f277-kserve-6c49cf5dcb-h4x74\" (UID: \"9eb6ff54-b77b-453c-a166-988b98cf8f49\") " pod="e2e-models-e2e-worker-w1-f9f277/distinct-2-w1-f9f277-kserve-6c49cf5dcb-h4x74" Oct 30 20:58:59.603522 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:58:59.603426 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 20:58:59.730460 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:58:59.730382 5378 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-f9f277/distinct-2-w1-f9f277-kserve-6c49cf5dcb-h4x74" Oct 30 20:59:00.172916 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:59:00.172889 5378 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-f9f277/distinct-2-w1-f9f277-kserve-6c49cf5dcb-h4x74"] Oct 30 20:59:00.173414 ip-10-0-138-149 kubenswrapper[5378]: W1030 20:59:00.173386 5378 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod9eb6ff54_b77b_453c_a166_988b98cf8f49.slice/crio-f345bf1c4e602939b92c2bade819f1827576f5bdaa1049a3ce7065c10153ba43 WatchSource:0}: Error finding container f345bf1c4e602939b92c2bade819f1827576f5bdaa1049a3ce7065c10153ba43: Status 404 returned error can't find the container with id f345bf1c4e602939b92c2bade819f1827576f5bdaa1049a3ce7065c10153ba43 Oct 30 20:59:01.052250 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:59:01.052214 5378 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-f9f277/distinct-2-w1-f9f277-kserve-6c49cf5dcb-h4x74" event={"ID":"9eb6ff54-b77b-453c-a166-988b98cf8f49","Type":"ContainerStarted","Data":"ac0610f90854b97e1bc8b4176433cada60c54acd61664c390b22d899d59017e0"} Oct 30 20:59:01.052666 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:59:01.052259 5378 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-f9f277/distinct-2-w1-f9f277-kserve-6c49cf5dcb-h4x74" event={"ID":"9eb6ff54-b77b-453c-a166-988b98cf8f49","Type":"ContainerStarted","Data":"f345bf1c4e602939b92c2bade819f1827576f5bdaa1049a3ce7065c10153ba43"} Oct 30 20:59:01.168052 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:59:01.167999 5378 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w1-f9f277/distinct-2-w1-f9f277-kserve-6c49cf5dcb-h4x74" podStartSLOduration=2.1679859710000002 podStartE2EDuration="2.167985971s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:58:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:59:01.167842818 +0000 UTC m=+1231.595956738" watchObservedRunningTime="2026-10-30 20:59:01.167985971 +0000 UTC m=+1231.596099890" Oct 30 20:59:01.731358 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:59:01.731327 5378 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-f9f277/distinct-2-w1-f9f277-kserve-6c49cf5dcb-h4x74" Oct 30 20:59:02.796956 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:59:02.796915 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 20:59:04.048741 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:59:04.048702 5378 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c0c05792-jrcz7"] Oct 30 20:59:04.099327 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:59:04.099291 5378 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c0c05792-jrcz7"] Oct 30 20:59:05.741255 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:59:05.741220 5378 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a5d5a88d-6977-4900-948e-7171ce1cb445" path="/var/lib/kubelet/pods/a5d5a88d-6977-4900-948e-7171ce1cb445/volumes" Oct 30 20:59:06.707809 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:59:06.707774 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 20:59:07.586293 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:59:07.586258 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 20:59:07.707436 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:59:07.707402 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 20:59:09.730521 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:59:09.730483 5378 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w1-f9f277/distinct-2-w1-f9f277-kserve-6c49cf5dcb-h4x74" Oct 30 20:59:09.743482 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:59:09.743456 5378 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w1-f9f277/distinct-2-w1-f9f277-kserve-6c49cf5dcb-h4x74" Oct 30 20:59:10.126235 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:59:10.126208 5378 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-f9f277/distinct-2-w1-f9f277-kserve-6c49cf5dcb-h4x74" Oct 30 20:59:12.632003 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:59:12.631970 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 20:59:16.761871 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:59:16.761836 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 20:59:16.887129 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:59:16.887092 5378 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w5-2b425d-openshift-default-f547dbcbf-wllbd"] Oct 30 20:59:16.891046 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:59:16.891024 5378 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-2b425d-openshift-default-f547dbcbf-wllbd" Oct 30 20:59:16.948223 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:59:16.948188 5378 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ce8609a7-1869-4d3e-b1d7-adab0908fbb9-credential-socket\") pod \"e2e-worker-w5-2b425d-openshift-default-f547dbcbf-wllbd\" (UID: \"ce8609a7-1869-4d3e-b1d7-adab0908fbb9\") " pod="openshift-ingress/e2e-worker-w5-2b425d-openshift-default-f547dbcbf-wllbd" Oct 30 20:59:16.948223 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:59:16.948228 5378 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z8w2w\" (UniqueName: \"kubernetes.io/projected/ce8609a7-1869-4d3e-b1d7-adab0908fbb9-kube-api-access-z8w2w\") pod \"e2e-worker-w5-2b425d-openshift-default-f547dbcbf-wllbd\" (UID: \"ce8609a7-1869-4d3e-b1d7-adab0908fbb9\") " pod="openshift-ingress/e2e-worker-w5-2b425d-openshift-default-f547dbcbf-wllbd" Oct 30 20:59:16.948456 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:59:16.948253 5378 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ce8609a7-1869-4d3e-b1d7-adab0908fbb9-workload-socket\") pod \"e2e-worker-w5-2b425d-openshift-default-f547dbcbf-wllbd\" (UID: \"ce8609a7-1869-4d3e-b1d7-adab0908fbb9\") " pod="openshift-ingress/e2e-worker-w5-2b425d-openshift-default-f547dbcbf-wllbd" Oct 30 20:59:16.948456 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:59:16.948282 5378 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ce8609a7-1869-4d3e-b1d7-adab0908fbb9-workload-certs\") pod \"e2e-worker-w5-2b425d-openshift-default-f547dbcbf-wllbd\" (UID: \"ce8609a7-1869-4d3e-b1d7-adab0908fbb9\") " pod="openshift-ingress/e2e-worker-w5-2b425d-openshift-default-f547dbcbf-wllbd" Oct 30 20:59:16.948539 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:59:16.948461 5378 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ce8609a7-1869-4d3e-b1d7-adab0908fbb9-istiod-ca-cert\") pod \"e2e-worker-w5-2b425d-openshift-default-f547dbcbf-wllbd\" (UID: \"ce8609a7-1869-4d3e-b1d7-adab0908fbb9\") " pod="openshift-ingress/e2e-worker-w5-2b425d-openshift-default-f547dbcbf-wllbd" Oct 30 20:59:16.948539 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:59:16.948506 5378 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ce8609a7-1869-4d3e-b1d7-adab0908fbb9-istio-data\") pod \"e2e-worker-w5-2b425d-openshift-default-f547dbcbf-wllbd\" (UID: \"ce8609a7-1869-4d3e-b1d7-adab0908fbb9\") " pod="openshift-ingress/e2e-worker-w5-2b425d-openshift-default-f547dbcbf-wllbd" Oct 30 20:59:16.948637 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:59:16.948618 5378 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ce8609a7-1869-4d3e-b1d7-adab0908fbb9-istio-podinfo\") pod \"e2e-worker-w5-2b425d-openshift-default-f547dbcbf-wllbd\" (UID: \"ce8609a7-1869-4d3e-b1d7-adab0908fbb9\") " pod="openshift-ingress/e2e-worker-w5-2b425d-openshift-default-f547dbcbf-wllbd" Oct 30 20:59:16.948685 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:59:16.948673 5378 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ce8609a7-1869-4d3e-b1d7-adab0908fbb9-istio-token\") pod \"e2e-worker-w5-2b425d-openshift-default-f547dbcbf-wllbd\" (UID: \"ce8609a7-1869-4d3e-b1d7-adab0908fbb9\") " pod="openshift-ingress/e2e-worker-w5-2b425d-openshift-default-f547dbcbf-wllbd" Oct 30 20:59:16.948724 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:59:16.948699 5378 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ce8609a7-1869-4d3e-b1d7-adab0908fbb9-istio-envoy\") pod \"e2e-worker-w5-2b425d-openshift-default-f547dbcbf-wllbd\" (UID: \"ce8609a7-1869-4d3e-b1d7-adab0908fbb9\") " pod="openshift-ingress/e2e-worker-w5-2b425d-openshift-default-f547dbcbf-wllbd" Oct 30 20:59:16.998827 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:59:16.998795 5378 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-2b425d-openshift-default-f547dbcbf-wllbd"] Oct 30 20:59:17.049827 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:59:17.049743 5378 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ce8609a7-1869-4d3e-b1d7-adab0908fbb9-credential-socket\") pod \"e2e-worker-w5-2b425d-openshift-default-f547dbcbf-wllbd\" (UID: \"ce8609a7-1869-4d3e-b1d7-adab0908fbb9\") " pod="openshift-ingress/e2e-worker-w5-2b425d-openshift-default-f547dbcbf-wllbd" Oct 30 20:59:17.049827 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:59:17.049775 5378 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z8w2w\" (UniqueName: \"kubernetes.io/projected/ce8609a7-1869-4d3e-b1d7-adab0908fbb9-kube-api-access-z8w2w\") pod \"e2e-worker-w5-2b425d-openshift-default-f547dbcbf-wllbd\" (UID: \"ce8609a7-1869-4d3e-b1d7-adab0908fbb9\") " pod="openshift-ingress/e2e-worker-w5-2b425d-openshift-default-f547dbcbf-wllbd" Oct 30 20:59:17.049827 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:59:17.049799 5378 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ce8609a7-1869-4d3e-b1d7-adab0908fbb9-workload-socket\") pod \"e2e-worker-w5-2b425d-openshift-default-f547dbcbf-wllbd\" (UID: \"ce8609a7-1869-4d3e-b1d7-adab0908fbb9\") " pod="openshift-ingress/e2e-worker-w5-2b425d-openshift-default-f547dbcbf-wllbd" Oct 30 20:59:17.049827 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:59:17.049825 5378 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ce8609a7-1869-4d3e-b1d7-adab0908fbb9-workload-certs\") pod \"e2e-worker-w5-2b425d-openshift-default-f547dbcbf-wllbd\" (UID: \"ce8609a7-1869-4d3e-b1d7-adab0908fbb9\") " pod="openshift-ingress/e2e-worker-w5-2b425d-openshift-default-f547dbcbf-wllbd" Oct 30 20:59:17.050222 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:59:17.049905 5378 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ce8609a7-1869-4d3e-b1d7-adab0908fbb9-istiod-ca-cert\") pod \"e2e-worker-w5-2b425d-openshift-default-f547dbcbf-wllbd\" (UID: \"ce8609a7-1869-4d3e-b1d7-adab0908fbb9\") " pod="openshift-ingress/e2e-worker-w5-2b425d-openshift-default-f547dbcbf-wllbd" Oct 30 20:59:17.050222 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:59:17.049951 5378 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ce8609a7-1869-4d3e-b1d7-adab0908fbb9-istio-data\") pod \"e2e-worker-w5-2b425d-openshift-default-f547dbcbf-wllbd\" (UID: \"ce8609a7-1869-4d3e-b1d7-adab0908fbb9\") " pod="openshift-ingress/e2e-worker-w5-2b425d-openshift-default-f547dbcbf-wllbd" Oct 30 20:59:17.050222 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:59:17.049988 5378 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ce8609a7-1869-4d3e-b1d7-adab0908fbb9-istio-podinfo\") pod \"e2e-worker-w5-2b425d-openshift-default-f547dbcbf-wllbd\" (UID: \"ce8609a7-1869-4d3e-b1d7-adab0908fbb9\") " pod="openshift-ingress/e2e-worker-w5-2b425d-openshift-default-f547dbcbf-wllbd" Oct 30 20:59:17.050222 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:59:17.050041 5378 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ce8609a7-1869-4d3e-b1d7-adab0908fbb9-istio-token\") pod \"e2e-worker-w5-2b425d-openshift-default-f547dbcbf-wllbd\" (UID: \"ce8609a7-1869-4d3e-b1d7-adab0908fbb9\") " pod="openshift-ingress/e2e-worker-w5-2b425d-openshift-default-f547dbcbf-wllbd" Oct 30 20:59:17.050222 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:59:17.050074 5378 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ce8609a7-1869-4d3e-b1d7-adab0908fbb9-istio-envoy\") pod \"e2e-worker-w5-2b425d-openshift-default-f547dbcbf-wllbd\" (UID: \"ce8609a7-1869-4d3e-b1d7-adab0908fbb9\") " pod="openshift-ingress/e2e-worker-w5-2b425d-openshift-default-f547dbcbf-wllbd" Oct 30 20:59:17.050222 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:59:17.050183 5378 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ce8609a7-1869-4d3e-b1d7-adab0908fbb9-credential-socket\") pod \"e2e-worker-w5-2b425d-openshift-default-f547dbcbf-wllbd\" (UID: \"ce8609a7-1869-4d3e-b1d7-adab0908fbb9\") " pod="openshift-ingress/e2e-worker-w5-2b425d-openshift-default-f547dbcbf-wllbd" Oct 30 20:59:17.050525 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:59:17.050346 5378 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ce8609a7-1869-4d3e-b1d7-adab0908fbb9-istio-data\") pod \"e2e-worker-w5-2b425d-openshift-default-f547dbcbf-wllbd\" (UID: \"ce8609a7-1869-4d3e-b1d7-adab0908fbb9\") " pod="openshift-ingress/e2e-worker-w5-2b425d-openshift-default-f547dbcbf-wllbd" Oct 30 20:59:17.050525 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:59:17.050381 5378 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ce8609a7-1869-4d3e-b1d7-adab0908fbb9-workload-socket\") pod \"e2e-worker-w5-2b425d-openshift-default-f547dbcbf-wllbd\" (UID: \"ce8609a7-1869-4d3e-b1d7-adab0908fbb9\") " pod="openshift-ingress/e2e-worker-w5-2b425d-openshift-default-f547dbcbf-wllbd" Oct 30 20:59:17.050683 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:59:17.050666 5378 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ce8609a7-1869-4d3e-b1d7-adab0908fbb9-workload-certs\") pod \"e2e-worker-w5-2b425d-openshift-default-f547dbcbf-wllbd\" (UID: \"ce8609a7-1869-4d3e-b1d7-adab0908fbb9\") " pod="openshift-ingress/e2e-worker-w5-2b425d-openshift-default-f547dbcbf-wllbd" Oct 30 20:59:17.050847 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:59:17.050821 5378 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ce8609a7-1869-4d3e-b1d7-adab0908fbb9-istiod-ca-cert\") pod \"e2e-worker-w5-2b425d-openshift-default-f547dbcbf-wllbd\" (UID: \"ce8609a7-1869-4d3e-b1d7-adab0908fbb9\") " pod="openshift-ingress/e2e-worker-w5-2b425d-openshift-default-f547dbcbf-wllbd" Oct 30 20:59:17.052446 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:59:17.052421 5378 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ce8609a7-1869-4d3e-b1d7-adab0908fbb9-istio-envoy\") pod \"e2e-worker-w5-2b425d-openshift-default-f547dbcbf-wllbd\" (UID: \"ce8609a7-1869-4d3e-b1d7-adab0908fbb9\") " pod="openshift-ingress/e2e-worker-w5-2b425d-openshift-default-f547dbcbf-wllbd" Oct 30 20:59:17.052706 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:59:17.052688 5378 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ce8609a7-1869-4d3e-b1d7-adab0908fbb9-istio-podinfo\") pod \"e2e-worker-w5-2b425d-openshift-default-f547dbcbf-wllbd\" (UID: \"ce8609a7-1869-4d3e-b1d7-adab0908fbb9\") " pod="openshift-ingress/e2e-worker-w5-2b425d-openshift-default-f547dbcbf-wllbd" Oct 30 20:59:17.107934 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:59:17.107895 5378 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ce8609a7-1869-4d3e-b1d7-adab0908fbb9-istio-token\") pod \"e2e-worker-w5-2b425d-openshift-default-f547dbcbf-wllbd\" (UID: \"ce8609a7-1869-4d3e-b1d7-adab0908fbb9\") " pod="openshift-ingress/e2e-worker-w5-2b425d-openshift-default-f547dbcbf-wllbd" Oct 30 20:59:17.108062 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:59:17.108005 5378 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z8w2w\" (UniqueName: \"kubernetes.io/projected/ce8609a7-1869-4d3e-b1d7-adab0908fbb9-kube-api-access-z8w2w\") pod \"e2e-worker-w5-2b425d-openshift-default-f547dbcbf-wllbd\" (UID: \"ce8609a7-1869-4d3e-b1d7-adab0908fbb9\") " pod="openshift-ingress/e2e-worker-w5-2b425d-openshift-default-f547dbcbf-wllbd" Oct 30 20:59:17.205485 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:59:17.205457 5378 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-2b425d-openshift-default-f547dbcbf-wllbd" Oct 30 20:59:17.242723 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:59:17.242689 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 20:59:17.631453 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:59:17.631427 5378 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-2b425d-openshift-default-f547dbcbf-wllbd"] Oct 30 20:59:17.632072 ip-10-0-138-149 kubenswrapper[5378]: W1030 20:59:17.632042 5378 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podce8609a7_1869_4d3e_b1d7_adab0908fbb9.slice/crio-825c240375b0ed6aed7fa079d19dcf7d31217064d1e8ec1e8e29af5f4f180d86 WatchSource:0}: Error finding container 825c240375b0ed6aed7fa079d19dcf7d31217064d1e8ec1e8e29af5f4f180d86: Status 404 returned error can't find the container with id 825c240375b0ed6aed7fa079d19dcf7d31217064d1e8ec1e8e29af5f4f180d86 Oct 30 20:59:17.634660 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:59:17.634631 5378 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 20:59:17.634746 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:59:17.634698 5378 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 20:59:17.634746 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:59:17.634726 5378 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 20:59:18.185088 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:59:18.185049 5378 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-2b425d-openshift-default-f547dbcbf-wllbd" event={"ID":"ce8609a7-1869-4d3e-b1d7-adab0908fbb9","Type":"ContainerStarted","Data":"9e0befd66939484f1b072e143f08f7c1abdabd630a986edd3396fb5315945ddb"} Oct 30 20:59:18.185476 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:59:18.185111 5378 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-2b425d-openshift-default-f547dbcbf-wllbd" event={"ID":"ce8609a7-1869-4d3e-b1d7-adab0908fbb9","Type":"ContainerStarted","Data":"825c240375b0ed6aed7fa079d19dcf7d31217064d1e8ec1e8e29af5f4f180d86"} Oct 30 20:59:18.206558 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:59:18.206533 5378 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w5-2b425d-openshift-default-f547dbcbf-wllbd" Oct 30 20:59:18.208181 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:59:18.208151 5378 patch_prober.go:29] interesting pod/e2e-worker-w5-2b425d-openshift-default-f547dbcbf-wllbd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused" start-of-body= Oct 30 20:59:18.208308 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:59:18.208206 5378 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-2b425d-openshift-default-f547dbcbf-wllbd" podUID="ce8609a7-1869-4d3e-b1d7-adab0908fbb9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused" Oct 30 20:59:18.305134 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:59:18.305075 5378 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w5-2b425d-openshift-default-f547dbcbf-wllbd" podStartSLOduration=2.305057347 podStartE2EDuration="2.305057347s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:59:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:59:18.30381142 +0000 UTC m=+1248.731925342" watchObservedRunningTime="2026-10-30 20:59:18.305057347 +0000 UTC m=+1248.733171267" Oct 30 20:59:18.837687 ip-10-0-138-149 kubenswrapper[5378]: E1030 20:59:18.837653 5378 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q6mx7" containerName="extractor" Oct 30 20:59:19.205922 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:59:19.205886 5378 patch_prober.go:29] interesting pod/e2e-worker-w5-2b425d-openshift-default-f547dbcbf-wllbd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused" start-of-body= Oct 30 20:59:19.206328 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:59:19.205946 5378 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-2b425d-openshift-default-f547dbcbf-wllbd" podUID="ce8609a7-1869-4d3e-b1d7-adab0908fbb9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused" Oct 30 20:59:20.206005 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:59:20.205973 5378 patch_prober.go:29] interesting pod/e2e-worker-w5-2b425d-openshift-default-f547dbcbf-wllbd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused" start-of-body= Oct 30 20:59:20.206407 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:59:20.206044 5378 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-2b425d-openshift-default-f547dbcbf-wllbd" podUID="ce8609a7-1869-4d3e-b1d7-adab0908fbb9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused" Oct 30 20:59:20.281493 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:59:20.281442 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 20:59:21.207497 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:59:21.207454 5378 patch_prober.go:29] interesting pod/e2e-worker-w5-2b425d-openshift-default-f547dbcbf-wllbd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused" start-of-body= Oct 30 20:59:21.207999 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:59:21.207515 5378 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-2b425d-openshift-default-f547dbcbf-wllbd" podUID="ce8609a7-1869-4d3e-b1d7-adab0908fbb9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused" Oct 30 20:59:22.202254 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:59:22.202217 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 20:59:22.206408 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:59:22.206382 5378 patch_prober.go:29] interesting pod/e2e-worker-w5-2b425d-openshift-default-f547dbcbf-wllbd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused" start-of-body= Oct 30 20:59:22.206546 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:59:22.206435 5378 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-2b425d-openshift-default-f547dbcbf-wllbd" podUID="ce8609a7-1869-4d3e-b1d7-adab0908fbb9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused" Oct 30 20:59:23.206529 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:59:23.206493 5378 patch_prober.go:29] interesting pod/e2e-worker-w5-2b425d-openshift-default-f547dbcbf-wllbd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused" start-of-body= Oct 30 20:59:23.206962 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:59:23.206548 5378 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-2b425d-openshift-default-f547dbcbf-wllbd" podUID="ce8609a7-1869-4d3e-b1d7-adab0908fbb9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused" Oct 30 20:59:24.206780 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:59:24.206744 5378 patch_prober.go:29] interesting pod/e2e-worker-w5-2b425d-openshift-default-f547dbcbf-wllbd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused" start-of-body= Oct 30 20:59:24.207174 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:59:24.206796 5378 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-2b425d-openshift-default-f547dbcbf-wllbd" podUID="ce8609a7-1869-4d3e-b1d7-adab0908fbb9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused" Oct 30 20:59:25.211013 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:59:25.210982 5378 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w5-2b425d-openshift-default-f547dbcbf-wllbd" Oct 30 20:59:25.211477 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:59:25.211256 5378 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w5-2b425d-openshift-default-f547dbcbf-wllbd" Oct 30 20:59:25.211916 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:59:25.211895 5378 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w5-2b425d-openshift-default-f547dbcbf-wllbd" Oct 30 20:59:27.183436 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:59:27.183402 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 20:59:28.419313 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:59:28.419268 5378 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-r9g59"] Oct 30 20:59:30.773706 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:59:30.773655 5378 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-f9f277/distinct-2-w1-f9f277-kserve-6c49cf5dcb-h4x74" podUID="9eb6ff54-b77b-453c-a166-988b98cf8f49" containerName="main" probeResult="failure" output="Get \"https://10.133.0.76:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:59:30.799705 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:59:30.799661 5378 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-d061d0/distinct-w2-d061d0-kserve-679c7cd7c4-8czrv" podUID="2c139b9f-c7df-4675-b431-95d4136095e5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:59:30.952591 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:59:30.951646 5378 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-1fdaa7/distinct-w4-1fdaa7-kserve-86dc9774b9-6x2gf" podUID="2e162c95-0527-4cb3-85a3-795aae212d29" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:59:31.138588 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:59:31.137642 5378 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-f9f277/simulator-w1-f9f277-kserve-5d9685f85c-2h78x" podUID="dc662e9d-914c-4dfb-a86f-0fb048f7b690" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:59:31.138588 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:59:31.138043 5378 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-f9f277/distinct-2-w1-f9f277-kserve-6c49cf5dcb-h4x74" podUID="9eb6ff54-b77b-453c-a166-988b98cf8f49" containerName="main" probeResult="failure" output="Get \"https://10.133.0.76:8000/ready\": context deadline exceeded" Oct 30 20:59:31.138870 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:59:31.138615 5378 patch_prober.go:29] interesting pod/e2e-worker-w4-1fdaa7-openshift-default-84586db7d5-sbpk8 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.44:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 20:59:31.138870 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:59:31.138667 5378 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-1fdaa7-openshift-default-84586db7d5-sbpk8" podUID="2391a719-0c9b-4c56-b229-8f286a9aac96" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.44:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 20:59:31.215694 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:59:31.215647 5378 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-8fa2a7/simulator-w6-8fa2a7-kserve-6cbf5bd894-lfpcq" podUID="f409f469-71fc-445b-bdc8-b8e84a4d1dc6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:59:31.351936 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:59:31.351891 5378 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-1fdaa7/simulator-w4-1fdaa7-kserve-9866d468c-fwmsk" podUID="09219832-010e-4ba8-bfb9-5a2884b358e1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:59:31.412650 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:59:31.411651 5378 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-d061d0/simulator-w2-d061d0-kserve-8484f9678b-lnszm" podUID="525fac02-b21c-4dd5-b553-72bcc36062c5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/ready\": context deadline exceeded" Oct 30 20:59:31.442588 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:59:31.441651 5378 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-781058/distinct-w0-781058-kserve-8bdf9bfcd-hzdfk" podUID="a9678fc2-1c56-4d45-9007-eee5049e0f48" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:59:31.690762 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:59:31.690649 5378 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-8fa2a7/premium-w6-8fa2a7-kserve-64bf4bf6c-dxrdr" podUID="484cba5f-b415-430a-b52d-50b1293be948" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:59:31.800700 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:59:31.800654 5378 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-1fdaa7/premium-w4-1fdaa7-kserve-55dffbd5cd-gvpxq" podUID="ec282a4b-742f-4382-a296-335dcd2e9636" containerName="main" probeResult="failure" output="Get \"https://10.133.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:59:31.802674 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:59:31.802640 5378 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-d061d0/distinct-w2-d061d0-kserve-679c7cd7c4-8czrv" podUID="2c139b9f-c7df-4675-b431-95d4136095e5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:59:31.803158 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:59:31.803138 5378 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-d061d0/distinct-w2-d061d0-kserve-679c7cd7c4-8czrv" Oct 30 20:59:31.973646 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:59:31.967373 5378 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-1fdaa7/distinct-w4-1fdaa7-kserve-86dc9774b9-6x2gf" podUID="2e162c95-0527-4cb3-85a3-795aae212d29" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/ready\": context deadline exceeded" Oct 30 20:59:31.973646 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:59:31.968035 5378 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-1fdaa7/distinct-w4-1fdaa7-kserve-86dc9774b9-6x2gf" Oct 30 20:59:32.093589 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:59:32.083701 5378 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-781058/distinct-w0-781058-kserve-8bdf9bfcd-hzdfk" podUID="a9678fc2-1c56-4d45-9007-eee5049e0f48" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:59:32.219590 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:59:32.145662 5378 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-f9f277/distinct-2-w1-f9f277-kserve-6c49cf5dcb-h4x74" podUID="9eb6ff54-b77b-453c-a166-988b98cf8f49" containerName="main" probeResult="failure" output="Get \"https://10.133.0.76:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:59:32.219590 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:59:32.145703 5378 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-f9f277/simulator-w1-f9f277-kserve-5d9685f85c-2h78x" podUID="dc662e9d-914c-4dfb-a86f-0fb048f7b690" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": context deadline exceeded" Oct 30 20:59:32.219590 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:59:32.147483 5378 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-f9f277/simulator-w1-f9f277-kserve-5d9685f85c-2h78x" Oct 30 20:59:32.219590 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:59:32.172669 5378 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-f9f277/distinct-2-w1-f9f277-kserve-6c49cf5dcb-h4x74" Oct 30 20:59:32.229282 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:59:32.220040 5378 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-8fa2a7/simulator-w6-8fa2a7-kserve-6cbf5bd894-lfpcq" podUID="f409f469-71fc-445b-bdc8-b8e84a4d1dc6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:59:32.229282 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:59:32.220455 5378 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-8fa2a7/simulator-w6-8fa2a7-kserve-6cbf5bd894-lfpcq" Oct 30 20:59:32.364999 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:59:32.352822 5378 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-1fdaa7/simulator-w4-1fdaa7-kserve-9866d468c-fwmsk" podUID="09219832-010e-4ba8-bfb9-5a2884b358e1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:59:32.364999 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:59:32.359605 5378 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-1fdaa7/simulator-w4-1fdaa7-kserve-9866d468c-fwmsk" Oct 30 20:59:32.432590 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:59:32.431987 5378 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-d061d0/simulator-w2-d061d0-kserve-8484f9678b-lnszm" podUID="525fac02-b21c-4dd5-b553-72bcc36062c5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:59:32.432590 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:59:32.432367 5378 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-d061d0/simulator-w2-d061d0-kserve-8484f9678b-lnszm" Oct 30 20:59:32.470595 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:59:32.443302 5378 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-781058/distinct-w0-781058-kserve-8bdf9bfcd-hzdfk" podUID="a9678fc2-1c56-4d45-9007-eee5049e0f48" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:59:32.509648 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:59:32.471202 5378 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-781058/distinct-w0-781058-kserve-8bdf9bfcd-hzdfk" Oct 30 20:59:32.824634 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:59:32.795723 5378 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-d061d0/distinct-w2-d061d0-kserve-679c7cd7c4-8czrv" Oct 30 20:59:32.947588 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:59:32.947062 5378 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-781058-openshift-default-55fb495b7-hpgv5"] Oct 30 20:59:32.965920 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:59:32.802452 5378 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-8fa2a7/premium-w6-8fa2a7-kserve-64bf4bf6c-dxrdr" podUID="484cba5f-b415-430a-b52d-50b1293be948" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:59:32.969355 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:59:32.969312 5378 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-1fdaa7/distinct-w4-1fdaa7-kserve-86dc9774b9-6x2gf" podUID="2e162c95-0527-4cb3-85a3-795aae212d29" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/ready\": context deadline exceeded" Oct 30 20:59:32.975608 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:59:32.803359 5378 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-1fdaa7/premium-w4-1fdaa7-kserve-55dffbd5cd-gvpxq" podUID="ec282a4b-742f-4382-a296-335dcd2e9636" containerName="main" probeResult="failure" output="Get \"https://10.133.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:59:33.052633 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:59:33.048708 5378 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-781058-openshift-default-55fb495b7-hpgv5"] Oct 30 20:59:33.052633 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:59:33.048849 5378 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-f9f277/distinct-2-w1-f9f277-kserve-6c49cf5dcb-h4x74" Oct 30 20:59:33.052633 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:59:33.048923 5378 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-8fa2a7/premium-w6-8fa2a7-kserve-64bf4bf6c-dxrdr" Oct 30 20:59:33.052633 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:59:33.049070 5378 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-1fdaa7/premium-w4-1fdaa7-kserve-55dffbd5cd-gvpxq" Oct 30 20:59:33.052633 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:59:33.050182 5378 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-781058-openshift-default-55fb495b7-hpgv5" Oct 30 20:59:33.052633 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:59:33.051940 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 20:59:33.220315 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:59:33.220258 5378 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d43ee748-feb2-4556-be54-3662f8dfa98d-istio-envoy\") pod \"e2e-worker-w0-781058-openshift-default-55fb495b7-hpgv5\" (UID: \"d43ee748-feb2-4556-be54-3662f8dfa98d\") " pod="openshift-ingress/e2e-worker-w0-781058-openshift-default-55fb495b7-hpgv5" Oct 30 20:59:33.220551 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:59:33.220528 5378 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d43ee748-feb2-4556-be54-3662f8dfa98d-istio-token\") pod \"e2e-worker-w0-781058-openshift-default-55fb495b7-hpgv5\" (UID: \"d43ee748-feb2-4556-be54-3662f8dfa98d\") " pod="openshift-ingress/e2e-worker-w0-781058-openshift-default-55fb495b7-hpgv5" Oct 30 20:59:33.220733 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:59:33.220715 5378 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d43ee748-feb2-4556-be54-3662f8dfa98d-istiod-ca-cert\") pod \"e2e-worker-w0-781058-openshift-default-55fb495b7-hpgv5\" (UID: \"d43ee748-feb2-4556-be54-3662f8dfa98d\") " pod="openshift-ingress/e2e-worker-w0-781058-openshift-default-55fb495b7-hpgv5" Oct 30 20:59:33.220918 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:59:33.220898 5378 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d43ee748-feb2-4556-be54-3662f8dfa98d-istio-data\") pod \"e2e-worker-w0-781058-openshift-default-55fb495b7-hpgv5\" (UID: \"d43ee748-feb2-4556-be54-3662f8dfa98d\") " pod="openshift-ingress/e2e-worker-w0-781058-openshift-default-55fb495b7-hpgv5" Oct 30 20:59:33.221102 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:59:33.221082 5378 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d43ee748-feb2-4556-be54-3662f8dfa98d-workload-socket\") pod \"e2e-worker-w0-781058-openshift-default-55fb495b7-hpgv5\" (UID: \"d43ee748-feb2-4556-be54-3662f8dfa98d\") " pod="openshift-ingress/e2e-worker-w0-781058-openshift-default-55fb495b7-hpgv5" Oct 30 20:59:33.221320 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:59:33.221302 5378 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d43ee748-feb2-4556-be54-3662f8dfa98d-workload-certs\") pod \"e2e-worker-w0-781058-openshift-default-55fb495b7-hpgv5\" (UID: \"d43ee748-feb2-4556-be54-3662f8dfa98d\") " pod="openshift-ingress/e2e-worker-w0-781058-openshift-default-55fb495b7-hpgv5" Oct 30 20:59:33.221440 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:59:33.221425 5378 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d43ee748-feb2-4556-be54-3662f8dfa98d-istio-podinfo\") pod \"e2e-worker-w0-781058-openshift-default-55fb495b7-hpgv5\" (UID: \"d43ee748-feb2-4556-be54-3662f8dfa98d\") " pod="openshift-ingress/e2e-worker-w0-781058-openshift-default-55fb495b7-hpgv5" Oct 30 20:59:33.224692 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:59:33.224653 5378 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-8fa2a7/simulator-w6-8fa2a7-kserve-6cbf5bd894-lfpcq" podUID="f409f469-71fc-445b-bdc8-b8e84a4d1dc6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:59:33.241451 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:59:33.234989 5378 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-f9f277/simulator-w1-f9f277-kserve-5d9685f85c-2h78x" podUID="dc662e9d-914c-4dfb-a86f-0fb048f7b690" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:59:33.241451 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:59:33.240053 5378 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-1fdaa7/simulator-w4-1fdaa7-kserve-9866d468c-fwmsk" Oct 30 20:59:33.252551 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:59:33.252508 5378 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-8fa2a7/premium-w6-8fa2a7-kserve-64bf4bf6c-dxrdr" Oct 30 20:59:33.256592 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:59:33.255753 5378 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d43ee748-feb2-4556-be54-3662f8dfa98d-credential-socket\") pod \"e2e-worker-w0-781058-openshift-default-55fb495b7-hpgv5\" (UID: \"d43ee748-feb2-4556-be54-3662f8dfa98d\") " pod="openshift-ingress/e2e-worker-w0-781058-openshift-default-55fb495b7-hpgv5" Oct 30 20:59:33.256592 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:59:33.255857 5378 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4fgvd\" (UniqueName: \"kubernetes.io/projected/d43ee748-feb2-4556-be54-3662f8dfa98d-kube-api-access-4fgvd\") pod \"e2e-worker-w0-781058-openshift-default-55fb495b7-hpgv5\" (UID: \"d43ee748-feb2-4556-be54-3662f8dfa98d\") " pod="openshift-ingress/e2e-worker-w0-781058-openshift-default-55fb495b7-hpgv5" Oct 30 20:59:33.265581 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:59:33.261141 5378 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-d061d0/simulator-w2-d061d0-kserve-8484f9678b-lnszm" Oct 30 20:59:33.275002 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:59:33.274977 5378 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-781058/distinct-w0-781058-kserve-8bdf9bfcd-hzdfk" Oct 30 20:59:33.283549 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:59:33.283528 5378 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-8fa2a7/simulator-w6-8fa2a7-kserve-6cbf5bd894-lfpcq" Oct 30 20:59:33.286835 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:59:33.286818 5378 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-1fdaa7/distinct-w4-1fdaa7-kserve-86dc9774b9-6x2gf" Oct 30 20:59:33.299462 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:59:33.299436 5378 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-1fdaa7/premium-w4-1fdaa7-kserve-55dffbd5cd-gvpxq" Oct 30 20:59:33.308266 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:59:33.308244 5378 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-f9f277/simulator-w1-f9f277-kserve-5d9685f85c-2h78x" Oct 30 20:59:33.357657 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:59:33.357621 5378 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4fgvd\" (UniqueName: \"kubernetes.io/projected/d43ee748-feb2-4556-be54-3662f8dfa98d-kube-api-access-4fgvd\") pod \"e2e-worker-w0-781058-openshift-default-55fb495b7-hpgv5\" (UID: \"d43ee748-feb2-4556-be54-3662f8dfa98d\") " pod="openshift-ingress/e2e-worker-w0-781058-openshift-default-55fb495b7-hpgv5" Oct 30 20:59:33.357856 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:59:33.357710 5378 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d43ee748-feb2-4556-be54-3662f8dfa98d-istio-envoy\") pod \"e2e-worker-w0-781058-openshift-default-55fb495b7-hpgv5\" (UID: \"d43ee748-feb2-4556-be54-3662f8dfa98d\") " pod="openshift-ingress/e2e-worker-w0-781058-openshift-default-55fb495b7-hpgv5" Oct 30 20:59:33.358025 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:59:33.357998 5378 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d43ee748-feb2-4556-be54-3662f8dfa98d-istio-token\") pod \"e2e-worker-w0-781058-openshift-default-55fb495b7-hpgv5\" (UID: \"d43ee748-feb2-4556-be54-3662f8dfa98d\") " pod="openshift-ingress/e2e-worker-w0-781058-openshift-default-55fb495b7-hpgv5" Oct 30 20:59:33.358110 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:59:33.358059 5378 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d43ee748-feb2-4556-be54-3662f8dfa98d-istiod-ca-cert\") pod \"e2e-worker-w0-781058-openshift-default-55fb495b7-hpgv5\" (UID: \"d43ee748-feb2-4556-be54-3662f8dfa98d\") " pod="openshift-ingress/e2e-worker-w0-781058-openshift-default-55fb495b7-hpgv5" Oct 30 20:59:33.358150 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:59:33.358119 5378 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d43ee748-feb2-4556-be54-3662f8dfa98d-istio-data\") pod \"e2e-worker-w0-781058-openshift-default-55fb495b7-hpgv5\" (UID: \"d43ee748-feb2-4556-be54-3662f8dfa98d\") " pod="openshift-ingress/e2e-worker-w0-781058-openshift-default-55fb495b7-hpgv5" Oct 30 20:59:33.358207 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:59:33.358189 5378 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d43ee748-feb2-4556-be54-3662f8dfa98d-workload-socket\") pod \"e2e-worker-w0-781058-openshift-default-55fb495b7-hpgv5\" (UID: \"d43ee748-feb2-4556-be54-3662f8dfa98d\") " pod="openshift-ingress/e2e-worker-w0-781058-openshift-default-55fb495b7-hpgv5" Oct 30 20:59:33.358340 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:59:33.358322 5378 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d43ee748-feb2-4556-be54-3662f8dfa98d-workload-certs\") pod \"e2e-worker-w0-781058-openshift-default-55fb495b7-hpgv5\" (UID: \"d43ee748-feb2-4556-be54-3662f8dfa98d\") " pod="openshift-ingress/e2e-worker-w0-781058-openshift-default-55fb495b7-hpgv5" Oct 30 20:59:33.358424 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:59:33.358355 5378 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d43ee748-feb2-4556-be54-3662f8dfa98d-istio-podinfo\") pod \"e2e-worker-w0-781058-openshift-default-55fb495b7-hpgv5\" (UID: \"d43ee748-feb2-4556-be54-3662f8dfa98d\") " pod="openshift-ingress/e2e-worker-w0-781058-openshift-default-55fb495b7-hpgv5" Oct 30 20:59:33.358489 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:59:33.358462 5378 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d43ee748-feb2-4556-be54-3662f8dfa98d-credential-socket\") pod \"e2e-worker-w0-781058-openshift-default-55fb495b7-hpgv5\" (UID: \"d43ee748-feb2-4556-be54-3662f8dfa98d\") " pod="openshift-ingress/e2e-worker-w0-781058-openshift-default-55fb495b7-hpgv5" Oct 30 20:59:33.358869 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:59:33.358842 5378 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d43ee748-feb2-4556-be54-3662f8dfa98d-workload-certs\") pod \"e2e-worker-w0-781058-openshift-default-55fb495b7-hpgv5\" (UID: \"d43ee748-feb2-4556-be54-3662f8dfa98d\") " pod="openshift-ingress/e2e-worker-w0-781058-openshift-default-55fb495b7-hpgv5" Oct 30 20:59:33.358970 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:59:33.358909 5378 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d43ee748-feb2-4556-be54-3662f8dfa98d-istiod-ca-cert\") pod \"e2e-worker-w0-781058-openshift-default-55fb495b7-hpgv5\" (UID: \"d43ee748-feb2-4556-be54-3662f8dfa98d\") " pod="openshift-ingress/e2e-worker-w0-781058-openshift-default-55fb495b7-hpgv5" Oct 30 20:59:33.359326 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:59:33.359255 5378 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d43ee748-feb2-4556-be54-3662f8dfa98d-workload-socket\") pod \"e2e-worker-w0-781058-openshift-default-55fb495b7-hpgv5\" (UID: \"d43ee748-feb2-4556-be54-3662f8dfa98d\") " pod="openshift-ingress/e2e-worker-w0-781058-openshift-default-55fb495b7-hpgv5" Oct 30 20:59:33.359326 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:59:33.359291 5378 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d43ee748-feb2-4556-be54-3662f8dfa98d-istio-data\") pod \"e2e-worker-w0-781058-openshift-default-55fb495b7-hpgv5\" (UID: \"d43ee748-feb2-4556-be54-3662f8dfa98d\") " pod="openshift-ingress/e2e-worker-w0-781058-openshift-default-55fb495b7-hpgv5" Oct 30 20:59:33.359326 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:59:33.359298 5378 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d43ee748-feb2-4556-be54-3662f8dfa98d-credential-socket\") pod \"e2e-worker-w0-781058-openshift-default-55fb495b7-hpgv5\" (UID: \"d43ee748-feb2-4556-be54-3662f8dfa98d\") " pod="openshift-ingress/e2e-worker-w0-781058-openshift-default-55fb495b7-hpgv5" Oct 30 20:59:33.361307 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:59:33.361284 5378 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d43ee748-feb2-4556-be54-3662f8dfa98d-istio-envoy\") pod \"e2e-worker-w0-781058-openshift-default-55fb495b7-hpgv5\" (UID: \"d43ee748-feb2-4556-be54-3662f8dfa98d\") " pod="openshift-ingress/e2e-worker-w0-781058-openshift-default-55fb495b7-hpgv5" Oct 30 20:59:33.361797 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:59:33.361779 5378 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d43ee748-feb2-4556-be54-3662f8dfa98d-istio-podinfo\") pod \"e2e-worker-w0-781058-openshift-default-55fb495b7-hpgv5\" (UID: \"d43ee748-feb2-4556-be54-3662f8dfa98d\") " pod="openshift-ingress/e2e-worker-w0-781058-openshift-default-55fb495b7-hpgv5" Oct 30 20:59:33.416643 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:59:33.416066 5378 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d43ee748-feb2-4556-be54-3662f8dfa98d-istio-token\") pod \"e2e-worker-w0-781058-openshift-default-55fb495b7-hpgv5\" (UID: \"d43ee748-feb2-4556-be54-3662f8dfa98d\") " pod="openshift-ingress/e2e-worker-w0-781058-openshift-default-55fb495b7-hpgv5" Oct 30 20:59:33.416643 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:59:33.416213 5378 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4fgvd\" (UniqueName: \"kubernetes.io/projected/d43ee748-feb2-4556-be54-3662f8dfa98d-kube-api-access-4fgvd\") pod \"e2e-worker-w0-781058-openshift-default-55fb495b7-hpgv5\" (UID: \"d43ee748-feb2-4556-be54-3662f8dfa98d\") " pod="openshift-ingress/e2e-worker-w0-781058-openshift-default-55fb495b7-hpgv5" Oct 30 20:59:33.546286 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:59:33.546243 5378 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-781058-openshift-default-55fb495b7-hpgv5" Oct 30 20:59:33.986080 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:59:33.986048 5378 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-781058-openshift-default-55fb495b7-hpgv5"] Oct 30 20:59:33.986946 ip-10-0-138-149 kubenswrapper[5378]: W1030 20:59:33.986915 5378 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd43ee748_feb2_4556_be54_3662f8dfa98d.slice/crio-3bf9d6cf7c660dcd7f135f5c46ad910548ceae44eef66e936654ff2ead417481 WatchSource:0}: Error finding container 3bf9d6cf7c660dcd7f135f5c46ad910548ceae44eef66e936654ff2ead417481: Status 404 returned error can't find the container with id 3bf9d6cf7c660dcd7f135f5c46ad910548ceae44eef66e936654ff2ead417481 Oct 30 20:59:33.989735 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:59:33.989698 5378 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 20:59:33.989805 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:59:33.989788 5378 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 20:59:33.989845 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:59:33.989835 5378 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 20:59:34.450605 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:59:34.449615 5378 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-781058-openshift-default-55fb495b7-hpgv5" event={"ID":"d43ee748-feb2-4556-be54-3662f8dfa98d","Type":"ContainerStarted","Data":"2dfda93c6c9252b225103f327e91f54fc6eded5a91be4d9f9d7e8893160a3184"} Oct 30 20:59:34.450605 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:59:34.449700 5378 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-781058-openshift-default-55fb495b7-hpgv5" event={"ID":"d43ee748-feb2-4556-be54-3662f8dfa98d","Type":"ContainerStarted","Data":"3bf9d6cf7c660dcd7f135f5c46ad910548ceae44eef66e936654ff2ead417481"} Oct 30 20:59:34.546965 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:59:34.546936 5378 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-781058-openshift-default-55fb495b7-hpgv5" Oct 30 20:59:34.748548 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:59:34.748439 5378 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-781058-openshift-default-55fb495b7-hpgv5" podStartSLOduration=2.748424868 podStartE2EDuration="2.748424868s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:59:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:59:34.746628259 +0000 UTC m=+1265.174742179" watchObservedRunningTime="2026-10-30 20:59:34.748424868 +0000 UTC m=+1265.176538788" Oct 30 20:59:35.548065 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:59:35.548022 5378 patch_prober.go:29] interesting pod/e2e-worker-w0-781058-openshift-default-55fb495b7-hpgv5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused" start-of-body= Oct 30 20:59:35.548517 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:59:35.548084 5378 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-781058-openshift-default-55fb495b7-hpgv5" podUID="d43ee748-feb2-4556-be54-3662f8dfa98d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused" Oct 30 20:59:36.547405 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:59:36.547372 5378 patch_prober.go:29] interesting pod/e2e-worker-w0-781058-openshift-default-55fb495b7-hpgv5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused" start-of-body= Oct 30 20:59:36.547556 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:59:36.547426 5378 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-781058-openshift-default-55fb495b7-hpgv5" podUID="d43ee748-feb2-4556-be54-3662f8dfa98d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused" Oct 30 20:59:37.546953 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:59:37.546922 5378 patch_prober.go:29] interesting pod/e2e-worker-w0-781058-openshift-default-55fb495b7-hpgv5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused" start-of-body= Oct 30 20:59:37.547392 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:59:37.546980 5378 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-781058-openshift-default-55fb495b7-hpgv5" podUID="d43ee748-feb2-4556-be54-3662f8dfa98d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused" Oct 30 20:59:38.111242 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:59:38.111203 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 20:59:38.546936 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:59:38.546829 5378 patch_prober.go:29] interesting pod/e2e-worker-w0-781058-openshift-default-55fb495b7-hpgv5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused" start-of-body= Oct 30 20:59:38.546936 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:59:38.546894 5378 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-781058-openshift-default-55fb495b7-hpgv5" podUID="d43ee748-feb2-4556-be54-3662f8dfa98d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused" Oct 30 20:59:39.547160 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:59:39.547124 5378 patch_prober.go:29] interesting pod/e2e-worker-w0-781058-openshift-default-55fb495b7-hpgv5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused" start-of-body= Oct 30 20:59:39.547546 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:59:39.547186 5378 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-781058-openshift-default-55fb495b7-hpgv5" podUID="d43ee748-feb2-4556-be54-3662f8dfa98d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused" Oct 30 20:59:40.546799 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:59:40.546763 5378 patch_prober.go:29] interesting pod/e2e-worker-w0-781058-openshift-default-55fb495b7-hpgv5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused" start-of-body= Oct 30 20:59:40.547095 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:59:40.546829 5378 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-781058-openshift-default-55fb495b7-hpgv5" podUID="d43ee748-feb2-4556-be54-3662f8dfa98d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused" Oct 30 20:59:41.550838 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:59:41.550804 5378 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-781058-openshift-default-55fb495b7-hpgv5" Oct 30 20:59:41.551284 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:59:41.551275 5378 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-781058-openshift-default-55fb495b7-hpgv5" Oct 30 20:59:41.551861 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:59:41.551838 5378 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-781058-openshift-default-55fb495b7-hpgv5" Oct 30 20:59:42.848376 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:59:42.848340 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 20:59:42.971480 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:59:42.971447 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 20:59:47.790653 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:59:47.790620 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 20:59:48.050848 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:59:48.050767 5378 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0adcf026-8wzrg"] Oct 30 20:59:48.100285 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:59:48.100257 5378 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0adcf026-8wzrg"] Oct 30 20:59:48.838108 ip-10-0-138-149 kubenswrapper[5378]: E1030 20:59:48.838075 5378 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q6mx7" containerName="extractor" Oct 30 20:59:49.742161 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:59:49.742123 5378 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7c00b187-6511-415b-bf45-1db52954347e" path="/var/lib/kubelet/pods/7c00b187-6511-415b-bf45-1db52954347e/volumes" Oct 30 20:59:52.584959 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:59:52.584926 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 20:59:57.461836 ip-10-0-138-149 kubenswrapper[5378]: I1030 20:59:57.461803 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:00:00.192153 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:00:00.192118 5378 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-f9f277-29889900-b7hrn"] Oct 30 21:00:00.196468 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:00:00.196446 5378 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889900-8slvz"] Oct 30 21:00:00.196595 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:00:00.196558 5378 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-f9f277-29889900-b7hrn" Oct 30 21:00:00.201252 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:00:00.201235 5378 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889900-8slvz" Oct 30 21:00:00.207463 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:00:00.207432 5378 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/372b312c-da86-453f-8287-fe4603fd018c-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-f9f277-29889900-b7hrn\" (UID: \"372b312c-da86-453f-8287-fe4603fd018c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-f9f277-29889900-b7hrn" Oct 30 21:00:00.207629 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:00:00.207600 5378 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vgkgb\" (UniqueName: \"kubernetes.io/projected/372b312c-da86-453f-8287-fe4603fd018c-kube-api-access-vgkgb\") pod \"maas-api-key-cleanup-e2e-worker-w1-f9f277-29889900-b7hrn\" (UID: \"372b312c-da86-453f-8287-fe4603fd018c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-f9f277-29889900-b7hrn" Oct 30 21:00:00.244584 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:00:00.243862 5378 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-682999-29889900-97hqz"] Oct 30 21:00:00.248694 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:00:00.248668 5378 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-682999-29889900-97hqz" Oct 30 21:00:00.256369 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:00:00.256343 5378 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-f9f277-29889900-b7hrn"] Oct 30 21:00:00.277859 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:00:00.277829 5378 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889900-8slvz"] Oct 30 21:00:00.305642 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:00:00.305611 5378 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-682999-29889900-97hqz"] Oct 30 21:00:00.345924 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:00:00.316399 5378 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cbc0e737-1972-4f73-8457-7081264afdb7-maas-api-service-ca\") pod \"maas-api-key-cleanup-29889900-8slvz\" (UID: \"cbc0e737-1972-4f73-8457-7081264afdb7\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889900-8slvz" Oct 30 21:00:00.348471 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:00:00.346469 5378 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-q2mgc\"" Oct 30 21:00:00.348640 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:00:00.348474 5378 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/372b312c-da86-453f-8287-fe4603fd018c-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-f9f277-29889900-b7hrn\" (UID: \"372b312c-da86-453f-8287-fe4603fd018c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-f9f277-29889900-b7hrn" Oct 30 21:00:00.348640 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:00:00.348549 5378 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n8mxj\" (UniqueName: \"kubernetes.io/projected/0e8b014e-a0de-474e-a123-01775c2e4c83-kube-api-access-n8mxj\") pod \"maas-api-key-cleanup-e2e-worker-w3-682999-29889900-97hqz\" (UID: \"0e8b014e-a0de-474e-a123-01775c2e4c83\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-682999-29889900-97hqz" Oct 30 21:00:00.348640 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:00:00.348623 5378 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0e8b014e-a0de-474e-a123-01775c2e4c83-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-682999-29889900-97hqz\" (UID: \"0e8b014e-a0de-474e-a123-01775c2e4c83\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-682999-29889900-97hqz" Oct 30 21:00:00.348808 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:00:00.348681 5378 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vgkgb\" (UniqueName: \"kubernetes.io/projected/372b312c-da86-453f-8287-fe4603fd018c-kube-api-access-vgkgb\") pod \"maas-api-key-cleanup-e2e-worker-w1-f9f277-29889900-b7hrn\" (UID: \"372b312c-da86-453f-8287-fe4603fd018c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-f9f277-29889900-b7hrn" Oct 30 21:00:00.348808 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:00:00.348777 5378 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7m9nx\" (UniqueName: \"kubernetes.io/projected/cbc0e737-1972-4f73-8457-7081264afdb7-kube-api-access-7m9nx\") pod \"maas-api-key-cleanup-29889900-8slvz\" (UID: \"cbc0e737-1972-4f73-8457-7081264afdb7\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889900-8slvz" Oct 30 21:00:00.349397 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:00:00.349366 5378 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/372b312c-da86-453f-8287-fe4603fd018c-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-f9f277-29889900-b7hrn\" (UID: \"372b312c-da86-453f-8287-fe4603fd018c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-f9f277-29889900-b7hrn" Oct 30 21:00:00.407713 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:00:00.407677 5378 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vgkgb\" (UniqueName: \"kubernetes.io/projected/372b312c-da86-453f-8287-fe4603fd018c-kube-api-access-vgkgb\") pod \"maas-api-key-cleanup-e2e-worker-w1-f9f277-29889900-b7hrn\" (UID: \"372b312c-da86-453f-8287-fe4603fd018c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-f9f277-29889900-b7hrn" Oct 30 21:00:00.449810 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:00:00.449732 5378 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0e8b014e-a0de-474e-a123-01775c2e4c83-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-682999-29889900-97hqz\" (UID: \"0e8b014e-a0de-474e-a123-01775c2e4c83\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-682999-29889900-97hqz" Oct 30 21:00:00.449958 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:00:00.449845 5378 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7m9nx\" (UniqueName: \"kubernetes.io/projected/cbc0e737-1972-4f73-8457-7081264afdb7-kube-api-access-7m9nx\") pod \"maas-api-key-cleanup-29889900-8slvz\" (UID: \"cbc0e737-1972-4f73-8457-7081264afdb7\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889900-8slvz" Oct 30 21:00:00.450022 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:00:00.449984 5378 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cbc0e737-1972-4f73-8457-7081264afdb7-maas-api-service-ca\") pod \"maas-api-key-cleanup-29889900-8slvz\" (UID: \"cbc0e737-1972-4f73-8457-7081264afdb7\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889900-8slvz" Oct 30 21:00:00.450107 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:00:00.450092 5378 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n8mxj\" (UniqueName: \"kubernetes.io/projected/0e8b014e-a0de-474e-a123-01775c2e4c83-kube-api-access-n8mxj\") pod \"maas-api-key-cleanup-e2e-worker-w3-682999-29889900-97hqz\" (UID: \"0e8b014e-a0de-474e-a123-01775c2e4c83\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-682999-29889900-97hqz" Oct 30 21:00:00.450434 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:00:00.450413 5378 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0e8b014e-a0de-474e-a123-01775c2e4c83-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-682999-29889900-97hqz\" (UID: \"0e8b014e-a0de-474e-a123-01775c2e4c83\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-682999-29889900-97hqz" Oct 30 21:00:00.450623 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:00:00.450606 5378 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cbc0e737-1972-4f73-8457-7081264afdb7-maas-api-service-ca\") pod \"maas-api-key-cleanup-29889900-8slvz\" (UID: \"cbc0e737-1972-4f73-8457-7081264afdb7\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889900-8slvz" Oct 30 21:00:00.508654 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:00:00.508627 5378 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n8mxj\" (UniqueName: \"kubernetes.io/projected/0e8b014e-a0de-474e-a123-01775c2e4c83-kube-api-access-n8mxj\") pod \"maas-api-key-cleanup-e2e-worker-w3-682999-29889900-97hqz\" (UID: \"0e8b014e-a0de-474e-a123-01775c2e4c83\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-682999-29889900-97hqz" Oct 30 21:00:00.508785 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:00:00.508686 5378 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7m9nx\" (UniqueName: \"kubernetes.io/projected/cbc0e737-1972-4f73-8457-7081264afdb7-kube-api-access-7m9nx\") pod \"maas-api-key-cleanup-29889900-8slvz\" (UID: \"cbc0e737-1972-4f73-8457-7081264afdb7\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889900-8slvz" Oct 30 21:00:00.512492 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:00:00.512473 5378 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-f9f277-29889900-b7hrn" Oct 30 21:00:00.519667 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:00:00.519647 5378 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889900-8slvz" Oct 30 21:00:00.570096 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:00:00.570064 5378 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-682999-29889900-97hqz" Oct 30 21:00:00.872211 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:00:00.872180 5378 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bd507c85-vz2br"] Oct 30 21:00:00.877257 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:00:00.877240 5378 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bd507c85-vz2br" Oct 30 21:00:00.889448 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:00:00.889427 5378 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-f9f277-29889900-b7hrn"] Oct 30 21:00:00.889619 ip-10-0-138-149 kubenswrapper[5378]: W1030 21:00:00.889599 5378 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod372b312c_da86_453f_8287_fe4603fd018c.slice/crio-578ad2bb20b1b315cf1b49a1f29589f6e21c3497a400941c067182dd9e802657 WatchSource:0}: Error finding container 578ad2bb20b1b315cf1b49a1f29589f6e21c3497a400941c067182dd9e802657: Status 404 returned error can't find the container with id 578ad2bb20b1b315cf1b49a1f29589f6e21c3497a400941c067182dd9e802657 Oct 30 21:00:00.930878 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:00:00.930855 5378 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 21:00:00.934615 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:00:00.934579 5378 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bd507c85-vz2br"] Oct 30 21:00:00.941825 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:00:00.941802 5378 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889900-8slvz"] Oct 30 21:00:00.942318 ip-10-0-138-149 kubenswrapper[5378]: W1030 21:00:00.942290 5378 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcbc0e737_1972_4f73_8457_7081264afdb7.slice/crio-9483a72071b3532b6616e43bfbb85ec1433da41cc7e8ff415cc3fdf8644590cb WatchSource:0}: Error finding container 9483a72071b3532b6616e43bfbb85ec1433da41cc7e8ff415cc3fdf8644590cb: Status 404 returned error can't find the container with id 9483a72071b3532b6616e43bfbb85ec1433da41cc7e8ff415cc3fdf8644590cb Oct 30 21:00:00.956811 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:00:00.956786 5378 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2b1ddcab-5592-429c-9329-b7c2a9bbdfe9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-bd507c85-vz2br\" (UID: \"2b1ddcab-5592-429c-9329-b7c2a9bbdfe9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bd507c85-vz2br" Oct 30 21:00:00.956921 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:00:00.956864 5378 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2b1ddcab-5592-429c-9329-b7c2a9bbdfe9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-bd507c85-vz2br\" (UID: \"2b1ddcab-5592-429c-9329-b7c2a9bbdfe9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bd507c85-vz2br" Oct 30 21:00:00.957124 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:00:00.957084 5378 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9b4fh\" (UniqueName: \"kubernetes.io/projected/2b1ddcab-5592-429c-9329-b7c2a9bbdfe9-kube-api-access-9b4fh\") pod \"maas-api-revoke-sub-bd507c85-vz2br\" (UID: \"2b1ddcab-5592-429c-9329-b7c2a9bbdfe9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bd507c85-vz2br" Oct 30 21:00:00.990466 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:00:00.990432 5378 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-682999-29889900-97hqz"] Oct 30 21:00:00.991665 ip-10-0-138-149 kubenswrapper[5378]: W1030 21:00:00.991643 5378 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0e8b014e_a0de_474e_a123_01775c2e4c83.slice/crio-a81a3502862688eaaead918208902f5402dc495ef5ef49146a1dfc941a7f1fb1 WatchSource:0}: Error finding container a81a3502862688eaaead918208902f5402dc495ef5ef49146a1dfc941a7f1fb1: Status 404 returned error can't find the container with id a81a3502862688eaaead918208902f5402dc495ef5ef49146a1dfc941a7f1fb1 Oct 30 21:00:01.059819 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:00:01.059774 5378 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2b1ddcab-5592-429c-9329-b7c2a9bbdfe9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-bd507c85-vz2br\" (UID: \"2b1ddcab-5592-429c-9329-b7c2a9bbdfe9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bd507c85-vz2br" Oct 30 21:00:01.059982 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:00:01.059966 5378 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9b4fh\" (UniqueName: \"kubernetes.io/projected/2b1ddcab-5592-429c-9329-b7c2a9bbdfe9-kube-api-access-9b4fh\") pod \"maas-api-revoke-sub-bd507c85-vz2br\" (UID: \"2b1ddcab-5592-429c-9329-b7c2a9bbdfe9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bd507c85-vz2br" Oct 30 21:00:01.060089 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:00:01.060074 5378 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2b1ddcab-5592-429c-9329-b7c2a9bbdfe9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-bd507c85-vz2br\" (UID: \"2b1ddcab-5592-429c-9329-b7c2a9bbdfe9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bd507c85-vz2br" Oct 30 21:00:01.061682 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:00:01.061654 5378 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2b1ddcab-5592-429c-9329-b7c2a9bbdfe9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-bd507c85-vz2br\" (UID: \"2b1ddcab-5592-429c-9329-b7c2a9bbdfe9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bd507c85-vz2br" Oct 30 21:00:01.153835 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:00:01.153754 5378 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2b1ddcab-5592-429c-9329-b7c2a9bbdfe9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-bd507c85-vz2br\" (UID: \"2b1ddcab-5592-429c-9329-b7c2a9bbdfe9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bd507c85-vz2br" Oct 30 21:00:01.154040 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:00:01.154013 5378 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9b4fh\" (UniqueName: \"kubernetes.io/projected/2b1ddcab-5592-429c-9329-b7c2a9bbdfe9-kube-api-access-9b4fh\") pod \"maas-api-revoke-sub-bd507c85-vz2br\" (UID: \"2b1ddcab-5592-429c-9329-b7c2a9bbdfe9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bd507c85-vz2br" Oct 30 21:00:01.193490 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:00:01.193460 5378 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bd507c85-vz2br" Oct 30 21:00:01.575334 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:00:01.575307 5378 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bd507c85-vz2br"] Oct 30 21:00:01.575513 ip-10-0-138-149 kubenswrapper[5378]: W1030 21:00:01.575485 5378 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2b1ddcab_5592_429c_9329_b7c2a9bbdfe9.slice/crio-47002a0577fdeb8845e1e2693fb534af906be19a41dabf413649fafa6e6f3094 WatchSource:0}: Error finding container 47002a0577fdeb8845e1e2693fb534af906be19a41dabf413649fafa6e6f3094: Status 404 returned error can't find the container with id 47002a0577fdeb8845e1e2693fb534af906be19a41dabf413649fafa6e6f3094 Oct 30 21:00:01.660063 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:00:01.660028 5378 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889900-8slvz" event={"ID":"cbc0e737-1972-4f73-8457-7081264afdb7","Type":"ContainerStarted","Data":"b3c1ca6d6d0e879d0737552f92d4e5b7e02a5923c85930bf5732503b4a7b0f04"} Oct 30 21:00:01.660258 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:00:01.660098 5378 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889900-8slvz" event={"ID":"cbc0e737-1972-4f73-8457-7081264afdb7","Type":"ContainerStarted","Data":"9483a72071b3532b6616e43bfbb85ec1433da41cc7e8ff415cc3fdf8644590cb"} Oct 30 21:00:01.661643 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:00:01.661607 5378 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-f9f277-29889900-b7hrn" event={"ID":"372b312c-da86-453f-8287-fe4603fd018c","Type":"ContainerStarted","Data":"d93db541b8aa665d48862ddaff6abc555ad2235c3bedee62ed38e483210027ee"} Oct 30 21:00:01.661771 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:00:01.661661 5378 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-f9f277-29889900-b7hrn" event={"ID":"372b312c-da86-453f-8287-fe4603fd018c","Type":"ContainerStarted","Data":"578ad2bb20b1b315cf1b49a1f29589f6e21c3497a400941c067182dd9e802657"} Oct 30 21:00:01.663309 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:00:01.663269 5378 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-682999-29889900-97hqz" event={"ID":"0e8b014e-a0de-474e-a123-01775c2e4c83","Type":"ContainerStarted","Data":"df63e1d069dc0c0ba5e1a2255db745d565cf4e2aca96cf625124ce8a4ccee0e8"} Oct 30 21:00:01.663438 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:00:01.663323 5378 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-682999-29889900-97hqz" event={"ID":"0e8b014e-a0de-474e-a123-01775c2e4c83","Type":"ContainerStarted","Data":"a81a3502862688eaaead918208902f5402dc495ef5ef49146a1dfc941a7f1fb1"} Oct 30 21:00:01.664764 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:00:01.664729 5378 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bd507c85-vz2br" event={"ID":"2b1ddcab-5592-429c-9329-b7c2a9bbdfe9","Type":"ContainerStarted","Data":"47002a0577fdeb8845e1e2693fb534af906be19a41dabf413649fafa6e6f3094"} Oct 30 21:00:01.781715 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:00:01.781666 5378 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889900-8slvz" podStartSLOduration=1.78165142 podStartE2EDuration="1.78165142s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:00:01.778004025 +0000 UTC m=+1292.206117966" watchObservedRunningTime="2026-10-30 21:00:01.78165142 +0000 UTC m=+1292.209765339" Oct 30 21:00:01.812696 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:00:01.812669 5378 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-682999-openshift-default-54dcb797b5-r27vc"] Oct 30 21:00:01.818389 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:00:01.818363 5378 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-682999-openshift-default-54dcb797b5-r27vc" Oct 30 21:00:01.887841 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:00:01.887516 5378 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-682999-openshift-default-54dcb797b5-r27vc"] Oct 30 21:00:01.981963 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:00:01.981936 5378 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/feae394a-bbbf-4624-8057-cf046f9ffb93-credential-socket\") pod \"e2e-worker-w3-682999-openshift-default-54dcb797b5-r27vc\" (UID: \"feae394a-bbbf-4624-8057-cf046f9ffb93\") " pod="openshift-ingress/e2e-worker-w3-682999-openshift-default-54dcb797b5-r27vc" Oct 30 21:00:01.982108 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:00:01.982073 5378 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/feae394a-bbbf-4624-8057-cf046f9ffb93-istio-envoy\") pod \"e2e-worker-w3-682999-openshift-default-54dcb797b5-r27vc\" (UID: \"feae394a-bbbf-4624-8057-cf046f9ffb93\") " pod="openshift-ingress/e2e-worker-w3-682999-openshift-default-54dcb797b5-r27vc" Oct 30 21:00:01.982294 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:00:01.982127 5378 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/feae394a-bbbf-4624-8057-cf046f9ffb93-istio-podinfo\") pod \"e2e-worker-w3-682999-openshift-default-54dcb797b5-r27vc\" (UID: \"feae394a-bbbf-4624-8057-cf046f9ffb93\") " pod="openshift-ingress/e2e-worker-w3-682999-openshift-default-54dcb797b5-r27vc" Oct 30 21:00:01.982294 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:00:01.982146 5378 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s9mtl\" (UniqueName: \"kubernetes.io/projected/feae394a-bbbf-4624-8057-cf046f9ffb93-kube-api-access-s9mtl\") pod \"e2e-worker-w3-682999-openshift-default-54dcb797b5-r27vc\" (UID: \"feae394a-bbbf-4624-8057-cf046f9ffb93\") " pod="openshift-ingress/e2e-worker-w3-682999-openshift-default-54dcb797b5-r27vc" Oct 30 21:00:01.982422 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:00:01.982286 5378 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/feae394a-bbbf-4624-8057-cf046f9ffb93-istio-token\") pod \"e2e-worker-w3-682999-openshift-default-54dcb797b5-r27vc\" (UID: \"feae394a-bbbf-4624-8057-cf046f9ffb93\") " pod="openshift-ingress/e2e-worker-w3-682999-openshift-default-54dcb797b5-r27vc" Oct 30 21:00:01.982422 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:00:01.982327 5378 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/feae394a-bbbf-4624-8057-cf046f9ffb93-istiod-ca-cert\") pod \"e2e-worker-w3-682999-openshift-default-54dcb797b5-r27vc\" (UID: \"feae394a-bbbf-4624-8057-cf046f9ffb93\") " pod="openshift-ingress/e2e-worker-w3-682999-openshift-default-54dcb797b5-r27vc" Oct 30 21:00:01.982422 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:00:01.982403 5378 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/feae394a-bbbf-4624-8057-cf046f9ffb93-istio-data\") pod \"e2e-worker-w3-682999-openshift-default-54dcb797b5-r27vc\" (UID: \"feae394a-bbbf-4624-8057-cf046f9ffb93\") " pod="openshift-ingress/e2e-worker-w3-682999-openshift-default-54dcb797b5-r27vc" Oct 30 21:00:01.982578 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:00:01.982435 5378 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/feae394a-bbbf-4624-8057-cf046f9ffb93-workload-certs\") pod \"e2e-worker-w3-682999-openshift-default-54dcb797b5-r27vc\" (UID: \"feae394a-bbbf-4624-8057-cf046f9ffb93\") " pod="openshift-ingress/e2e-worker-w3-682999-openshift-default-54dcb797b5-r27vc" Oct 30 21:00:01.982578 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:00:01.982468 5378 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/feae394a-bbbf-4624-8057-cf046f9ffb93-workload-socket\") pod \"e2e-worker-w3-682999-openshift-default-54dcb797b5-r27vc\" (UID: \"feae394a-bbbf-4624-8057-cf046f9ffb93\") " pod="openshift-ingress/e2e-worker-w3-682999-openshift-default-54dcb797b5-r27vc" Oct 30 21:00:02.024390 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:00:02.024274 5378 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-682999-29889900-97hqz" podStartSLOduration=2.024257132 podStartE2EDuration="2.024257132s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:00:02.022936823 +0000 UTC m=+1292.451050744" watchObservedRunningTime="2026-10-30 21:00:02.024257132 +0000 UTC m=+1292.452371052" Oct 30 21:00:02.027026 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:00:02.026977 5378 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-f9f277-29889900-b7hrn" podStartSLOduration=2.026966477 podStartE2EDuration="2.026966477s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:00:01.907039403 +0000 UTC m=+1292.335153321" watchObservedRunningTime="2026-10-30 21:00:02.026966477 +0000 UTC m=+1292.455080398" Oct 30 21:00:02.088492 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:00:02.086358 5378 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/feae394a-bbbf-4624-8057-cf046f9ffb93-istio-token\") pod \"e2e-worker-w3-682999-openshift-default-54dcb797b5-r27vc\" (UID: \"feae394a-bbbf-4624-8057-cf046f9ffb93\") " pod="openshift-ingress/e2e-worker-w3-682999-openshift-default-54dcb797b5-r27vc" Oct 30 21:00:02.088492 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:00:02.086414 5378 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/feae394a-bbbf-4624-8057-cf046f9ffb93-istiod-ca-cert\") pod \"e2e-worker-w3-682999-openshift-default-54dcb797b5-r27vc\" (UID: \"feae394a-bbbf-4624-8057-cf046f9ffb93\") " pod="openshift-ingress/e2e-worker-w3-682999-openshift-default-54dcb797b5-r27vc" Oct 30 21:00:02.088492 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:00:02.086477 5378 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/feae394a-bbbf-4624-8057-cf046f9ffb93-istio-data\") pod \"e2e-worker-w3-682999-openshift-default-54dcb797b5-r27vc\" (UID: \"feae394a-bbbf-4624-8057-cf046f9ffb93\") " pod="openshift-ingress/e2e-worker-w3-682999-openshift-default-54dcb797b5-r27vc" Oct 30 21:00:02.088492 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:00:02.086520 5378 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/feae394a-bbbf-4624-8057-cf046f9ffb93-workload-certs\") pod \"e2e-worker-w3-682999-openshift-default-54dcb797b5-r27vc\" (UID: \"feae394a-bbbf-4624-8057-cf046f9ffb93\") " pod="openshift-ingress/e2e-worker-w3-682999-openshift-default-54dcb797b5-r27vc" Oct 30 21:00:02.089593 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:00:02.088917 5378 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/feae394a-bbbf-4624-8057-cf046f9ffb93-workload-socket\") pod \"e2e-worker-w3-682999-openshift-default-54dcb797b5-r27vc\" (UID: \"feae394a-bbbf-4624-8057-cf046f9ffb93\") " pod="openshift-ingress/e2e-worker-w3-682999-openshift-default-54dcb797b5-r27vc" Oct 30 21:00:02.089593 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:00:02.089042 5378 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/feae394a-bbbf-4624-8057-cf046f9ffb93-credential-socket\") pod \"e2e-worker-w3-682999-openshift-default-54dcb797b5-r27vc\" (UID: \"feae394a-bbbf-4624-8057-cf046f9ffb93\") " pod="openshift-ingress/e2e-worker-w3-682999-openshift-default-54dcb797b5-r27vc" Oct 30 21:00:02.089593 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:00:02.089178 5378 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/feae394a-bbbf-4624-8057-cf046f9ffb93-istio-envoy\") pod \"e2e-worker-w3-682999-openshift-default-54dcb797b5-r27vc\" (UID: \"feae394a-bbbf-4624-8057-cf046f9ffb93\") " pod="openshift-ingress/e2e-worker-w3-682999-openshift-default-54dcb797b5-r27vc" Oct 30 21:00:02.089593 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:00:02.089246 5378 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/feae394a-bbbf-4624-8057-cf046f9ffb93-istio-podinfo\") pod \"e2e-worker-w3-682999-openshift-default-54dcb797b5-r27vc\" (UID: \"feae394a-bbbf-4624-8057-cf046f9ffb93\") " pod="openshift-ingress/e2e-worker-w3-682999-openshift-default-54dcb797b5-r27vc" Oct 30 21:00:02.089593 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:00:02.089272 5378 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s9mtl\" (UniqueName: \"kubernetes.io/projected/feae394a-bbbf-4624-8057-cf046f9ffb93-kube-api-access-s9mtl\") pod \"e2e-worker-w3-682999-openshift-default-54dcb797b5-r27vc\" (UID: \"feae394a-bbbf-4624-8057-cf046f9ffb93\") " pod="openshift-ingress/e2e-worker-w3-682999-openshift-default-54dcb797b5-r27vc" Oct 30 21:00:02.094302 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:00:02.094275 5378 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/feae394a-bbbf-4624-8057-cf046f9ffb93-workload-certs\") pod \"e2e-worker-w3-682999-openshift-default-54dcb797b5-r27vc\" (UID: \"feae394a-bbbf-4624-8057-cf046f9ffb93\") " pod="openshift-ingress/e2e-worker-w3-682999-openshift-default-54dcb797b5-r27vc" Oct 30 21:00:02.104316 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:00:02.094776 5378 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/feae394a-bbbf-4624-8057-cf046f9ffb93-credential-socket\") pod \"e2e-worker-w3-682999-openshift-default-54dcb797b5-r27vc\" (UID: \"feae394a-bbbf-4624-8057-cf046f9ffb93\") " pod="openshift-ingress/e2e-worker-w3-682999-openshift-default-54dcb797b5-r27vc" Oct 30 21:00:02.104509 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:00:02.094995 5378 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/feae394a-bbbf-4624-8057-cf046f9ffb93-workload-socket\") pod \"e2e-worker-w3-682999-openshift-default-54dcb797b5-r27vc\" (UID: \"feae394a-bbbf-4624-8057-cf046f9ffb93\") " pod="openshift-ingress/e2e-worker-w3-682999-openshift-default-54dcb797b5-r27vc" Oct 30 21:00:02.111757 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:00:02.096377 5378 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/feae394a-bbbf-4624-8057-cf046f9ffb93-istiod-ca-cert\") pod \"e2e-worker-w3-682999-openshift-default-54dcb797b5-r27vc\" (UID: \"feae394a-bbbf-4624-8057-cf046f9ffb93\") " pod="openshift-ingress/e2e-worker-w3-682999-openshift-default-54dcb797b5-r27vc" Oct 30 21:00:02.111883 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:00:02.096616 5378 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/feae394a-bbbf-4624-8057-cf046f9ffb93-istio-data\") pod \"e2e-worker-w3-682999-openshift-default-54dcb797b5-r27vc\" (UID: \"feae394a-bbbf-4624-8057-cf046f9ffb93\") " pod="openshift-ingress/e2e-worker-w3-682999-openshift-default-54dcb797b5-r27vc" Oct 30 21:00:02.111974 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:00:02.102084 5378 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/feae394a-bbbf-4624-8057-cf046f9ffb93-istio-envoy\") pod \"e2e-worker-w3-682999-openshift-default-54dcb797b5-r27vc\" (UID: \"feae394a-bbbf-4624-8057-cf046f9ffb93\") " pod="openshift-ingress/e2e-worker-w3-682999-openshift-default-54dcb797b5-r27vc" Oct 30 21:00:02.123169 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:00:02.122248 5378 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/feae394a-bbbf-4624-8057-cf046f9ffb93-istio-podinfo\") pod \"e2e-worker-w3-682999-openshift-default-54dcb797b5-r27vc\" (UID: \"feae394a-bbbf-4624-8057-cf046f9ffb93\") " pod="openshift-ingress/e2e-worker-w3-682999-openshift-default-54dcb797b5-r27vc" Oct 30 21:00:02.166202 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:00:02.166132 5378 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/feae394a-bbbf-4624-8057-cf046f9ffb93-istio-token\") pod \"e2e-worker-w3-682999-openshift-default-54dcb797b5-r27vc\" (UID: \"feae394a-bbbf-4624-8057-cf046f9ffb93\") " pod="openshift-ingress/e2e-worker-w3-682999-openshift-default-54dcb797b5-r27vc" Oct 30 21:00:02.187590 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:00:02.184395 5378 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s9mtl\" (UniqueName: \"kubernetes.io/projected/feae394a-bbbf-4624-8057-cf046f9ffb93-kube-api-access-s9mtl\") pod \"e2e-worker-w3-682999-openshift-default-54dcb797b5-r27vc\" (UID: \"feae394a-bbbf-4624-8057-cf046f9ffb93\") " pod="openshift-ingress/e2e-worker-w3-682999-openshift-default-54dcb797b5-r27vc" Oct 30 21:00:02.263275 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:00:02.263241 5378 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b95e32e5-kzw5p"] Oct 30 21:00:02.268683 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:00:02.268649 5378 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b95e32e5-kzw5p" Oct 30 21:00:02.324617 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:00:02.324579 5378 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b95e32e5-kzw5p"] Oct 30 21:00:02.392323 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:00:02.392280 5378 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/03973ac1-3f01-42f4-bc88-fa8ae0ddf93b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b95e32e5-kzw5p\" (UID: \"03973ac1-3f01-42f4-bc88-fa8ae0ddf93b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b95e32e5-kzw5p" Oct 30 21:00:02.392498 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:00:02.392454 5378 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-895ht\" (UniqueName: \"kubernetes.io/projected/03973ac1-3f01-42f4-bc88-fa8ae0ddf93b-kube-api-access-895ht\") pod \"maas-api-revoke-sub-b95e32e5-kzw5p\" (UID: \"03973ac1-3f01-42f4-bc88-fa8ae0ddf93b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b95e32e5-kzw5p" Oct 30 21:00:02.392588 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:00:02.392504 5378 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/03973ac1-3f01-42f4-bc88-fa8ae0ddf93b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b95e32e5-kzw5p\" (UID: \"03973ac1-3f01-42f4-bc88-fa8ae0ddf93b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b95e32e5-kzw5p" Oct 30 21:00:02.409156 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:00:02.409121 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:00:02.435791 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:00:02.435757 5378 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-682999-openshift-default-54dcb797b5-r27vc" Oct 30 21:00:02.494245 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:00:02.493771 5378 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/03973ac1-3f01-42f4-bc88-fa8ae0ddf93b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b95e32e5-kzw5p\" (UID: \"03973ac1-3f01-42f4-bc88-fa8ae0ddf93b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b95e32e5-kzw5p" Oct 30 21:00:02.494245 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:00:02.493949 5378 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-895ht\" (UniqueName: \"kubernetes.io/projected/03973ac1-3f01-42f4-bc88-fa8ae0ddf93b-kube-api-access-895ht\") pod \"maas-api-revoke-sub-b95e32e5-kzw5p\" (UID: \"03973ac1-3f01-42f4-bc88-fa8ae0ddf93b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b95e32e5-kzw5p" Oct 30 21:00:02.494245 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:00:02.494014 5378 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/03973ac1-3f01-42f4-bc88-fa8ae0ddf93b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b95e32e5-kzw5p\" (UID: \"03973ac1-3f01-42f4-bc88-fa8ae0ddf93b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b95e32e5-kzw5p" Oct 30 21:00:02.505869 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:00:02.494972 5378 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/03973ac1-3f01-42f4-bc88-fa8ae0ddf93b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b95e32e5-kzw5p\" (UID: \"03973ac1-3f01-42f4-bc88-fa8ae0ddf93b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b95e32e5-kzw5p" Oct 30 21:00:02.505869 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:00:02.500600 5378 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/03973ac1-3f01-42f4-bc88-fa8ae0ddf93b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b95e32e5-kzw5p\" (UID: \"03973ac1-3f01-42f4-bc88-fa8ae0ddf93b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b95e32e5-kzw5p" Oct 30 21:00:02.702225 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:00:02.690349 5378 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-895ht\" (UniqueName: \"kubernetes.io/projected/03973ac1-3f01-42f4-bc88-fa8ae0ddf93b-kube-api-access-895ht\") pod \"maas-api-revoke-sub-b95e32e5-kzw5p\" (UID: \"03973ac1-3f01-42f4-bc88-fa8ae0ddf93b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b95e32e5-kzw5p" Oct 30 21:00:02.762064 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:00:02.760803 5378 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bd507c85-vz2br" event={"ID":"2b1ddcab-5592-429c-9329-b7c2a9bbdfe9","Type":"ContainerStarted","Data":"56a7dee9064f0f42b11e3587f69a27cdf9b83c617ffd30f92e0cd9ae9e312570"} Oct 30 21:00:02.766534 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:00:02.765124 5378 generic.go:356] "Generic (PLEG): container finished" podID="cbc0e737-1972-4f73-8457-7081264afdb7" containerID="b3c1ca6d6d0e879d0737552f92d4e5b7e02a5923c85930bf5732503b4a7b0f04" exitCode=60 Oct 30 21:00:02.766534 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:00:02.765220 5378 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889900-8slvz" event={"ID":"cbc0e737-1972-4f73-8457-7081264afdb7","Type":"ContainerDied","Data":"b3c1ca6d6d0e879d0737552f92d4e5b7e02a5923c85930bf5732503b4a7b0f04"} Oct 30 21:00:02.766534 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:00:02.765672 5378 scope.go:122] "RemoveContainer" containerID="b3c1ca6d6d0e879d0737552f92d4e5b7e02a5923c85930bf5732503b4a7b0f04" Oct 30 21:00:02.817745 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:00:02.816711 5378 generic.go:356] "Generic (PLEG): container finished" podID="372b312c-da86-453f-8287-fe4603fd018c" containerID="d93db541b8aa665d48862ddaff6abc555ad2235c3bedee62ed38e483210027ee" exitCode=60 Oct 30 21:00:02.817745 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:00:02.816886 5378 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-f9f277-29889900-b7hrn" event={"ID":"372b312c-da86-453f-8287-fe4603fd018c","Type":"ContainerDied","Data":"d93db541b8aa665d48862ddaff6abc555ad2235c3bedee62ed38e483210027ee"} Oct 30 21:00:02.817745 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:00:02.817318 5378 scope.go:122] "RemoveContainer" containerID="d93db541b8aa665d48862ddaff6abc555ad2235c3bedee62ed38e483210027ee" Oct 30 21:00:02.820722 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:00:02.819844 5378 generic.go:356] "Generic (PLEG): container finished" podID="0e8b014e-a0de-474e-a123-01775c2e4c83" containerID="df63e1d069dc0c0ba5e1a2255db745d565cf4e2aca96cf625124ce8a4ccee0e8" exitCode=60 Oct 30 21:00:02.820722 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:00:02.819918 5378 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-682999-29889900-97hqz" event={"ID":"0e8b014e-a0de-474e-a123-01775c2e4c83","Type":"ContainerDied","Data":"df63e1d069dc0c0ba5e1a2255db745d565cf4e2aca96cf625124ce8a4ccee0e8"} Oct 30 21:00:02.820722 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:00:02.820263 5378 scope.go:122] "RemoveContainer" containerID="df63e1d069dc0c0ba5e1a2255db745d565cf4e2aca96cf625124ce8a4ccee0e8" Oct 30 21:00:02.879849 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:00:02.879823 5378 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-d061d0-openshift-default-86b87f495-k5sd4"] Oct 30 21:00:02.884707 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:00:02.884677 5378 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b95e32e5-kzw5p" Oct 30 21:00:02.904764 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:00:02.904738 5378 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-d061d0-openshift-default-86b87f495-j9jnw"] Oct 30 21:00:02.913715 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:00:02.913688 5378 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-d061d0-openshift-default-86b87f495-k5sd4" Oct 30 21:00:03.053660 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:00:03.043978 5378 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bd507c85-vz2br" podStartSLOduration=3.043958732 podStartE2EDuration="3.043958732s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:00:03.041494618 +0000 UTC m=+1293.469608538" watchObservedRunningTime="2026-10-30 21:00:03.043958732 +0000 UTC m=+1293.472072654" Oct 30 21:00:03.092329 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:00:03.092286 5378 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ddde4251-43ac-4def-b4d4-e1f9a2a7dc30-workload-certs\") pod \"e2e-worker-w2-d061d0-openshift-default-86b87f495-k5sd4\" (UID: \"ddde4251-43ac-4def-b4d4-e1f9a2a7dc30\") " pod="openshift-ingress/e2e-worker-w2-d061d0-openshift-default-86b87f495-k5sd4" Oct 30 21:00:03.092710 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:00:03.092685 5378 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ddde4251-43ac-4def-b4d4-e1f9a2a7dc30-istio-podinfo\") pod \"e2e-worker-w2-d061d0-openshift-default-86b87f495-k5sd4\" (UID: \"ddde4251-43ac-4def-b4d4-e1f9a2a7dc30\") " pod="openshift-ingress/e2e-worker-w2-d061d0-openshift-default-86b87f495-k5sd4" Oct 30 21:00:03.092945 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:00:03.092926 5378 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s4f9m\" (UniqueName: \"kubernetes.io/projected/ddde4251-43ac-4def-b4d4-e1f9a2a7dc30-kube-api-access-s4f9m\") pod \"e2e-worker-w2-d061d0-openshift-default-86b87f495-k5sd4\" (UID: \"ddde4251-43ac-4def-b4d4-e1f9a2a7dc30\") " pod="openshift-ingress/e2e-worker-w2-d061d0-openshift-default-86b87f495-k5sd4" Oct 30 21:00:03.093104 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:00:03.093087 5378 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ddde4251-43ac-4def-b4d4-e1f9a2a7dc30-istio-envoy\") pod \"e2e-worker-w2-d061d0-openshift-default-86b87f495-k5sd4\" (UID: \"ddde4251-43ac-4def-b4d4-e1f9a2a7dc30\") " pod="openshift-ingress/e2e-worker-w2-d061d0-openshift-default-86b87f495-k5sd4" Oct 30 21:00:03.097442 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:00:03.097401 5378 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ddde4251-43ac-4def-b4d4-e1f9a2a7dc30-credential-socket\") pod \"e2e-worker-w2-d061d0-openshift-default-86b87f495-k5sd4\" (UID: \"ddde4251-43ac-4def-b4d4-e1f9a2a7dc30\") " pod="openshift-ingress/e2e-worker-w2-d061d0-openshift-default-86b87f495-k5sd4" Oct 30 21:00:03.097640 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:00:03.097622 5378 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ddde4251-43ac-4def-b4d4-e1f9a2a7dc30-istio-data\") pod \"e2e-worker-w2-d061d0-openshift-default-86b87f495-k5sd4\" (UID: \"ddde4251-43ac-4def-b4d4-e1f9a2a7dc30\") " pod="openshift-ingress/e2e-worker-w2-d061d0-openshift-default-86b87f495-k5sd4" Oct 30 21:00:03.097791 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:00:03.097774 5378 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ddde4251-43ac-4def-b4d4-e1f9a2a7dc30-istio-token\") pod \"e2e-worker-w2-d061d0-openshift-default-86b87f495-k5sd4\" (UID: \"ddde4251-43ac-4def-b4d4-e1f9a2a7dc30\") " pod="openshift-ingress/e2e-worker-w2-d061d0-openshift-default-86b87f495-k5sd4" Oct 30 21:00:03.098111 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:00:03.098092 5378 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ddde4251-43ac-4def-b4d4-e1f9a2a7dc30-istiod-ca-cert\") pod \"e2e-worker-w2-d061d0-openshift-default-86b87f495-k5sd4\" (UID: \"ddde4251-43ac-4def-b4d4-e1f9a2a7dc30\") " pod="openshift-ingress/e2e-worker-w2-d061d0-openshift-default-86b87f495-k5sd4" Oct 30 21:00:03.098258 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:00:03.098242 5378 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ddde4251-43ac-4def-b4d4-e1f9a2a7dc30-workload-socket\") pod \"e2e-worker-w2-d061d0-openshift-default-86b87f495-k5sd4\" (UID: \"ddde4251-43ac-4def-b4d4-e1f9a2a7dc30\") " pod="openshift-ingress/e2e-worker-w2-d061d0-openshift-default-86b87f495-k5sd4" Oct 30 21:00:03.248928 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:00:03.247786 5378 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s4f9m\" (UniqueName: \"kubernetes.io/projected/ddde4251-43ac-4def-b4d4-e1f9a2a7dc30-kube-api-access-s4f9m\") pod \"e2e-worker-w2-d061d0-openshift-default-86b87f495-k5sd4\" (UID: \"ddde4251-43ac-4def-b4d4-e1f9a2a7dc30\") " pod="openshift-ingress/e2e-worker-w2-d061d0-openshift-default-86b87f495-k5sd4" Oct 30 21:00:03.248928 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:00:03.247841 5378 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ddde4251-43ac-4def-b4d4-e1f9a2a7dc30-istio-envoy\") pod \"e2e-worker-w2-d061d0-openshift-default-86b87f495-k5sd4\" (UID: \"ddde4251-43ac-4def-b4d4-e1f9a2a7dc30\") " pod="openshift-ingress/e2e-worker-w2-d061d0-openshift-default-86b87f495-k5sd4" Oct 30 21:00:03.248928 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:00:03.247884 5378 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ddde4251-43ac-4def-b4d4-e1f9a2a7dc30-credential-socket\") pod \"e2e-worker-w2-d061d0-openshift-default-86b87f495-k5sd4\" (UID: \"ddde4251-43ac-4def-b4d4-e1f9a2a7dc30\") " pod="openshift-ingress/e2e-worker-w2-d061d0-openshift-default-86b87f495-k5sd4" Oct 30 21:00:03.248928 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:00:03.247909 5378 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ddde4251-43ac-4def-b4d4-e1f9a2a7dc30-istio-data\") pod \"e2e-worker-w2-d061d0-openshift-default-86b87f495-k5sd4\" (UID: \"ddde4251-43ac-4def-b4d4-e1f9a2a7dc30\") " pod="openshift-ingress/e2e-worker-w2-d061d0-openshift-default-86b87f495-k5sd4" Oct 30 21:00:03.248928 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:00:03.247936 5378 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ddde4251-43ac-4def-b4d4-e1f9a2a7dc30-istio-token\") pod \"e2e-worker-w2-d061d0-openshift-default-86b87f495-k5sd4\" (UID: \"ddde4251-43ac-4def-b4d4-e1f9a2a7dc30\") " pod="openshift-ingress/e2e-worker-w2-d061d0-openshift-default-86b87f495-k5sd4" Oct 30 21:00:03.248928 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:00:03.248084 5378 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ddde4251-43ac-4def-b4d4-e1f9a2a7dc30-istiod-ca-cert\") pod \"e2e-worker-w2-d061d0-openshift-default-86b87f495-k5sd4\" (UID: \"ddde4251-43ac-4def-b4d4-e1f9a2a7dc30\") " pod="openshift-ingress/e2e-worker-w2-d061d0-openshift-default-86b87f495-k5sd4" Oct 30 21:00:03.248928 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:00:03.248114 5378 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ddde4251-43ac-4def-b4d4-e1f9a2a7dc30-workload-socket\") pod \"e2e-worker-w2-d061d0-openshift-default-86b87f495-k5sd4\" (UID: \"ddde4251-43ac-4def-b4d4-e1f9a2a7dc30\") " pod="openshift-ingress/e2e-worker-w2-d061d0-openshift-default-86b87f495-k5sd4" Oct 30 21:00:03.248928 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:00:03.248193 5378 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ddde4251-43ac-4def-b4d4-e1f9a2a7dc30-workload-certs\") pod \"e2e-worker-w2-d061d0-openshift-default-86b87f495-k5sd4\" (UID: \"ddde4251-43ac-4def-b4d4-e1f9a2a7dc30\") " pod="openshift-ingress/e2e-worker-w2-d061d0-openshift-default-86b87f495-k5sd4" Oct 30 21:00:03.248928 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:00:03.248300 5378 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ddde4251-43ac-4def-b4d4-e1f9a2a7dc30-istio-podinfo\") pod \"e2e-worker-w2-d061d0-openshift-default-86b87f495-k5sd4\" (UID: \"ddde4251-43ac-4def-b4d4-e1f9a2a7dc30\") " pod="openshift-ingress/e2e-worker-w2-d061d0-openshift-default-86b87f495-k5sd4" Oct 30 21:00:03.250468 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:00:03.249902 5378 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ddde4251-43ac-4def-b4d4-e1f9a2a7dc30-istio-data\") pod \"e2e-worker-w2-d061d0-openshift-default-86b87f495-k5sd4\" (UID: \"ddde4251-43ac-4def-b4d4-e1f9a2a7dc30\") " pod="openshift-ingress/e2e-worker-w2-d061d0-openshift-default-86b87f495-k5sd4" Oct 30 21:00:03.254584 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:00:03.251104 5378 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ddde4251-43ac-4def-b4d4-e1f9a2a7dc30-credential-socket\") pod \"e2e-worker-w2-d061d0-openshift-default-86b87f495-k5sd4\" (UID: \"ddde4251-43ac-4def-b4d4-e1f9a2a7dc30\") " pod="openshift-ingress/e2e-worker-w2-d061d0-openshift-default-86b87f495-k5sd4" Oct 30 21:00:03.270622 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:00:03.255386 5378 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ddde4251-43ac-4def-b4d4-e1f9a2a7dc30-istiod-ca-cert\") pod \"e2e-worker-w2-d061d0-openshift-default-86b87f495-k5sd4\" (UID: \"ddde4251-43ac-4def-b4d4-e1f9a2a7dc30\") " pod="openshift-ingress/e2e-worker-w2-d061d0-openshift-default-86b87f495-k5sd4" Oct 30 21:00:03.271201 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:00:03.257616 5378 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ddde4251-43ac-4def-b4d4-e1f9a2a7dc30-workload-socket\") pod \"e2e-worker-w2-d061d0-openshift-default-86b87f495-k5sd4\" (UID: \"ddde4251-43ac-4def-b4d4-e1f9a2a7dc30\") " pod="openshift-ingress/e2e-worker-w2-d061d0-openshift-default-86b87f495-k5sd4" Oct 30 21:00:03.271348 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:00:03.257829 5378 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ddde4251-43ac-4def-b4d4-e1f9a2a7dc30-workload-certs\") pod \"e2e-worker-w2-d061d0-openshift-default-86b87f495-k5sd4\" (UID: \"ddde4251-43ac-4def-b4d4-e1f9a2a7dc30\") " pod="openshift-ingress/e2e-worker-w2-d061d0-openshift-default-86b87f495-k5sd4" Oct 30 21:00:03.303887 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:00:03.293699 5378 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ddde4251-43ac-4def-b4d4-e1f9a2a7dc30-istio-podinfo\") pod \"e2e-worker-w2-d061d0-openshift-default-86b87f495-k5sd4\" (UID: \"ddde4251-43ac-4def-b4d4-e1f9a2a7dc30\") " pod="openshift-ingress/e2e-worker-w2-d061d0-openshift-default-86b87f495-k5sd4" Oct 30 21:00:03.322453 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:00:03.322412 5378 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-d061d0-openshift-default-86b87f495-k5sd4"] Oct 30 21:00:03.322606 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:00:03.322597 5378 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-d061d0-openshift-default-86b87f495-j9jnw" Oct 30 21:00:03.329586 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:00:03.323333 5378 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-d061d0-openshift-default-86b87f495-j9jnw"] Oct 30 21:00:03.341034 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:00:03.340872 5378 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-682999-openshift-default-54dcb797b5-r27vc"] Oct 30 21:00:03.427589 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:00:03.423532 5378 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/81398598-ebe6-417f-91f7-90241e726083-istio-envoy\") pod \"e2e-worker-w2-d061d0-openshift-default-86b87f495-j9jnw\" (UID: \"81398598-ebe6-417f-91f7-90241e726083\") " pod="openshift-ingress/e2e-worker-w2-d061d0-openshift-default-86b87f495-j9jnw" Oct 30 21:00:03.573641 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:00:03.546081 5378 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/81398598-ebe6-417f-91f7-90241e726083-workload-certs\") pod \"e2e-worker-w2-d061d0-openshift-default-86b87f495-j9jnw\" (UID: \"81398598-ebe6-417f-91f7-90241e726083\") " pod="openshift-ingress/e2e-worker-w2-d061d0-openshift-default-86b87f495-j9jnw" Oct 30 21:00:03.573641 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:00:03.546204 5378 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/81398598-ebe6-417f-91f7-90241e726083-workload-socket\") pod \"e2e-worker-w2-d061d0-openshift-default-86b87f495-j9jnw\" (UID: \"81398598-ebe6-417f-91f7-90241e726083\") " pod="openshift-ingress/e2e-worker-w2-d061d0-openshift-default-86b87f495-j9jnw" Oct 30 21:00:03.573641 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:00:03.546238 5378 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/81398598-ebe6-417f-91f7-90241e726083-istio-data\") pod \"e2e-worker-w2-d061d0-openshift-default-86b87f495-j9jnw\" (UID: \"81398598-ebe6-417f-91f7-90241e726083\") " pod="openshift-ingress/e2e-worker-w2-d061d0-openshift-default-86b87f495-j9jnw" Oct 30 21:00:03.573641 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:00:03.546342 5378 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/81398598-ebe6-417f-91f7-90241e726083-credential-socket\") pod \"e2e-worker-w2-d061d0-openshift-default-86b87f495-j9jnw\" (UID: \"81398598-ebe6-417f-91f7-90241e726083\") " pod="openshift-ingress/e2e-worker-w2-d061d0-openshift-default-86b87f495-j9jnw" Oct 30 21:00:03.573641 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:00:03.546368 5378 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-85qn4\" (UniqueName: \"kubernetes.io/projected/81398598-ebe6-417f-91f7-90241e726083-kube-api-access-85qn4\") pod \"e2e-worker-w2-d061d0-openshift-default-86b87f495-j9jnw\" (UID: \"81398598-ebe6-417f-91f7-90241e726083\") " pod="openshift-ingress/e2e-worker-w2-d061d0-openshift-default-86b87f495-j9jnw" Oct 30 21:00:03.573641 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:00:03.546422 5378 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/81398598-ebe6-417f-91f7-90241e726083-istio-podinfo\") pod \"e2e-worker-w2-d061d0-openshift-default-86b87f495-j9jnw\" (UID: \"81398598-ebe6-417f-91f7-90241e726083\") " pod="openshift-ingress/e2e-worker-w2-d061d0-openshift-default-86b87f495-j9jnw" Oct 30 21:00:03.573641 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:00:03.546694 5378 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/81398598-ebe6-417f-91f7-90241e726083-istio-token\") pod \"e2e-worker-w2-d061d0-openshift-default-86b87f495-j9jnw\" (UID: \"81398598-ebe6-417f-91f7-90241e726083\") " pod="openshift-ingress/e2e-worker-w2-d061d0-openshift-default-86b87f495-j9jnw" Oct 30 21:00:03.573641 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:00:03.546777 5378 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/81398598-ebe6-417f-91f7-90241e726083-istiod-ca-cert\") pod \"e2e-worker-w2-d061d0-openshift-default-86b87f495-j9jnw\" (UID: \"81398598-ebe6-417f-91f7-90241e726083\") " pod="openshift-ingress/e2e-worker-w2-d061d0-openshift-default-86b87f495-j9jnw" Oct 30 21:00:03.573641 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:00:03.545405 5378 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s4f9m\" (UniqueName: \"kubernetes.io/projected/ddde4251-43ac-4def-b4d4-e1f9a2a7dc30-kube-api-access-s4f9m\") pod \"e2e-worker-w2-d061d0-openshift-default-86b87f495-k5sd4\" (UID: \"ddde4251-43ac-4def-b4d4-e1f9a2a7dc30\") " pod="openshift-ingress/e2e-worker-w2-d061d0-openshift-default-86b87f495-k5sd4" Oct 30 21:00:03.582591 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:00:03.576333 5378 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ddde4251-43ac-4def-b4d4-e1f9a2a7dc30-istio-envoy\") pod \"e2e-worker-w2-d061d0-openshift-default-86b87f495-k5sd4\" (UID: \"ddde4251-43ac-4def-b4d4-e1f9a2a7dc30\") " pod="openshift-ingress/e2e-worker-w2-d061d0-openshift-default-86b87f495-k5sd4" Oct 30 21:00:03.613603 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:00:03.613065 5378 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ddde4251-43ac-4def-b4d4-e1f9a2a7dc30-istio-token\") pod \"e2e-worker-w2-d061d0-openshift-default-86b87f495-k5sd4\" (UID: \"ddde4251-43ac-4def-b4d4-e1f9a2a7dc30\") " pod="openshift-ingress/e2e-worker-w2-d061d0-openshift-default-86b87f495-k5sd4" Oct 30 21:00:03.642704 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:00:03.641862 5378 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-d061d0-openshift-default-86b87f495-k5sd4" Oct 30 21:00:03.670423 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:00:03.670385 5378 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/81398598-ebe6-417f-91f7-90241e726083-istio-token\") pod \"e2e-worker-w2-d061d0-openshift-default-86b87f495-j9jnw\" (UID: \"81398598-ebe6-417f-91f7-90241e726083\") " pod="openshift-ingress/e2e-worker-w2-d061d0-openshift-default-86b87f495-j9jnw" Oct 30 21:00:03.670946 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:00:03.670908 5378 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/81398598-ebe6-417f-91f7-90241e726083-istiod-ca-cert\") pod \"e2e-worker-w2-d061d0-openshift-default-86b87f495-j9jnw\" (UID: \"81398598-ebe6-417f-91f7-90241e726083\") " pod="openshift-ingress/e2e-worker-w2-d061d0-openshift-default-86b87f495-j9jnw" Oct 30 21:00:03.671440 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:00:03.671415 5378 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/81398598-ebe6-417f-91f7-90241e726083-workload-certs\") pod \"e2e-worker-w2-d061d0-openshift-default-86b87f495-j9jnw\" (UID: \"81398598-ebe6-417f-91f7-90241e726083\") " pod="openshift-ingress/e2e-worker-w2-d061d0-openshift-default-86b87f495-j9jnw" Oct 30 21:00:03.672118 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:00:03.672095 5378 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/81398598-ebe6-417f-91f7-90241e726083-workload-socket\") pod \"e2e-worker-w2-d061d0-openshift-default-86b87f495-j9jnw\" (UID: \"81398598-ebe6-417f-91f7-90241e726083\") " pod="openshift-ingress/e2e-worker-w2-d061d0-openshift-default-86b87f495-j9jnw" Oct 30 21:00:03.672513 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:00:03.672269 5378 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/81398598-ebe6-417f-91f7-90241e726083-istio-data\") pod \"e2e-worker-w2-d061d0-openshift-default-86b87f495-j9jnw\" (UID: \"81398598-ebe6-417f-91f7-90241e726083\") " pod="openshift-ingress/e2e-worker-w2-d061d0-openshift-default-86b87f495-j9jnw" Oct 30 21:00:03.673066 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:00:03.672825 5378 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/81398598-ebe6-417f-91f7-90241e726083-credential-socket\") pod \"e2e-worker-w2-d061d0-openshift-default-86b87f495-j9jnw\" (UID: \"81398598-ebe6-417f-91f7-90241e726083\") " pod="openshift-ingress/e2e-worker-w2-d061d0-openshift-default-86b87f495-j9jnw" Oct 30 21:00:03.673411 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:00:03.673205 5378 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-85qn4\" (UniqueName: \"kubernetes.io/projected/81398598-ebe6-417f-91f7-90241e726083-kube-api-access-85qn4\") pod \"e2e-worker-w2-d061d0-openshift-default-86b87f495-j9jnw\" (UID: \"81398598-ebe6-417f-91f7-90241e726083\") " pod="openshift-ingress/e2e-worker-w2-d061d0-openshift-default-86b87f495-j9jnw" Oct 30 21:00:03.673803 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:00:03.673579 5378 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/81398598-ebe6-417f-91f7-90241e726083-istio-podinfo\") pod \"e2e-worker-w2-d061d0-openshift-default-86b87f495-j9jnw\" (UID: \"81398598-ebe6-417f-91f7-90241e726083\") " pod="openshift-ingress/e2e-worker-w2-d061d0-openshift-default-86b87f495-j9jnw" Oct 30 21:00:03.676423 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:00:03.676399 5378 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/81398598-ebe6-417f-91f7-90241e726083-istiod-ca-cert\") pod \"e2e-worker-w2-d061d0-openshift-default-86b87f495-j9jnw\" (UID: \"81398598-ebe6-417f-91f7-90241e726083\") " pod="openshift-ingress/e2e-worker-w2-d061d0-openshift-default-86b87f495-j9jnw" Oct 30 21:00:03.678216 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:00:03.678187 5378 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/81398598-ebe6-417f-91f7-90241e726083-istio-data\") pod \"e2e-worker-w2-d061d0-openshift-default-86b87f495-j9jnw\" (UID: \"81398598-ebe6-417f-91f7-90241e726083\") " pod="openshift-ingress/e2e-worker-w2-d061d0-openshift-default-86b87f495-j9jnw" Oct 30 21:00:03.678690 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:00:03.678664 5378 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/81398598-ebe6-417f-91f7-90241e726083-workload-certs\") pod \"e2e-worker-w2-d061d0-openshift-default-86b87f495-j9jnw\" (UID: \"81398598-ebe6-417f-91f7-90241e726083\") " pod="openshift-ingress/e2e-worker-w2-d061d0-openshift-default-86b87f495-j9jnw" Oct 30 21:00:03.679434 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:00:03.679205 5378 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/81398598-ebe6-417f-91f7-90241e726083-workload-socket\") pod \"e2e-worker-w2-d061d0-openshift-default-86b87f495-j9jnw\" (UID: \"81398598-ebe6-417f-91f7-90241e726083\") " pod="openshift-ingress/e2e-worker-w2-d061d0-openshift-default-86b87f495-j9jnw" Oct 30 21:00:03.679928 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:00:03.679896 5378 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/81398598-ebe6-417f-91f7-90241e726083-credential-socket\") pod \"e2e-worker-w2-d061d0-openshift-default-86b87f495-j9jnw\" (UID: \"81398598-ebe6-417f-91f7-90241e726083\") " pod="openshift-ingress/e2e-worker-w2-d061d0-openshift-default-86b87f495-j9jnw" Oct 30 21:00:03.682330 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:00:03.682202 5378 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/81398598-ebe6-417f-91f7-90241e726083-istio-podinfo\") pod \"e2e-worker-w2-d061d0-openshift-default-86b87f495-j9jnw\" (UID: \"81398598-ebe6-417f-91f7-90241e726083\") " pod="openshift-ingress/e2e-worker-w2-d061d0-openshift-default-86b87f495-j9jnw" Oct 30 21:00:03.792589 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:00:03.764585 5378 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 21:00:03.793538 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:00:03.793499 5378 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 21:00:03.801587 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:00:03.793708 5378 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 21:00:03.810993 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:00:03.795042 5378 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/81398598-ebe6-417f-91f7-90241e726083-istio-token\") pod \"e2e-worker-w2-d061d0-openshift-default-86b87f495-j9jnw\" (UID: \"81398598-ebe6-417f-91f7-90241e726083\") " pod="openshift-ingress/e2e-worker-w2-d061d0-openshift-default-86b87f495-j9jnw" Oct 30 21:00:03.811194 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:00:03.788409 5378 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/81398598-ebe6-417f-91f7-90241e726083-istio-envoy\") pod \"e2e-worker-w2-d061d0-openshift-default-86b87f495-j9jnw\" (UID: \"81398598-ebe6-417f-91f7-90241e726083\") " pod="openshift-ingress/e2e-worker-w2-d061d0-openshift-default-86b87f495-j9jnw" Oct 30 21:00:03.912274 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:00:03.905959 5378 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/81398598-ebe6-417f-91f7-90241e726083-istio-envoy\") pod \"e2e-worker-w2-d061d0-openshift-default-86b87f495-j9jnw\" (UID: \"81398598-ebe6-417f-91f7-90241e726083\") " pod="openshift-ingress/e2e-worker-w2-d061d0-openshift-default-86b87f495-j9jnw" Oct 30 21:00:04.028685 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:00:04.001437 5378 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-85qn4\" (UniqueName: \"kubernetes.io/projected/81398598-ebe6-417f-91f7-90241e726083-kube-api-access-85qn4\") pod \"e2e-worker-w2-d061d0-openshift-default-86b87f495-j9jnw\" (UID: \"81398598-ebe6-417f-91f7-90241e726083\") " pod="openshift-ingress/e2e-worker-w2-d061d0-openshift-default-86b87f495-j9jnw" Oct 30 21:00:04.320590 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:00:04.269696 5378 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-d061d0-openshift-default-86b87f495-j9jnw" Oct 30 21:00:04.419787 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:00:04.287219 5378 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-781058/distinct-w0-781058-kserve-8bdf9bfcd-hzdfk" podUID="a9678fc2-1c56-4d45-9007-eee5049e0f48" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/ready\": context deadline exceeded" Oct 30 21:00:04.419787 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:00:04.289143 5378 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-8fa2a7/premium-w6-8fa2a7-kserve-64bf4bf6c-dxrdr" podUID="484cba5f-b415-430a-b52d-50b1293be948" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:00:04.419787 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:00:04.348655 5378 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-1fdaa7/premium-w4-1fdaa7-kserve-55dffbd5cd-gvpxq" podUID="ec282a4b-742f-4382-a296-335dcd2e9636" containerName="main" probeResult="failure" output="Get \"https://10.133.0.59:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:00:04.465656 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:00:04.465618 5378 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3ed48889-mnnln"] Oct 30 21:00:04.469296 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:00:04.469270 5378 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3ed48889-mnnln"] Oct 30 21:00:04.687654 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:00:04.675883 5378 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b95e32e5-kzw5p"] Oct 30 21:00:04.737073 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:00:04.736500 5378 generic.go:356] "Generic (PLEG): container finished" podID="2b1ddcab-5592-429c-9329-b7c2a9bbdfe9" containerID="56a7dee9064f0f42b11e3587f69a27cdf9b83c617ffd30f92e0cd9ae9e312570" exitCode=0 Oct 30 21:00:04.737073 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:00:04.736691 5378 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bd507c85-vz2br" event={"ID":"2b1ddcab-5592-429c-9329-b7c2a9bbdfe9","Type":"ContainerDied","Data":"56a7dee9064f0f42b11e3587f69a27cdf9b83c617ffd30f92e0cd9ae9e312570"} Oct 30 21:00:04.750813 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:00:04.750730 5378 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-682999-openshift-default-54dcb797b5-r27vc" event={"ID":"feae394a-bbbf-4624-8057-cf046f9ffb93","Type":"ContainerStarted","Data":"919530f442fabc9c6940fba40a1b00fbe28153e954de4f432e3356e9568d065d"} Oct 30 21:00:04.938655 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:00:04.937815 5378 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-d061d0-openshift-default-86b87f495-k5sd4"] Oct 30 21:00:04.944798 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:00:04.944761 5378 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 21:00:04.944904 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:00:04.944845 5378 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 21:00:04.944904 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:00:04.944895 5378 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 21:00:05.311470 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:00:05.311422 5378 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-d061d0-openshift-default-86b87f495-j9jnw"] Oct 30 21:00:05.326097 ip-10-0-138-149 kubenswrapper[5378]: W1030 21:00:05.326066 5378 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod81398598_ebe6_417f_91f7_90241e726083.slice/crio-3365e7dde15529f84bb7dab346cf2f0b40eed53428a2d1459c637d2e43a98560 WatchSource:0}: Error finding container 3365e7dde15529f84bb7dab346cf2f0b40eed53428a2d1459c637d2e43a98560: Status 404 returned error can't find the container with id 3365e7dde15529f84bb7dab346cf2f0b40eed53428a2d1459c637d2e43a98560 Oct 30 21:00:05.330855 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:00:05.329909 5378 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 21:00:05.330855 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:00:05.329999 5378 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 21:00:05.330855 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:00:05.330044 5378 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 21:00:05.745191 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:00:05.745107 5378 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fb6656bf-9b3b-4373-bb00-cf650d4f2bcb" path="/var/lib/kubelet/pods/fb6656bf-9b3b-4373-bb00-cf650d4f2bcb/volumes" Oct 30 21:00:05.846529 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:00:05.846490 5378 generic.go:356] "Generic (PLEG): container finished" podID="cbc0e737-1972-4f73-8457-7081264afdb7" containerID="5b39630fd34a5f5a38248308b0844a1124a7d7b3891b33f8eb43e6df640c617f" exitCode=60 Oct 30 21:00:05.846798 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:00:05.846596 5378 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889900-8slvz" event={"ID":"cbc0e737-1972-4f73-8457-7081264afdb7","Type":"ContainerDied","Data":"5b39630fd34a5f5a38248308b0844a1124a7d7b3891b33f8eb43e6df640c617f"} Oct 30 21:00:05.846798 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:00:05.846682 5378 scope.go:122] "RemoveContainer" containerID="b3c1ca6d6d0e879d0737552f92d4e5b7e02a5923c85930bf5732503b4a7b0f04" Oct 30 21:00:05.847166 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:00:05.847143 5378 scope.go:122] "RemoveContainer" containerID="5b39630fd34a5f5a38248308b0844a1124a7d7b3891b33f8eb43e6df640c617f" Oct 30 21:00:05.847529 ip-10-0-138-149 kubenswrapper[5378]: E1030 21:00:05.847506 5378 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29889900-8slvz_odh-ai-gateway-infra(cbc0e737-1972-4f73-8457-7081264afdb7)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889900-8slvz" podUID="cbc0e737-1972-4f73-8457-7081264afdb7" Oct 30 21:00:05.848798 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:00:05.848769 5378 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-d061d0-openshift-default-86b87f495-k5sd4" event={"ID":"ddde4251-43ac-4def-b4d4-e1f9a2a7dc30","Type":"ContainerStarted","Data":"73fdb355e362e5e5332c460f6b2853ee26e3fcc20a164506e4e54a3aeff4d1c9"} Oct 30 21:00:05.848880 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:00:05.848814 5378 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-d061d0-openshift-default-86b87f495-k5sd4" event={"ID":"ddde4251-43ac-4def-b4d4-e1f9a2a7dc30","Type":"ContainerStarted","Data":"d2389f0c98bf4ee678373dab6e5727803eb94c352fc117ba4cc09dccd021b17d"} Oct 30 21:00:05.850553 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:00:05.850515 5378 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b95e32e5-kzw5p" event={"ID":"03973ac1-3f01-42f4-bc88-fa8ae0ddf93b","Type":"ContainerStarted","Data":"323ffdb133d1b6c43258e5efb100df0729756f618bb2c4b41c5468333b6ea6b7"} Oct 30 21:00:05.850666 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:00:05.850606 5378 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b95e32e5-kzw5p" event={"ID":"03973ac1-3f01-42f4-bc88-fa8ae0ddf93b","Type":"ContainerStarted","Data":"3200e5f9f0e21acf4bc1e22b5a63e4fd2ccc4876815290e963f69a072db798bc"} Oct 30 21:00:05.944891 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:00:05.944855 5378 generic.go:356] "Generic (PLEG): container finished" podID="372b312c-da86-453f-8287-fe4603fd018c" containerID="2e1f04e33ec25169d49fc25ea9fe91f1681d35751fda61dea2f3a4d2b5248e30" exitCode=60 Oct 30 21:00:05.952879 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:00:05.946727 5378 scope.go:122] "RemoveContainer" containerID="2e1f04e33ec25169d49fc25ea9fe91f1681d35751fda61dea2f3a4d2b5248e30" Oct 30 21:00:06.021917 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:00:05.948175 5378 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-f9f277-29889900-b7hrn" event={"ID":"372b312c-da86-453f-8287-fe4603fd018c","Type":"ContainerDied","Data":"2e1f04e33ec25169d49fc25ea9fe91f1681d35751fda61dea2f3a4d2b5248e30"} Oct 30 21:00:06.023785 ip-10-0-138-149 kubenswrapper[5378]: E1030 21:00:06.023753 5378 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-f9f277-29889900-b7hrn_odh-ai-gateway-infra(372b312c-da86-453f-8287-fe4603fd018c)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-f9f277-29889900-b7hrn" podUID="372b312c-da86-453f-8287-fe4603fd018c" Oct 30 21:00:06.031586 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:00:06.024363 5378 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-682999-openshift-default-54dcb797b5-r27vc" event={"ID":"feae394a-bbbf-4624-8057-cf046f9ffb93","Type":"ContainerStarted","Data":"92f037c57c81aec3aac423afef2a5de0ad66cac77aad6a892e410e96d93b5543"} Oct 30 21:00:06.031586 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:00:06.027499 5378 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-d061d0-openshift-default-86b87f495-j9jnw" event={"ID":"81398598-ebe6-417f-91f7-90241e726083","Type":"ContainerStarted","Data":"a32847f6d47e11c6ad3d564e8628408756fe6a9c4b0467a1c0d45fb609c4b84d"} Oct 30 21:00:06.031586 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:00:06.027543 5378 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-d061d0-openshift-default-86b87f495-j9jnw" event={"ID":"81398598-ebe6-417f-91f7-90241e726083","Type":"ContainerStarted","Data":"3365e7dde15529f84bb7dab346cf2f0b40eed53428a2d1459c637d2e43a98560"} Oct 30 21:00:06.050077 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:00:06.050044 5378 generic.go:356] "Generic (PLEG): container finished" podID="0e8b014e-a0de-474e-a123-01775c2e4c83" containerID="215e581006b136f1b81fc1c6b4f58f65c3bf53ed1658efbe96dc8b7120119afe" exitCode=60 Oct 30 21:00:06.051872 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:00:06.050970 5378 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-682999-29889900-97hqz" event={"ID":"0e8b014e-a0de-474e-a123-01775c2e4c83","Type":"ContainerDied","Data":"215e581006b136f1b81fc1c6b4f58f65c3bf53ed1658efbe96dc8b7120119afe"} Oct 30 21:00:06.109230 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:00:06.052253 5378 scope.go:122] "RemoveContainer" containerID="215e581006b136f1b81fc1c6b4f58f65c3bf53ed1658efbe96dc8b7120119afe" Oct 30 21:00:06.128782 ip-10-0-138-149 kubenswrapper[5378]: E1030 21:00:06.128740 5378 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-682999-29889900-97hqz_odh-ai-gateway-infra(0e8b014e-a0de-474e-a123-01775c2e4c83)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-682999-29889900-97hqz" podUID="0e8b014e-a0de-474e-a123-01775c2e4c83" Oct 30 21:00:06.132721 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:00:06.132680 5378 scope.go:122] "RemoveContainer" containerID="d93db541b8aa665d48862ddaff6abc555ad2235c3bedee62ed38e483210027ee" Oct 30 21:00:06.237893 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:00:06.237866 5378 scope.go:122] "RemoveContainer" containerID="df63e1d069dc0c0ba5e1a2255db745d565cf4e2aca96cf625124ce8a4ccee0e8" Oct 30 21:00:06.301647 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:00:06.295627 5378 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-d061d0-openshift-default-86b87f495-j9jnw" Oct 30 21:00:06.301647 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:00:06.298829 5378 patch_prober.go:29] interesting pod/e2e-worker-w2-d061d0-openshift-default-86b87f495-j9jnw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.86:15021/healthz/ready\": dial tcp 10.133.0.86:15021: connect: connection refused" start-of-body= Oct 30 21:00:06.301647 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:00:06.298881 5378 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-d061d0-openshift-default-86b87f495-j9jnw" podUID="81398598-ebe6-417f-91f7-90241e726083" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.86:15021/healthz/ready\": dial tcp 10.133.0.86:15021: connect: connection refused" Oct 30 21:00:06.325824 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:00:06.325763 5378 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-d061d0-openshift-default-86b87f495-k5sd4" podStartSLOduration=4.325740384 podStartE2EDuration="4.325740384s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:00:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:00:06.311027395 +0000 UTC m=+1296.739141326" watchObservedRunningTime="2026-10-30 21:00:06.325740384 +0000 UTC m=+1296.753854304" Oct 30 21:00:06.439760 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:00:06.439727 5378 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-682999-openshift-default-54dcb797b5-r27vc" Oct 30 21:00:06.440393 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:00:06.440019 5378 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b95e32e5-kzw5p" podStartSLOduration=4.439999941 podStartE2EDuration="4.439999941s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:00:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:00:06.434899979 +0000 UTC m=+1296.863013898" watchObservedRunningTime="2026-10-30 21:00:06.439999941 +0000 UTC m=+1296.868113861" Oct 30 21:00:06.440608 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:00:06.440190 5378 patch_prober.go:29] interesting pod/e2e-worker-w3-682999-openshift-default-54dcb797b5-r27vc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused" start-of-body= Oct 30 21:00:06.442525 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:00:06.442476 5378 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-682999-openshift-default-54dcb797b5-r27vc" podUID="feae394a-bbbf-4624-8057-cf046f9ffb93" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused" Oct 30 21:00:06.656860 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:00:06.654746 5378 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bd507c85-vz2br" Oct 30 21:00:06.656860 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:00:06.655122 5378 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-d061d0-openshift-default-86b87f495-k5sd4" Oct 30 21:00:06.671591 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:00:06.659681 5378 patch_prober.go:29] interesting pod/e2e-worker-w2-d061d0-openshift-default-86b87f495-k5sd4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.85:15021/healthz/ready\": dial tcp 10.133.0.85:15021: connect: connection refused" start-of-body= Oct 30 21:00:06.671591 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:00:06.659734 5378 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-d061d0-openshift-default-86b87f495-k5sd4" podUID="ddde4251-43ac-4def-b4d4-e1f9a2a7dc30" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.85:15021/healthz/ready\": dial tcp 10.133.0.85:15021: connect: connection refused" Oct 30 21:00:06.746695 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:00:06.746660 5378 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2b1ddcab-5592-429c-9329-b7c2a9bbdfe9-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2b1ddcab-5592-429c-9329-b7c2a9bbdfe9-maas-api-openshift-ca\") pod \"2b1ddcab-5592-429c-9329-b7c2a9bbdfe9\" (UID: \"2b1ddcab-5592-429c-9329-b7c2a9bbdfe9\") " Oct 30 21:00:06.746985 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:00:06.746963 5378 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2b1ddcab-5592-429c-9329-b7c2a9bbdfe9-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2b1ddcab-5592-429c-9329-b7c2a9bbdfe9-maas-api-xks-ca\") pod \"2b1ddcab-5592-429c-9329-b7c2a9bbdfe9\" (UID: \"2b1ddcab-5592-429c-9329-b7c2a9bbdfe9\") " Oct 30 21:00:06.747192 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:00:06.747175 5378 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2b1ddcab-5592-429c-9329-b7c2a9bbdfe9-kube-api-access-9b4fh\" (UniqueName: \"kubernetes.io/projected/2b1ddcab-5592-429c-9329-b7c2a9bbdfe9-kube-api-access-9b4fh\") pod \"2b1ddcab-5592-429c-9329-b7c2a9bbdfe9\" (UID: \"2b1ddcab-5592-429c-9329-b7c2a9bbdfe9\") " Oct 30 21:00:06.747683 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:00:06.747224 5378 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2b1ddcab-5592-429c-9329-b7c2a9bbdfe9-maas-api-openshift-ca" pod "2b1ddcab-5592-429c-9329-b7c2a9bbdfe9" (UID: "2b1ddcab-5592-429c-9329-b7c2a9bbdfe9"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:00:06.747683 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:00:06.746983 5378 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-d061d0-openshift-default-86b87f495-j9jnw" podStartSLOduration=4.746963716 podStartE2EDuration="4.746963716s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:00:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:00:06.708063714 +0000 UTC m=+1297.136177634" watchObservedRunningTime="2026-10-30 21:00:06.746963716 +0000 UTC m=+1297.175077637" Oct 30 21:00:06.763607 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:00:06.748089 5378 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2b1ddcab-5592-429c-9329-b7c2a9bbdfe9-maas-api-openshift-ca\") on node \"ip-10-0-138-149.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:00:06.845646 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:00:06.750225 5378 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2b1ddcab-5592-429c-9329-b7c2a9bbdfe9-kube-api-access-9b4fh" pod "2b1ddcab-5592-429c-9329-b7c2a9bbdfe9" (UID: "2b1ddcab-5592-429c-9329-b7c2a9bbdfe9"). InnerVolumeSpecName "kube-api-access-9b4fh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:00:06.933982 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:00:06.853691 5378 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2b1ddcab-5592-429c-9329-b7c2a9bbdfe9-maas-api-xks-ca" pod "2b1ddcab-5592-429c-9329-b7c2a9bbdfe9" (UID: "2b1ddcab-5592-429c-9329-b7c2a9bbdfe9"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:00:06.943658 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:00:06.943586 5378 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2b1ddcab-5592-429c-9329-b7c2a9bbdfe9-maas-api-xks-ca\") on node \"ip-10-0-138-149.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:00:06.943658 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:00:06.943626 5378 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9b4fh\" (UniqueName: \"kubernetes.io/projected/2b1ddcab-5592-429c-9329-b7c2a9bbdfe9-kube-api-access-9b4fh\") on node \"ip-10-0-138-149.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:00:06.958589 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:00:06.949812 5378 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-682999-openshift-default-54dcb797b5-r27vc" podStartSLOduration=5.94979057 podStartE2EDuration="5.94979057s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:00:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:00:06.934959737 +0000 UTC m=+1297.363073657" watchObservedRunningTime="2026-10-30 21:00:06.94979057 +0000 UTC m=+1297.377904490" Oct 30 21:00:07.199749 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:00:07.199382 5378 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bd507c85-vz2br" event={"ID":"2b1ddcab-5592-429c-9329-b7c2a9bbdfe9","Type":"ContainerDied","Data":"47002a0577fdeb8845e1e2693fb534af906be19a41dabf413649fafa6e6f3094"} Oct 30 21:00:07.199749 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:00:07.199450 5378 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="47002a0577fdeb8845e1e2693fb534af906be19a41dabf413649fafa6e6f3094" Oct 30 21:00:07.199749 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:00:07.199524 5378 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bd507c85-vz2br" Oct 30 21:00:07.292905 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:00:07.291068 5378 scope.go:122] "RemoveContainer" containerID="5b39630fd34a5f5a38248308b0844a1124a7d7b3891b33f8eb43e6df640c617f" Oct 30 21:00:07.292905 ip-10-0-138-149 kubenswrapper[5378]: E1030 21:00:07.291457 5378 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29889900-8slvz_odh-ai-gateway-infra(cbc0e737-1972-4f73-8457-7081264afdb7)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889900-8slvz" podUID="cbc0e737-1972-4f73-8457-7081264afdb7" Oct 30 21:00:07.450227 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:00:07.447259 5378 patch_prober.go:29] interesting pod/e2e-worker-w3-682999-openshift-default-54dcb797b5-r27vc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused" start-of-body= Oct 30 21:00:07.450227 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:00:07.447318 5378 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-682999-openshift-default-54dcb797b5-r27vc" podUID="feae394a-bbbf-4624-8057-cf046f9ffb93" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused" Oct 30 21:00:07.450227 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:00:07.449794 5378 patch_prober.go:29] interesting pod/e2e-worker-w2-d061d0-openshift-default-86b87f495-j9jnw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.86:15021/healthz/ready\": dial tcp 10.133.0.86:15021: connect: connection refused" start-of-body= Oct 30 21:00:07.450227 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:00:07.449841 5378 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-d061d0-openshift-default-86b87f495-j9jnw" podUID="81398598-ebe6-417f-91f7-90241e726083" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.86:15021/healthz/ready\": dial tcp 10.133.0.86:15021: connect: connection refused" Oct 30 21:00:07.450227 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:00:07.450195 5378 generic.go:356] "Generic (PLEG): container finished" podID="03973ac1-3f01-42f4-bc88-fa8ae0ddf93b" containerID="323ffdb133d1b6c43258e5efb100df0729756f618bb2c4b41c5468333b6ea6b7" exitCode=0 Oct 30 21:00:07.450880 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:00:07.450294 5378 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b95e32e5-kzw5p" event={"ID":"03973ac1-3f01-42f4-bc88-fa8ae0ddf93b","Type":"ContainerDied","Data":"323ffdb133d1b6c43258e5efb100df0729756f618bb2c4b41c5468333b6ea6b7"} Oct 30 21:00:07.493612 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:00:07.489496 5378 scope.go:122] "RemoveContainer" containerID="2e1f04e33ec25169d49fc25ea9fe91f1681d35751fda61dea2f3a4d2b5248e30" Oct 30 21:00:07.493612 ip-10-0-138-149 kubenswrapper[5378]: E1030 21:00:07.489906 5378 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-f9f277-29889900-b7hrn_odh-ai-gateway-infra(372b312c-da86-453f-8287-fe4603fd018c)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-f9f277-29889900-b7hrn" podUID="372b312c-da86-453f-8287-fe4603fd018c" Oct 30 21:00:07.542177 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:00:07.542147 5378 scope.go:122] "RemoveContainer" containerID="215e581006b136f1b81fc1c6b4f58f65c3bf53ed1658efbe96dc8b7120119afe" Oct 30 21:00:07.544490 ip-10-0-138-149 kubenswrapper[5378]: E1030 21:00:07.544459 5378 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-682999-29889900-97hqz_odh-ai-gateway-infra(0e8b014e-a0de-474e-a123-01775c2e4c83)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-682999-29889900-97hqz" podUID="0e8b014e-a0de-474e-a123-01775c2e4c83" Oct 30 21:00:07.669715 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:00:07.669679 5378 patch_prober.go:29] interesting pod/e2e-worker-w2-d061d0-openshift-default-86b87f495-k5sd4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.85:15021/healthz/ready\": dial tcp 10.133.0.85:15021: connect: connection refused" start-of-body= Oct 30 21:00:07.669958 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:00:07.669930 5378 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-d061d0-openshift-default-86b87f495-k5sd4" podUID="ddde4251-43ac-4def-b4d4-e1f9a2a7dc30" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.85:15021/healthz/ready\": dial tcp 10.133.0.85:15021: connect: connection refused" Oct 30 21:00:07.902593 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:00:07.902078 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:00:08.302146 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:00:08.302045 5378 patch_prober.go:29] interesting pod/e2e-worker-w2-d061d0-openshift-default-86b87f495-j9jnw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.86:15021/healthz/ready\": dial tcp 10.133.0.86:15021: connect: connection refused" start-of-body= Oct 30 21:00:08.302432 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:00:08.302386 5378 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-d061d0-openshift-default-86b87f495-j9jnw" podUID="81398598-ebe6-417f-91f7-90241e726083" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.86:15021/healthz/ready\": dial tcp 10.133.0.86:15021: connect: connection refused" Oct 30 21:00:08.439636 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:00:08.438977 5378 patch_prober.go:29] interesting pod/e2e-worker-w3-682999-openshift-default-54dcb797b5-r27vc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused" start-of-body= Oct 30 21:00:08.439636 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:00:08.439042 5378 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-682999-openshift-default-54dcb797b5-r27vc" podUID="feae394a-bbbf-4624-8057-cf046f9ffb93" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused" Oct 30 21:00:08.656030 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:00:08.655993 5378 patch_prober.go:29] interesting pod/e2e-worker-w2-d061d0-openshift-default-86b87f495-k5sd4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.85:15021/healthz/ready\": dial tcp 10.133.0.85:15021: connect: connection refused" start-of-body= Oct 30 21:00:08.656630 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:00:08.656593 5378 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-d061d0-openshift-default-86b87f495-k5sd4" podUID="ddde4251-43ac-4def-b4d4-e1f9a2a7dc30" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.85:15021/healthz/ready\": dial tcp 10.133.0.85:15021: connect: connection refused" Oct 30 21:00:09.291306 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:00:09.289677 5378 patch_prober.go:29] interesting pod/e2e-worker-w2-d061d0-openshift-default-86b87f495-j9jnw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.86:15021/healthz/ready\": dial tcp 10.133.0.86:15021: connect: connection refused" start-of-body= Oct 30 21:00:09.291306 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:00:09.289737 5378 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-d061d0-openshift-default-86b87f495-j9jnw" podUID="81398598-ebe6-417f-91f7-90241e726083" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.86:15021/healthz/ready\": dial tcp 10.133.0.86:15021: connect: connection refused" Oct 30 21:00:09.518590 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:00:09.493410 5378 patch_prober.go:29] interesting pod/e2e-worker-w3-682999-openshift-default-54dcb797b5-r27vc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused" start-of-body= Oct 30 21:00:09.778694 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:00:09.769171 5378 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-682999-openshift-default-54dcb797b5-r27vc" podUID="feae394a-bbbf-4624-8057-cf046f9ffb93" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused" Oct 30 21:00:09.778694 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:00:09.768153 5378 patch_prober.go:29] interesting pod/e2e-worker-w2-d061d0-openshift-default-86b87f495-k5sd4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.85:15021/healthz/ready\": dial tcp 10.133.0.85:15021: connect: connection refused" start-of-body= Oct 30 21:00:09.778694 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:00:09.769655 5378 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-d061d0-openshift-default-86b87f495-k5sd4" podUID="ddde4251-43ac-4def-b4d4-e1f9a2a7dc30" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.85:15021/healthz/ready\": dial tcp 10.133.0.85:15021: connect: connection refused" Oct 30 21:00:10.300362 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:00:10.300315 5378 patch_prober.go:29] interesting pod/e2e-worker-w2-d061d0-openshift-default-86b87f495-j9jnw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.86:15021/healthz/ready\": dial tcp 10.133.0.86:15021: connect: connection refused" start-of-body= Oct 30 21:00:10.300593 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:00:10.300544 5378 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-d061d0-openshift-default-86b87f495-j9jnw" podUID="81398598-ebe6-417f-91f7-90241e726083" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.86:15021/healthz/ready\": dial tcp 10.133.0.86:15021: connect: connection refused" Oct 30 21:00:10.448662 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:00:10.448624 5378 patch_prober.go:29] interesting pod/e2e-worker-w3-682999-openshift-default-54dcb797b5-r27vc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused" start-of-body= Oct 30 21:00:10.448849 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:00:10.448702 5378 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-682999-openshift-default-54dcb797b5-r27vc" podUID="feae394a-bbbf-4624-8057-cf046f9ffb93" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused" Oct 30 21:00:10.486589 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:00:10.465254 5378 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b95e32e5-kzw5p" Oct 30 21:00:10.629580 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:00:10.629528 5378 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/03973ac1-3f01-42f4-bc88-fa8ae0ddf93b-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/03973ac1-3f01-42f4-bc88-fa8ae0ddf93b-maas-api-openshift-ca\") pod \"03973ac1-3f01-42f4-bc88-fa8ae0ddf93b\" (UID: \"03973ac1-3f01-42f4-bc88-fa8ae0ddf93b\") " Oct 30 21:00:10.630176 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:00:10.630149 5378 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/03973ac1-3f01-42f4-bc88-fa8ae0ddf93b-kube-api-access-895ht\" (UniqueName: \"kubernetes.io/projected/03973ac1-3f01-42f4-bc88-fa8ae0ddf93b-kube-api-access-895ht\") pod \"03973ac1-3f01-42f4-bc88-fa8ae0ddf93b\" (UID: \"03973ac1-3f01-42f4-bc88-fa8ae0ddf93b\") " Oct 30 21:00:10.630457 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:00:10.630433 5378 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/03973ac1-3f01-42f4-bc88-fa8ae0ddf93b-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/03973ac1-3f01-42f4-bc88-fa8ae0ddf93b-maas-api-xks-ca\") pod \"03973ac1-3f01-42f4-bc88-fa8ae0ddf93b\" (UID: \"03973ac1-3f01-42f4-bc88-fa8ae0ddf93b\") " Oct 30 21:00:10.630608 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:00:10.630586 5378 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/03973ac1-3f01-42f4-bc88-fa8ae0ddf93b-maas-api-openshift-ca" pod "03973ac1-3f01-42f4-bc88-fa8ae0ddf93b" (UID: "03973ac1-3f01-42f4-bc88-fa8ae0ddf93b"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:00:10.631238 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:00:10.631214 5378 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/03973ac1-3f01-42f4-bc88-fa8ae0ddf93b-maas-api-openshift-ca\") on node \"ip-10-0-138-149.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:00:10.635584 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:00:10.635545 5378 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/03973ac1-3f01-42f4-bc88-fa8ae0ddf93b-maas-api-xks-ca" pod "03973ac1-3f01-42f4-bc88-fa8ae0ddf93b" (UID: "03973ac1-3f01-42f4-bc88-fa8ae0ddf93b"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:00:10.686184 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:00:10.685741 5378 patch_prober.go:29] interesting pod/e2e-worker-w2-d061d0-openshift-default-86b87f495-k5sd4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.85:15021/healthz/ready\": dial tcp 10.133.0.85:15021: connect: connection refused" start-of-body= Oct 30 21:00:10.686184 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:00:10.685801 5378 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-d061d0-openshift-default-86b87f495-k5sd4" podUID="ddde4251-43ac-4def-b4d4-e1f9a2a7dc30" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.85:15021/healthz/ready\": dial tcp 10.133.0.85:15021: connect: connection refused" Oct 30 21:00:10.718595 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:00:10.717314 5378 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/03973ac1-3f01-42f4-bc88-fa8ae0ddf93b-kube-api-access-895ht" pod "03973ac1-3f01-42f4-bc88-fa8ae0ddf93b" (UID: "03973ac1-3f01-42f4-bc88-fa8ae0ddf93b"). InnerVolumeSpecName "kube-api-access-895ht". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:00:10.735480 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:00:10.734967 5378 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/03973ac1-3f01-42f4-bc88-fa8ae0ddf93b-maas-api-xks-ca\") on node \"ip-10-0-138-149.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:00:10.735767 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:00:10.735745 5378 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-895ht\" (UniqueName: \"kubernetes.io/projected/03973ac1-3f01-42f4-bc88-fa8ae0ddf93b-kube-api-access-895ht\") on node \"ip-10-0-138-149.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:00:10.847821 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:00:10.840836 5378 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b95e32e5-kzw5p" event={"ID":"03973ac1-3f01-42f4-bc88-fa8ae0ddf93b","Type":"ContainerDied","Data":"3200e5f9f0e21acf4bc1e22b5a63e4fd2ccc4876815290e963f69a072db798bc"} Oct 30 21:00:10.847821 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:00:10.840904 5378 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3200e5f9f0e21acf4bc1e22b5a63e4fd2ccc4876815290e963f69a072db798bc" Oct 30 21:00:10.847821 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:00:10.840977 5378 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b95e32e5-kzw5p" Oct 30 21:00:11.303589 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:00:11.286943 5378 patch_prober.go:29] interesting pod/e2e-worker-w2-d061d0-openshift-default-86b87f495-j9jnw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.86:15021/healthz/ready\": dial tcp 10.133.0.86:15021: connect: connection refused" start-of-body= Oct 30 21:00:11.303589 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:00:11.287025 5378 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-d061d0-openshift-default-86b87f495-j9jnw" podUID="81398598-ebe6-417f-91f7-90241e726083" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.86:15021/healthz/ready\": dial tcp 10.133.0.86:15021: connect: connection refused" Oct 30 21:00:11.438724 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:00:11.436661 5378 patch_prober.go:29] interesting pod/e2e-worker-w3-682999-openshift-default-54dcb797b5-r27vc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused" start-of-body= Oct 30 21:00:11.438724 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:00:11.436733 5378 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-682999-openshift-default-54dcb797b5-r27vc" podUID="feae394a-bbbf-4624-8057-cf046f9ffb93" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused" Oct 30 21:00:11.660611 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:00:11.657921 5378 patch_prober.go:29] interesting pod/e2e-worker-w2-d061d0-openshift-default-86b87f495-k5sd4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.85:15021/healthz/ready\": dial tcp 10.133.0.85:15021: connect: connection refused" start-of-body= Oct 30 21:00:11.660611 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:00:11.657986 5378 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-d061d0-openshift-default-86b87f495-k5sd4" podUID="ddde4251-43ac-4def-b4d4-e1f9a2a7dc30" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.85:15021/healthz/ready\": dial tcp 10.133.0.85:15021: connect: connection refused" Oct 30 21:00:12.283589 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:00:12.283214 5378 patch_prober.go:29] interesting pod/e2e-worker-w2-d061d0-openshift-default-86b87f495-j9jnw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.86:15021/healthz/ready\": dial tcp 10.133.0.86:15021: connect: connection refused" start-of-body= Oct 30 21:00:12.283589 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:00:12.283292 5378 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-d061d0-openshift-default-86b87f495-j9jnw" podUID="81398598-ebe6-417f-91f7-90241e726083" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.86:15021/healthz/ready\": dial tcp 10.133.0.86:15021: connect: connection refused" Oct 30 21:00:12.436588 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:00:12.436110 5378 patch_prober.go:29] interesting pod/e2e-worker-w3-682999-openshift-default-54dcb797b5-r27vc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused" start-of-body= Oct 30 21:00:12.436588 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:00:12.436179 5378 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-682999-openshift-default-54dcb797b5-r27vc" podUID="feae394a-bbbf-4624-8057-cf046f9ffb93" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused" Oct 30 21:00:12.915316 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:00:12.915261 5378 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a44eaf05-cfrpj"] Oct 30 21:00:12.916247 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:00:12.916218 5378 cpu_manager.go:414] "containerMap: removing container" podUID="2b1ddcab-5592-429c-9329-b7c2a9bbdfe9" containerName="revoke-keys" Oct 30 21:00:12.916247 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:00:12.916246 5378 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2b1ddcab-5592-429c-9329-b7c2a9bbdfe9" containerName="revoke-keys" Oct 30 21:00:12.916451 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:00:12.916273 5378 cpu_manager.go:414] "containerMap: removing container" podUID="03973ac1-3f01-42f4-bc88-fa8ae0ddf93b" containerName="revoke-keys" Oct 30 21:00:12.916451 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:00:12.916281 5378 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="03973ac1-3f01-42f4-bc88-fa8ae0ddf93b" containerName="revoke-keys" Oct 30 21:00:12.916583 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:00:12.916481 5378 memory_manager.go:365] "RemoveStaleState removing state" podUID="2b1ddcab-5592-429c-9329-b7c2a9bbdfe9" containerName="revoke-keys" Oct 30 21:00:12.916583 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:00:12.916499 5378 memory_manager.go:365] "RemoveStaleState removing state" podUID="03973ac1-3f01-42f4-bc88-fa8ae0ddf93b" containerName="revoke-keys" Oct 30 21:00:12.920335 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:00:12.920305 5378 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a44eaf05-cfrpj" Oct 30 21:00:12.986626 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:00:12.977899 5378 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lcfrf\" (UniqueName: \"kubernetes.io/projected/6107aaf1-6393-49ce-b59a-5c00a36ab3aa-kube-api-access-lcfrf\") pod \"maas-api-revoke-sub-a44eaf05-cfrpj\" (UID: \"6107aaf1-6393-49ce-b59a-5c00a36ab3aa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a44eaf05-cfrpj" Oct 30 21:00:12.986626 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:00:12.978121 5378 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6107aaf1-6393-49ce-b59a-5c00a36ab3aa-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a44eaf05-cfrpj\" (UID: \"6107aaf1-6393-49ce-b59a-5c00a36ab3aa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a44eaf05-cfrpj" Oct 30 21:00:12.986626 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:00:12.978169 5378 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6107aaf1-6393-49ce-b59a-5c00a36ab3aa-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a44eaf05-cfrpj\" (UID: \"6107aaf1-6393-49ce-b59a-5c00a36ab3aa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a44eaf05-cfrpj" Oct 30 21:00:12.986626 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:00:12.984873 5378 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a44eaf05-cfrpj"] Oct 30 21:00:13.029201 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:00:13.029101 5378 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 21:00:13.079816 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:00:13.079772 5378 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lcfrf\" (UniqueName: \"kubernetes.io/projected/6107aaf1-6393-49ce-b59a-5c00a36ab3aa-kube-api-access-lcfrf\") pod \"maas-api-revoke-sub-a44eaf05-cfrpj\" (UID: \"6107aaf1-6393-49ce-b59a-5c00a36ab3aa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a44eaf05-cfrpj" Oct 30 21:00:13.080333 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:00:13.080299 5378 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6107aaf1-6393-49ce-b59a-5c00a36ab3aa-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a44eaf05-cfrpj\" (UID: \"6107aaf1-6393-49ce-b59a-5c00a36ab3aa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a44eaf05-cfrpj" Oct 30 21:00:13.081382 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:00:13.081322 5378 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6107aaf1-6393-49ce-b59a-5c00a36ab3aa-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a44eaf05-cfrpj\" (UID: \"6107aaf1-6393-49ce-b59a-5c00a36ab3aa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a44eaf05-cfrpj" Oct 30 21:00:13.081525 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:00:13.080504 5378 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6107aaf1-6393-49ce-b59a-5c00a36ab3aa-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a44eaf05-cfrpj\" (UID: \"6107aaf1-6393-49ce-b59a-5c00a36ab3aa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a44eaf05-cfrpj" Oct 30 21:00:13.084281 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:00:13.084252 5378 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6107aaf1-6393-49ce-b59a-5c00a36ab3aa-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a44eaf05-cfrpj\" (UID: \"6107aaf1-6393-49ce-b59a-5c00a36ab3aa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a44eaf05-cfrpj" Oct 30 21:00:13.140707 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:00:13.139806 5378 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lcfrf\" (UniqueName: \"kubernetes.io/projected/6107aaf1-6393-49ce-b59a-5c00a36ab3aa-kube-api-access-lcfrf\") pod \"maas-api-revoke-sub-a44eaf05-cfrpj\" (UID: \"6107aaf1-6393-49ce-b59a-5c00a36ab3aa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a44eaf05-cfrpj" Oct 30 21:00:13.146367 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:00:13.146327 5378 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-d061d0-openshift-default-86b87f495-k5sd4" Oct 30 21:00:13.147605 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:00:13.147536 5378 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-d061d0-openshift-default-86b87f495-k5sd4" Oct 30 21:00:13.152269 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:00:13.151226 5378 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-d061d0-openshift-default-86b87f495-k5sd4" Oct 30 21:00:13.242260 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:00:13.242169 5378 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a44eaf05-cfrpj" Oct 30 21:00:13.282846 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:00:13.282816 5378 patch_prober.go:29] interesting pod/e2e-worker-w2-d061d0-openshift-default-86b87f495-j9jnw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.86:15021/healthz/ready\": dial tcp 10.133.0.86:15021: connect: connection refused" start-of-body= Oct 30 21:00:13.283037 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:00:13.282871 5378 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-d061d0-openshift-default-86b87f495-j9jnw" podUID="81398598-ebe6-417f-91f7-90241e726083" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.86:15021/healthz/ready\": dial tcp 10.133.0.86:15021: connect: connection refused" Oct 30 21:00:13.283369 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:00:13.283344 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:00:13.474886 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:00:13.474849 5378 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-682999-openshift-default-54dcb797b5-r27vc" Oct 30 21:00:13.476617 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:00:13.476268 5378 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-682999-openshift-default-54dcb797b5-r27vc" Oct 30 21:00:13.477063 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:00:13.477042 5378 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-682999-openshift-default-54dcb797b5-r27vc" Oct 30 21:00:13.635336 ip-10-0-138-149 kubenswrapper[5378]: W1030 21:00:13.635280 5378 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6107aaf1_6393_49ce_b59a_5c00a36ab3aa.slice/crio-d11812dd6f2ad4b2af1cc46b98b9881a98e2d0124123531433352c3d03ad0458 WatchSource:0}: Error finding container d11812dd6f2ad4b2af1cc46b98b9881a98e2d0124123531433352c3d03ad0458: Status 404 returned error can't find the container with id d11812dd6f2ad4b2af1cc46b98b9881a98e2d0124123531433352c3d03ad0458 Oct 30 21:00:13.635503 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:00:13.635413 5378 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a44eaf05-cfrpj"] Oct 30 21:00:13.927804 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:00:13.927717 5378 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a44eaf05-cfrpj" event={"ID":"6107aaf1-6393-49ce-b59a-5c00a36ab3aa","Type":"ContainerStarted","Data":"d11812dd6f2ad4b2af1cc46b98b9881a98e2d0124123531433352c3d03ad0458"} Oct 30 21:00:14.287796 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:00:14.287723 5378 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-d061d0-openshift-default-86b87f495-j9jnw" Oct 30 21:00:14.288214 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:00:14.288200 5378 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-d061d0-openshift-default-86b87f495-j9jnw" Oct 30 21:00:14.288879 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:00:14.288856 5378 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-d061d0-openshift-default-86b87f495-j9jnw" Oct 30 21:00:14.936450 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:00:14.936415 5378 generic.go:356] "Generic (PLEG): container finished" podID="6107aaf1-6393-49ce-b59a-5c00a36ab3aa" containerID="4eedd9f39fe7328f3b420ba428228ea8000bf66d9c46f94cee388b733ae7a7bf" exitCode=0 Oct 30 21:00:14.936914 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:00:14.936500 5378 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a44eaf05-cfrpj" event={"ID":"6107aaf1-6393-49ce-b59a-5c00a36ab3aa","Type":"ContainerDied","Data":"4eedd9f39fe7328f3b420ba428228ea8000bf66d9c46f94cee388b733ae7a7bf"} Oct 30 21:00:15.047233 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:00:15.047198 5378 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8d9f30fa-h8vv4"] Oct 30 21:00:15.054121 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:00:15.054093 5378 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8d9f30fa-h8vv4"] Oct 30 21:00:15.746692 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:00:15.744384 5378 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f05d1764-4d6a-4cc9-915b-0b0ba752091f" path="/var/lib/kubelet/pods/f05d1764-4d6a-4cc9-915b-0b0ba752091f/volumes" Oct 30 21:00:16.119860 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:00:16.119837 5378 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a44eaf05-cfrpj" Oct 30 21:00:16.239331 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:00:16.239296 5378 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6107aaf1-6393-49ce-b59a-5c00a36ab3aa-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6107aaf1-6393-49ce-b59a-5c00a36ab3aa-maas-api-xks-ca\") pod \"6107aaf1-6393-49ce-b59a-5c00a36ab3aa\" (UID: \"6107aaf1-6393-49ce-b59a-5c00a36ab3aa\") " Oct 30 21:00:16.240070 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:00:16.239826 5378 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6107aaf1-6393-49ce-b59a-5c00a36ab3aa-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6107aaf1-6393-49ce-b59a-5c00a36ab3aa-maas-api-openshift-ca\") pod \"6107aaf1-6393-49ce-b59a-5c00a36ab3aa\" (UID: \"6107aaf1-6393-49ce-b59a-5c00a36ab3aa\") " Oct 30 21:00:16.243734 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:00:16.242888 5378 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6107aaf1-6393-49ce-b59a-5c00a36ab3aa-kube-api-access-lcfrf\" (UniqueName: \"kubernetes.io/projected/6107aaf1-6393-49ce-b59a-5c00a36ab3aa-kube-api-access-lcfrf\") pod \"6107aaf1-6393-49ce-b59a-5c00a36ab3aa\" (UID: \"6107aaf1-6393-49ce-b59a-5c00a36ab3aa\") " Oct 30 21:00:16.243734 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:00:16.242167 5378 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6107aaf1-6393-49ce-b59a-5c00a36ab3aa-maas-api-xks-ca" pod "6107aaf1-6393-49ce-b59a-5c00a36ab3aa" (UID: "6107aaf1-6393-49ce-b59a-5c00a36ab3aa"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:00:16.244011 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:00:16.243900 5378 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6107aaf1-6393-49ce-b59a-5c00a36ab3aa-maas-api-xks-ca\") on node \"ip-10-0-138-149.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:00:16.244517 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:00:16.244493 5378 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6107aaf1-6393-49ce-b59a-5c00a36ab3aa-maas-api-openshift-ca" pod "6107aaf1-6393-49ce-b59a-5c00a36ab3aa" (UID: "6107aaf1-6393-49ce-b59a-5c00a36ab3aa"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:00:16.257920 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:00:16.257850 5378 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6107aaf1-6393-49ce-b59a-5c00a36ab3aa-kube-api-access-lcfrf" pod "6107aaf1-6393-49ce-b59a-5c00a36ab3aa" (UID: "6107aaf1-6393-49ce-b59a-5c00a36ab3aa"). InnerVolumeSpecName "kube-api-access-lcfrf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:00:16.345659 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:00:16.345627 5378 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lcfrf\" (UniqueName: \"kubernetes.io/projected/6107aaf1-6393-49ce-b59a-5c00a36ab3aa-kube-api-access-lcfrf\") on node \"ip-10-0-138-149.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:00:16.345659 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:00:16.345653 5378 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6107aaf1-6393-49ce-b59a-5c00a36ab3aa-maas-api-openshift-ca\") on node \"ip-10-0-138-149.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:00:16.682559 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:00:16.682528 5378 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-31baea91-bjrgz"] Oct 30 21:00:16.683162 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:00:16.683148 5378 cpu_manager.go:414] "containerMap: removing container" podUID="6107aaf1-6393-49ce-b59a-5c00a36ab3aa" containerName="revoke-keys" Oct 30 21:00:16.683162 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:00:16.683163 5378 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6107aaf1-6393-49ce-b59a-5c00a36ab3aa" containerName="revoke-keys" Oct 30 21:00:16.683282 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:00:16.683278 5378 memory_manager.go:365] "RemoveStaleState removing state" podUID="6107aaf1-6393-49ce-b59a-5c00a36ab3aa" containerName="revoke-keys" Oct 30 21:00:16.686746 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:00:16.686728 5378 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-31baea91-bjrgz" Oct 30 21:00:16.734667 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:00:16.734633 5378 scope.go:122] "RemoveContainer" containerID="215e581006b136f1b81fc1c6b4f58f65c3bf53ed1658efbe96dc8b7120119afe" Oct 30 21:00:16.735032 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:00:16.735016 5378 scope.go:122] "RemoveContainer" containerID="5b39630fd34a5f5a38248308b0844a1124a7d7b3891b33f8eb43e6df640c617f" Oct 30 21:00:16.743970 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:00:16.743944 5378 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-31baea91-bjrgz"] Oct 30 21:00:16.750610 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:00:16.750581 5378 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/965b8faf-458e-499d-8cf2-4001dc4f97ef-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-31baea91-bjrgz\" (UID: \"965b8faf-458e-499d-8cf2-4001dc4f97ef\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-31baea91-bjrgz" Oct 30 21:00:16.750785 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:00:16.750763 5378 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/965b8faf-458e-499d-8cf2-4001dc4f97ef-maas-api-xks-ca\") pod \"maas-api-revoke-sub-31baea91-bjrgz\" (UID: \"965b8faf-458e-499d-8cf2-4001dc4f97ef\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-31baea91-bjrgz" Oct 30 21:00:16.750856 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:00:16.750839 5378 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x5qsw\" (UniqueName: \"kubernetes.io/projected/965b8faf-458e-499d-8cf2-4001dc4f97ef-kube-api-access-x5qsw\") pod \"maas-api-revoke-sub-31baea91-bjrgz\" (UID: \"965b8faf-458e-499d-8cf2-4001dc4f97ef\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-31baea91-bjrgz" Oct 30 21:00:16.851997 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:00:16.851965 5378 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/965b8faf-458e-499d-8cf2-4001dc4f97ef-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-31baea91-bjrgz\" (UID: \"965b8faf-458e-499d-8cf2-4001dc4f97ef\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-31baea91-bjrgz" Oct 30 21:00:16.852746 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:00:16.852474 5378 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/965b8faf-458e-499d-8cf2-4001dc4f97ef-maas-api-xks-ca\") pod \"maas-api-revoke-sub-31baea91-bjrgz\" (UID: \"965b8faf-458e-499d-8cf2-4001dc4f97ef\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-31baea91-bjrgz" Oct 30 21:00:16.852865 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:00:16.852776 5378 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/965b8faf-458e-499d-8cf2-4001dc4f97ef-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-31baea91-bjrgz\" (UID: \"965b8faf-458e-499d-8cf2-4001dc4f97ef\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-31baea91-bjrgz" Oct 30 21:00:16.852865 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:00:16.852809 5378 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x5qsw\" (UniqueName: \"kubernetes.io/projected/965b8faf-458e-499d-8cf2-4001dc4f97ef-kube-api-access-x5qsw\") pod \"maas-api-revoke-sub-31baea91-bjrgz\" (UID: \"965b8faf-458e-499d-8cf2-4001dc4f97ef\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-31baea91-bjrgz" Oct 30 21:00:16.855173 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:00:16.855150 5378 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/965b8faf-458e-499d-8cf2-4001dc4f97ef-maas-api-xks-ca\") pod \"maas-api-revoke-sub-31baea91-bjrgz\" (UID: \"965b8faf-458e-499d-8cf2-4001dc4f97ef\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-31baea91-bjrgz" Oct 30 21:00:16.910421 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:00:16.910392 5378 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x5qsw\" (UniqueName: \"kubernetes.io/projected/965b8faf-458e-499d-8cf2-4001dc4f97ef-kube-api-access-x5qsw\") pod \"maas-api-revoke-sub-31baea91-bjrgz\" (UID: \"965b8faf-458e-499d-8cf2-4001dc4f97ef\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-31baea91-bjrgz" Oct 30 21:00:16.953402 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:00:16.953319 5378 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a44eaf05-cfrpj" Oct 30 21:00:16.953402 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:00:16.953353 5378 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a44eaf05-cfrpj" event={"ID":"6107aaf1-6393-49ce-b59a-5c00a36ab3aa","Type":"ContainerDied","Data":"d11812dd6f2ad4b2af1cc46b98b9881a98e2d0124123531433352c3d03ad0458"} Oct 30 21:00:16.953586 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:00:16.953411 5378 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d11812dd6f2ad4b2af1cc46b98b9881a98e2d0124123531433352c3d03ad0458" Oct 30 21:00:17.001141 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:00:17.001111 5378 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-31baea91-bjrgz" Oct 30 21:00:17.047114 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:00:17.046779 5378 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0c6ca373-f2mh2"] Oct 30 21:00:17.050180 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:00:17.050154 5378 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0c6ca373-f2mh2"] Oct 30 21:00:17.388580 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:00:17.388537 5378 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-31baea91-bjrgz"] Oct 30 21:00:17.389311 ip-10-0-138-149 kubenswrapper[5378]: W1030 21:00:17.389283 5378 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod965b8faf_458e_499d_8cf2_4001dc4f97ef.slice/crio-37b45bf8c84ce628b17ee852c9cc1f0450cb35e3e7d46a3c1d7e6c56bc65c8eb WatchSource:0}: Error finding container 37b45bf8c84ce628b17ee852c9cc1f0450cb35e3e7d46a3c1d7e6c56bc65c8eb: Status 404 returned error can't find the container with id 37b45bf8c84ce628b17ee852c9cc1f0450cb35e3e7d46a3c1d7e6c56bc65c8eb Oct 30 21:00:17.742482 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:00:17.742446 5378 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8abc6c54-d10b-4961-811d-f2b7df218958" path="/var/lib/kubelet/pods/8abc6c54-d10b-4961-811d-f2b7df218958/volumes" Oct 30 21:00:17.973835 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:00:17.973753 5378 generic.go:356] "Generic (PLEG): container finished" podID="0e8b014e-a0de-474e-a123-01775c2e4c83" containerID="d9166a81387c271c4faff6e5462a54987d87fb7d45959934f825e9b9e3c4a6f7" exitCode=60 Oct 30 21:00:17.974138 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:00:17.973963 5378 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-682999-29889900-97hqz" event={"ID":"0e8b014e-a0de-474e-a123-01775c2e4c83","Type":"ContainerDied","Data":"d9166a81387c271c4faff6e5462a54987d87fb7d45959934f825e9b9e3c4a6f7"} Oct 30 21:00:17.974263 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:00:17.974230 5378 scope.go:122] "RemoveContainer" containerID="215e581006b136f1b81fc1c6b4f58f65c3bf53ed1658efbe96dc8b7120119afe" Oct 30 21:00:17.974792 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:00:17.974732 5378 scope.go:122] "RemoveContainer" containerID="d9166a81387c271c4faff6e5462a54987d87fb7d45959934f825e9b9e3c4a6f7" Oct 30 21:00:17.975170 ip-10-0-138-149 kubenswrapper[5378]: E1030 21:00:17.975146 5378 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-682999-29889900-97hqz_odh-ai-gateway-infra(0e8b014e-a0de-474e-a123-01775c2e4c83)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-682999-29889900-97hqz" podUID="0e8b014e-a0de-474e-a123-01775c2e4c83" Oct 30 21:00:17.978044 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:00:17.978018 5378 generic.go:356] "Generic (PLEG): container finished" podID="cbc0e737-1972-4f73-8457-7081264afdb7" containerID="f1bf541ff61680f37ea1fd57aa08327458c80763332f4fb7ff088192dc3af004" exitCode=60 Oct 30 21:00:17.978142 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:00:17.978087 5378 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889900-8slvz" event={"ID":"cbc0e737-1972-4f73-8457-7081264afdb7","Type":"ContainerDied","Data":"f1bf541ff61680f37ea1fd57aa08327458c80763332f4fb7ff088192dc3af004"} Oct 30 21:00:17.978517 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:00:17.978493 5378 scope.go:122] "RemoveContainer" containerID="f1bf541ff61680f37ea1fd57aa08327458c80763332f4fb7ff088192dc3af004" Oct 30 21:00:17.978882 ip-10-0-138-149 kubenswrapper[5378]: E1030 21:00:17.978863 5378 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-29889900-8slvz_odh-ai-gateway-infra(cbc0e737-1972-4f73-8457-7081264afdb7)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889900-8slvz" podUID="cbc0e737-1972-4f73-8457-7081264afdb7" Oct 30 21:00:17.980127 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:00:17.980105 5378 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-31baea91-bjrgz" event={"ID":"965b8faf-458e-499d-8cf2-4001dc4f97ef","Type":"ContainerStarted","Data":"d89f3cef996a81cab6f492854996217d11de5c7d1fe066f9f3e49f565bcfe1ab"} Oct 30 21:00:17.980235 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:00:17.980152 5378 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-31baea91-bjrgz" event={"ID":"965b8faf-458e-499d-8cf2-4001dc4f97ef","Type":"ContainerStarted","Data":"37b45bf8c84ce628b17ee852c9cc1f0450cb35e3e7d46a3c1d7e6c56bc65c8eb"} Oct 30 21:00:17.988741 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:00:17.988722 5378 scope.go:122] "RemoveContainer" containerID="5b39630fd34a5f5a38248308b0844a1124a7d7b3891b33f8eb43e6df640c617f" Oct 30 21:00:18.218367 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:00:18.218335 5378 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ca08f19a-td59w"] Oct 30 21:00:18.223742 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:00:18.223718 5378 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ca08f19a-td59w" Oct 30 21:00:18.269956 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:00:18.269923 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:00:18.282844 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:00:18.282816 5378 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ca08f19a-td59w"] Oct 30 21:00:18.374258 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:00:18.374194 5378 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-31baea91-bjrgz" podStartSLOduration=2.37417686 podStartE2EDuration="2.37417686s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:00:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:00:18.353956128 +0000 UTC m=+1308.782070060" watchObservedRunningTime="2026-10-30 21:00:18.37417686 +0000 UTC m=+1308.802290779" Oct 30 21:00:18.374447 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:00:18.374333 5378 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8ce46c99-ec66-4e61-8b6f-135d76c640e1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ca08f19a-td59w\" (UID: \"8ce46c99-ec66-4e61-8b6f-135d76c640e1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ca08f19a-td59w" Oct 30 21:00:18.374447 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:00:18.374430 5378 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8ce46c99-ec66-4e61-8b6f-135d76c640e1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ca08f19a-td59w\" (UID: \"8ce46c99-ec66-4e61-8b6f-135d76c640e1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ca08f19a-td59w" Oct 30 21:00:18.374624 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:00:18.374463 5378 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w2kz7\" (UniqueName: \"kubernetes.io/projected/8ce46c99-ec66-4e61-8b6f-135d76c640e1-kube-api-access-w2kz7\") pod \"maas-api-revoke-sub-ca08f19a-td59w\" (UID: \"8ce46c99-ec66-4e61-8b6f-135d76c640e1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ca08f19a-td59w" Oct 30 21:00:18.475613 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:00:18.475545 5378 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8ce46c99-ec66-4e61-8b6f-135d76c640e1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ca08f19a-td59w\" (UID: \"8ce46c99-ec66-4e61-8b6f-135d76c640e1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ca08f19a-td59w" Oct 30 21:00:18.476019 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:00:18.475654 5378 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8ce46c99-ec66-4e61-8b6f-135d76c640e1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ca08f19a-td59w\" (UID: \"8ce46c99-ec66-4e61-8b6f-135d76c640e1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ca08f19a-td59w" Oct 30 21:00:18.476019 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:00:18.475675 5378 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w2kz7\" (UniqueName: \"kubernetes.io/projected/8ce46c99-ec66-4e61-8b6f-135d76c640e1-kube-api-access-w2kz7\") pod \"maas-api-revoke-sub-ca08f19a-td59w\" (UID: \"8ce46c99-ec66-4e61-8b6f-135d76c640e1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ca08f19a-td59w" Oct 30 21:00:18.476264 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:00:18.476244 5378 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8ce46c99-ec66-4e61-8b6f-135d76c640e1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ca08f19a-td59w\" (UID: \"8ce46c99-ec66-4e61-8b6f-135d76c640e1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ca08f19a-td59w" Oct 30 21:00:18.478012 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:00:18.477982 5378 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8ce46c99-ec66-4e61-8b6f-135d76c640e1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ca08f19a-td59w\" (UID: \"8ce46c99-ec66-4e61-8b6f-135d76c640e1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ca08f19a-td59w" Oct 30 21:00:18.535834 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:00:18.535772 5378 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w2kz7\" (UniqueName: \"kubernetes.io/projected/8ce46c99-ec66-4e61-8b6f-135d76c640e1-kube-api-access-w2kz7\") pod \"maas-api-revoke-sub-ca08f19a-td59w\" (UID: \"8ce46c99-ec66-4e61-8b6f-135d76c640e1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ca08f19a-td59w" Oct 30 21:00:18.538616 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:00:18.538593 5378 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ca08f19a-td59w" Oct 30 21:00:18.841496 ip-10-0-138-149 kubenswrapper[5378]: E1030 21:00:18.841412 5378 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q6mx7" containerName="extractor" Oct 30 21:00:18.914134 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:00:18.914105 5378 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ca08f19a-td59w"] Oct 30 21:00:18.914759 ip-10-0-138-149 kubenswrapper[5378]: W1030 21:00:18.914736 5378 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8ce46c99_ec66_4e61_8b6f_135d76c640e1.slice/crio-44594b82502af6de1d25b1806a109b70da10a61140556b2b7a48af3ee0f259c9 WatchSource:0}: Error finding container 44594b82502af6de1d25b1806a109b70da10a61140556b2b7a48af3ee0f259c9: Status 404 returned error can't find the container with id 44594b82502af6de1d25b1806a109b70da10a61140556b2b7a48af3ee0f259c9 Oct 30 21:00:18.992782 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:00:18.992753 5378 generic.go:356] "Generic (PLEG): container finished" podID="965b8faf-458e-499d-8cf2-4001dc4f97ef" containerID="d89f3cef996a81cab6f492854996217d11de5c7d1fe066f9f3e49f565bcfe1ab" exitCode=0 Oct 30 21:00:18.992916 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:00:18.992831 5378 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-31baea91-bjrgz" event={"ID":"965b8faf-458e-499d-8cf2-4001dc4f97ef","Type":"ContainerDied","Data":"d89f3cef996a81cab6f492854996217d11de5c7d1fe066f9f3e49f565bcfe1ab"} Oct 30 21:00:18.995843 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:00:18.995819 5378 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ca08f19a-td59w" event={"ID":"8ce46c99-ec66-4e61-8b6f-135d76c640e1","Type":"ContainerStarted","Data":"44594b82502af6de1d25b1806a109b70da10a61140556b2b7a48af3ee0f259c9"} Oct 30 21:00:19.101747 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:00:19.101721 5378 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-682999-29889900-97hqz"] Oct 30 21:00:19.216814 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:00:19.216786 5378 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889900-8slvz"] Oct 30 21:00:19.251285 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:00:19.251255 5378 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-682999-29889900-97hqz" Oct 30 21:00:19.389849 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:00:19.389817 5378 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0e8b014e-a0de-474e-a123-01775c2e4c83-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0e8b014e-a0de-474e-a123-01775c2e4c83-maas-api-service-ca\") pod \"0e8b014e-a0de-474e-a123-01775c2e4c83\" (UID: \"0e8b014e-a0de-474e-a123-01775c2e4c83\") " Oct 30 21:00:19.390038 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:00:19.390012 5378 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0e8b014e-a0de-474e-a123-01775c2e4c83-kube-api-access-n8mxj\" (UniqueName: \"kubernetes.io/projected/0e8b014e-a0de-474e-a123-01775c2e4c83-kube-api-access-n8mxj\") pod \"0e8b014e-a0de-474e-a123-01775c2e4c83\" (UID: \"0e8b014e-a0de-474e-a123-01775c2e4c83\") " Oct 30 21:00:19.390127 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:00:19.390095 5378 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0e8b014e-a0de-474e-a123-01775c2e4c83-maas-api-service-ca" pod "0e8b014e-a0de-474e-a123-01775c2e4c83" (UID: "0e8b014e-a0de-474e-a123-01775c2e4c83"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:00:19.390450 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:00:19.390432 5378 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0e8b014e-a0de-474e-a123-01775c2e4c83-maas-api-service-ca\") on node \"ip-10-0-138-149.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:00:19.391884 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:00:19.391866 5378 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0e8b014e-a0de-474e-a123-01775c2e4c83-kube-api-access-n8mxj" pod "0e8b014e-a0de-474e-a123-01775c2e4c83" (UID: "0e8b014e-a0de-474e-a123-01775c2e4c83"). InnerVolumeSpecName "kube-api-access-n8mxj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:00:19.414093 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:00:19.414065 5378 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889900-8slvz" Oct 30 21:00:19.491042 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:00:19.491013 5378 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cbc0e737-1972-4f73-8457-7081264afdb7-kube-api-access-7m9nx\" (UniqueName: \"kubernetes.io/projected/cbc0e737-1972-4f73-8457-7081264afdb7-kube-api-access-7m9nx\") pod \"cbc0e737-1972-4f73-8457-7081264afdb7\" (UID: \"cbc0e737-1972-4f73-8457-7081264afdb7\") " Oct 30 21:00:19.491507 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:00:19.491150 5378 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cbc0e737-1972-4f73-8457-7081264afdb7-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cbc0e737-1972-4f73-8457-7081264afdb7-maas-api-service-ca\") pod \"cbc0e737-1972-4f73-8457-7081264afdb7\" (UID: \"cbc0e737-1972-4f73-8457-7081264afdb7\") " Oct 30 21:00:19.491507 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:00:19.491480 5378 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cbc0e737-1972-4f73-8457-7081264afdb7-maas-api-service-ca" pod "cbc0e737-1972-4f73-8457-7081264afdb7" (UID: "cbc0e737-1972-4f73-8457-7081264afdb7"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:00:19.491705 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:00:19.491690 5378 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cbc0e737-1972-4f73-8457-7081264afdb7-maas-api-service-ca\") on node \"ip-10-0-138-149.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:00:19.491744 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:00:19.491708 5378 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n8mxj\" (UniqueName: \"kubernetes.io/projected/0e8b014e-a0de-474e-a123-01775c2e4c83-kube-api-access-n8mxj\") on node \"ip-10-0-138-149.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:00:19.493126 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:00:19.493085 5378 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cbc0e737-1972-4f73-8457-7081264afdb7-kube-api-access-7m9nx" pod "cbc0e737-1972-4f73-8457-7081264afdb7" (UID: "cbc0e737-1972-4f73-8457-7081264afdb7"). InnerVolumeSpecName "kube-api-access-7m9nx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:00:19.593250 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:00:19.593220 5378 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7m9nx\" (UniqueName: \"kubernetes.io/projected/cbc0e737-1972-4f73-8457-7081264afdb7-kube-api-access-7m9nx\") on node \"ip-10-0-138-149.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:00:19.738089 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:00:19.738007 5378 scope.go:122] "RemoveContainer" containerID="2e1f04e33ec25169d49fc25ea9fe91f1681d35751fda61dea2f3a4d2b5248e30" Oct 30 21:00:20.004173 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:00:20.004108 5378 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889900-8slvz" Oct 30 21:00:20.004173 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:00:20.004128 5378 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889900-8slvz" event={"ID":"cbc0e737-1972-4f73-8457-7081264afdb7","Type":"ContainerDied","Data":"9483a72071b3532b6616e43bfbb85ec1433da41cc7e8ff415cc3fdf8644590cb"} Oct 30 21:00:20.004362 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:00:20.004182 5378 scope.go:122] "RemoveContainer" containerID="f1bf541ff61680f37ea1fd57aa08327458c80763332f4fb7ff088192dc3af004" Oct 30 21:00:20.005918 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:00:20.005891 5378 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-682999-29889900-97hqz" Oct 30 21:00:20.005918 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:00:20.005902 5378 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-682999-29889900-97hqz" event={"ID":"0e8b014e-a0de-474e-a123-01775c2e4c83","Type":"ContainerDied","Data":"a81a3502862688eaaead918208902f5402dc495ef5ef49146a1dfc941a7f1fb1"} Oct 30 21:00:20.007722 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:00:20.007698 5378 generic.go:356] "Generic (PLEG): container finished" podID="8ce46c99-ec66-4e61-8b6f-135d76c640e1" containerID="35d3fc32d46868470080fb357ea2e31ee7e46852a14a162472bd4d9cf2ac8cd9" exitCode=0 Oct 30 21:00:20.007829 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:00:20.007774 5378 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ca08f19a-td59w" event={"ID":"8ce46c99-ec66-4e61-8b6f-135d76c640e1","Type":"ContainerDied","Data":"35d3fc32d46868470080fb357ea2e31ee7e46852a14a162472bd4d9cf2ac8cd9"} Oct 30 21:00:20.025782 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:00:20.025761 5378 scope.go:122] "RemoveContainer" containerID="d9166a81387c271c4faff6e5462a54987d87fb7d45959934f825e9b9e3c4a6f7" Oct 30 21:00:20.159655 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:00:20.159629 5378 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-31baea91-bjrgz" Oct 30 21:00:20.172396 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:00:20.172369 5378 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-682999-29889900-97hqz"] Oct 30 21:00:20.176074 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:00:20.176053 5378 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-682999-29889900-97hqz"] Oct 30 21:00:20.286016 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:00:20.285989 5378 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889900-8slvz"] Oct 30 21:00:20.288701 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:00:20.288677 5378 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889900-8slvz"] Oct 30 21:00:20.302144 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:00:20.302118 5378 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/965b8faf-458e-499d-8cf2-4001dc4f97ef-kube-api-access-x5qsw\" (UniqueName: \"kubernetes.io/projected/965b8faf-458e-499d-8cf2-4001dc4f97ef-kube-api-access-x5qsw\") pod \"965b8faf-458e-499d-8cf2-4001dc4f97ef\" (UID: \"965b8faf-458e-499d-8cf2-4001dc4f97ef\") " Oct 30 21:00:20.302302 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:00:20.302284 5378 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/965b8faf-458e-499d-8cf2-4001dc4f97ef-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/965b8faf-458e-499d-8cf2-4001dc4f97ef-maas-api-openshift-ca\") pod \"965b8faf-458e-499d-8cf2-4001dc4f97ef\" (UID: \"965b8faf-458e-499d-8cf2-4001dc4f97ef\") " Oct 30 21:00:20.302396 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:00:20.302324 5378 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/965b8faf-458e-499d-8cf2-4001dc4f97ef-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/965b8faf-458e-499d-8cf2-4001dc4f97ef-maas-api-xks-ca\") pod \"965b8faf-458e-499d-8cf2-4001dc4f97ef\" (UID: \"965b8faf-458e-499d-8cf2-4001dc4f97ef\") " Oct 30 21:00:20.302707 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:00:20.302679 5378 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/965b8faf-458e-499d-8cf2-4001dc4f97ef-maas-api-openshift-ca" pod "965b8faf-458e-499d-8cf2-4001dc4f97ef" (UID: "965b8faf-458e-499d-8cf2-4001dc4f97ef"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:00:20.303136 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:00:20.303108 5378 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/965b8faf-458e-499d-8cf2-4001dc4f97ef-maas-api-openshift-ca\") on node \"ip-10-0-138-149.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:00:20.304322 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:00:20.304301 5378 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/965b8faf-458e-499d-8cf2-4001dc4f97ef-kube-api-access-x5qsw" pod "965b8faf-458e-499d-8cf2-4001dc4f97ef" (UID: "965b8faf-458e-499d-8cf2-4001dc4f97ef"). InnerVolumeSpecName "kube-api-access-x5qsw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:00:20.304418 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:00:20.304398 5378 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/965b8faf-458e-499d-8cf2-4001dc4f97ef-maas-api-xks-ca" pod "965b8faf-458e-499d-8cf2-4001dc4f97ef" (UID: "965b8faf-458e-499d-8cf2-4001dc4f97ef"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:00:20.404166 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:00:20.404129 5378 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/965b8faf-458e-499d-8cf2-4001dc4f97ef-maas-api-xks-ca\") on node \"ip-10-0-138-149.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:00:20.404166 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:00:20.404160 5378 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x5qsw\" (UniqueName: \"kubernetes.io/projected/965b8faf-458e-499d-8cf2-4001dc4f97ef-kube-api-access-x5qsw\") on node \"ip-10-0-138-149.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:00:21.017866 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:00:21.017789 5378 generic.go:356] "Generic (PLEG): container finished" podID="372b312c-da86-453f-8287-fe4603fd018c" containerID="c9e9cbac3a22a6db114c693bfc5dd2d5ad05de127b196c94ad56f79dd9f217d5" exitCode=60 Oct 30 21:00:21.017866 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:00:21.017854 5378 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-f9f277-29889900-b7hrn" event={"ID":"372b312c-da86-453f-8287-fe4603fd018c","Type":"ContainerDied","Data":"c9e9cbac3a22a6db114c693bfc5dd2d5ad05de127b196c94ad56f79dd9f217d5"} Oct 30 21:00:21.018337 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:00:21.017908 5378 scope.go:122] "RemoveContainer" containerID="2e1f04e33ec25169d49fc25ea9fe91f1681d35751fda61dea2f3a4d2b5248e30" Oct 30 21:00:21.018337 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:00:21.018318 5378 scope.go:122] "RemoveContainer" containerID="c9e9cbac3a22a6db114c693bfc5dd2d5ad05de127b196c94ad56f79dd9f217d5" Oct 30 21:00:21.018684 ip-10-0-138-149 kubenswrapper[5378]: E1030 21:00:21.018660 5378 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-f9f277-29889900-b7hrn_odh-ai-gateway-infra(372b312c-da86-453f-8287-fe4603fd018c)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-f9f277-29889900-b7hrn" podUID="372b312c-da86-453f-8287-fe4603fd018c" Oct 30 21:00:21.019377 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:00:21.019354 5378 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-31baea91-bjrgz" event={"ID":"965b8faf-458e-499d-8cf2-4001dc4f97ef","Type":"ContainerDied","Data":"37b45bf8c84ce628b17ee852c9cc1f0450cb35e3e7d46a3c1d7e6c56bc65c8eb"} Oct 30 21:00:21.019444 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:00:21.019380 5378 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-31baea91-bjrgz" Oct 30 21:00:21.019444 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:00:21.019401 5378 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="37b45bf8c84ce628b17ee852c9cc1f0450cb35e3e7d46a3c1d7e6c56bc65c8eb" Oct 30 21:00:21.159670 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:00:21.159644 5378 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ca08f19a-td59w" Oct 30 21:00:21.316122 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:00:21.316042 5378 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8ce46c99-ec66-4e61-8b6f-135d76c640e1-kube-api-access-w2kz7\" (UniqueName: \"kubernetes.io/projected/8ce46c99-ec66-4e61-8b6f-135d76c640e1-kube-api-access-w2kz7\") pod \"8ce46c99-ec66-4e61-8b6f-135d76c640e1\" (UID: \"8ce46c99-ec66-4e61-8b6f-135d76c640e1\") " Oct 30 21:00:21.316278 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:00:21.316177 5378 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8ce46c99-ec66-4e61-8b6f-135d76c640e1-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8ce46c99-ec66-4e61-8b6f-135d76c640e1-maas-api-openshift-ca\") pod \"8ce46c99-ec66-4e61-8b6f-135d76c640e1\" (UID: \"8ce46c99-ec66-4e61-8b6f-135d76c640e1\") " Oct 30 21:00:21.316345 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:00:21.316329 5378 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8ce46c99-ec66-4e61-8b6f-135d76c640e1-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8ce46c99-ec66-4e61-8b6f-135d76c640e1-maas-api-xks-ca\") pod \"8ce46c99-ec66-4e61-8b6f-135d76c640e1\" (UID: \"8ce46c99-ec66-4e61-8b6f-135d76c640e1\") " Oct 30 21:00:21.316848 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:00:21.316817 5378 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8ce46c99-ec66-4e61-8b6f-135d76c640e1-maas-api-openshift-ca" pod "8ce46c99-ec66-4e61-8b6f-135d76c640e1" (UID: "8ce46c99-ec66-4e61-8b6f-135d76c640e1"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:00:21.318819 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:00:21.318793 5378 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8ce46c99-ec66-4e61-8b6f-135d76c640e1-maas-api-xks-ca" pod "8ce46c99-ec66-4e61-8b6f-135d76c640e1" (UID: "8ce46c99-ec66-4e61-8b6f-135d76c640e1"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:00:21.319006 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:00:21.318979 5378 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8ce46c99-ec66-4e61-8b6f-135d76c640e1-kube-api-access-w2kz7" pod "8ce46c99-ec66-4e61-8b6f-135d76c640e1" (UID: "8ce46c99-ec66-4e61-8b6f-135d76c640e1"). InnerVolumeSpecName "kube-api-access-w2kz7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:00:21.418210 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:00:21.418174 5378 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8ce46c99-ec66-4e61-8b6f-135d76c640e1-maas-api-xks-ca\") on node \"ip-10-0-138-149.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:00:21.418210 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:00:21.418213 5378 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w2kz7\" (UniqueName: \"kubernetes.io/projected/8ce46c99-ec66-4e61-8b6f-135d76c640e1-kube-api-access-w2kz7\") on node \"ip-10-0-138-149.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:00:21.418478 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:00:21.418233 5378 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8ce46c99-ec66-4e61-8b6f-135d76c640e1-maas-api-openshift-ca\") on node \"ip-10-0-138-149.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:00:21.742427 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:00:21.742385 5378 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0e8b014e-a0de-474e-a123-01775c2e4c83" path="/var/lib/kubelet/pods/0e8b014e-a0de-474e-a123-01775c2e4c83/volumes" Oct 30 21:00:21.742972 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:00:21.742949 5378 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cbc0e737-1972-4f73-8457-7081264afdb7" path="/var/lib/kubelet/pods/cbc0e737-1972-4f73-8457-7081264afdb7/volumes" Oct 30 21:00:22.028545 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:00:22.028445 5378 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ca08f19a-td59w" Oct 30 21:00:22.029061 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:00:22.028433 5378 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ca08f19a-td59w" event={"ID":"8ce46c99-ec66-4e61-8b6f-135d76c640e1","Type":"ContainerDied","Data":"44594b82502af6de1d25b1806a109b70da10a61140556b2b7a48af3ee0f259c9"} Oct 30 21:00:22.029061 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:00:22.028675 5378 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="44594b82502af6de1d25b1806a109b70da10a61140556b2b7a48af3ee0f259c9" Oct 30 21:00:22.143062 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:00:22.143030 5378 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-f9f277-29889900-b7hrn"] Oct 30 21:00:22.306706 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:00:22.306682 5378 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-f9f277-29889900-b7hrn" Oct 30 21:00:22.430220 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:00:22.430182 5378 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/372b312c-da86-453f-8287-fe4603fd018c-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/372b312c-da86-453f-8287-fe4603fd018c-maas-api-service-ca\") pod \"372b312c-da86-453f-8287-fe4603fd018c\" (UID: \"372b312c-da86-453f-8287-fe4603fd018c\") " Oct 30 21:00:22.430435 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:00:22.430327 5378 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/372b312c-da86-453f-8287-fe4603fd018c-kube-api-access-vgkgb\" (UniqueName: \"kubernetes.io/projected/372b312c-da86-453f-8287-fe4603fd018c-kube-api-access-vgkgb\") pod \"372b312c-da86-453f-8287-fe4603fd018c\" (UID: \"372b312c-da86-453f-8287-fe4603fd018c\") " Oct 30 21:00:22.430884 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:00:22.430859 5378 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/372b312c-da86-453f-8287-fe4603fd018c-maas-api-service-ca" pod "372b312c-da86-453f-8287-fe4603fd018c" (UID: "372b312c-da86-453f-8287-fe4603fd018c"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:00:22.431030 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:00:22.431007 5378 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/372b312c-da86-453f-8287-fe4603fd018c-maas-api-service-ca\") on node \"ip-10-0-138-149.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:00:22.433111 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:00:22.433080 5378 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/372b312c-da86-453f-8287-fe4603fd018c-kube-api-access-vgkgb" pod "372b312c-da86-453f-8287-fe4603fd018c" (UID: "372b312c-da86-453f-8287-fe4603fd018c"). InnerVolumeSpecName "kube-api-access-vgkgb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:00:22.532585 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:00:22.532531 5378 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vgkgb\" (UniqueName: \"kubernetes.io/projected/372b312c-da86-453f-8287-fe4603fd018c-kube-api-access-vgkgb\") on node \"ip-10-0-138-149.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:00:23.039069 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:00:23.039041 5378 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-f9f277-29889900-b7hrn" Oct 30 21:00:23.039485 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:00:23.039076 5378 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-f9f277-29889900-b7hrn" event={"ID":"372b312c-da86-453f-8287-fe4603fd018c","Type":"ContainerDied","Data":"578ad2bb20b1b315cf1b49a1f29589f6e21c3497a400941c067182dd9e802657"} Oct 30 21:00:23.039485 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:00:23.039135 5378 scope.go:122] "RemoveContainer" containerID="c9e9cbac3a22a6db114c693bfc5dd2d5ad05de127b196c94ad56f79dd9f217d5" Oct 30 21:00:23.205605 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:00:23.205554 5378 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-f9f277-29889900-b7hrn"] Oct 30 21:00:23.257935 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:00:23.257903 5378 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-f9f277-29889900-b7hrn"] Oct 30 21:00:23.273549 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:00:23.273520 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:00:23.648438 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:00:23.648404 5378 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ea167fdf-769pg"] Oct 30 21:00:23.649071 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:00:23.649051 5378 cpu_manager.go:414] "containerMap: removing container" podUID="cbc0e737-1972-4f73-8457-7081264afdb7" containerName="cleanup" Oct 30 21:00:23.649174 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:00:23.649073 5378 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cbc0e737-1972-4f73-8457-7081264afdb7" containerName="cleanup" Oct 30 21:00:23.649174 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:00:23.649087 5378 cpu_manager.go:414] "containerMap: removing container" podUID="372b312c-da86-453f-8287-fe4603fd018c" containerName="cleanup" Oct 30 21:00:23.649174 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:00:23.649094 5378 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="372b312c-da86-453f-8287-fe4603fd018c" containerName="cleanup" Oct 30 21:00:23.649174 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:00:23.649107 5378 cpu_manager.go:414] "containerMap: removing container" podUID="0e8b014e-a0de-474e-a123-01775c2e4c83" containerName="cleanup" Oct 30 21:00:23.649174 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:00:23.649115 5378 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0e8b014e-a0de-474e-a123-01775c2e4c83" containerName="cleanup" Oct 30 21:00:23.649174 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:00:23.649124 5378 cpu_manager.go:414] "containerMap: removing container" podUID="cbc0e737-1972-4f73-8457-7081264afdb7" containerName="cleanup" Oct 30 21:00:23.649174 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:00:23.649130 5378 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cbc0e737-1972-4f73-8457-7081264afdb7" containerName="cleanup" Oct 30 21:00:23.649174 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:00:23.649147 5378 cpu_manager.go:414] "containerMap: removing container" podUID="372b312c-da86-453f-8287-fe4603fd018c" containerName="cleanup" Oct 30 21:00:23.649174 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:00:23.649153 5378 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="372b312c-da86-453f-8287-fe4603fd018c" containerName="cleanup" Oct 30 21:00:23.649174 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:00:23.649166 5378 cpu_manager.go:414] "containerMap: removing container" podUID="0e8b014e-a0de-474e-a123-01775c2e4c83" containerName="cleanup" Oct 30 21:00:23.649174 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:00:23.649173 5378 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0e8b014e-a0de-474e-a123-01775c2e4c83" containerName="cleanup" Oct 30 21:00:23.649715 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:00:23.649183 5378 cpu_manager.go:414] "containerMap: removing container" podUID="8ce46c99-ec66-4e61-8b6f-135d76c640e1" containerName="revoke-keys" Oct 30 21:00:23.649715 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:00:23.649190 5378 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8ce46c99-ec66-4e61-8b6f-135d76c640e1" containerName="revoke-keys" Oct 30 21:00:23.649715 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:00:23.649237 5378 cpu_manager.go:414] "containerMap: removing container" podUID="965b8faf-458e-499d-8cf2-4001dc4f97ef" containerName="revoke-keys" Oct 30 21:00:23.649715 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:00:23.649245 5378 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="965b8faf-458e-499d-8cf2-4001dc4f97ef" containerName="revoke-keys" Oct 30 21:00:23.649715 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:00:23.649405 5378 memory_manager.go:365] "RemoveStaleState removing state" podUID="cbc0e737-1972-4f73-8457-7081264afdb7" containerName="cleanup" Oct 30 21:00:23.649715 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:00:23.649417 5378 memory_manager.go:365] "RemoveStaleState removing state" podUID="372b312c-da86-453f-8287-fe4603fd018c" containerName="cleanup" Oct 30 21:00:23.649715 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:00:23.649426 5378 memory_manager.go:365] "RemoveStaleState removing state" podUID="0e8b014e-a0de-474e-a123-01775c2e4c83" containerName="cleanup" Oct 30 21:00:23.649715 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:00:23.649434 5378 memory_manager.go:365] "RemoveStaleState removing state" podUID="372b312c-da86-453f-8287-fe4603fd018c" containerName="cleanup" Oct 30 21:00:23.649715 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:00:23.649444 5378 memory_manager.go:365] "RemoveStaleState removing state" podUID="0e8b014e-a0de-474e-a123-01775c2e4c83" containerName="cleanup" Oct 30 21:00:23.649715 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:00:23.649451 5378 memory_manager.go:365] "RemoveStaleState removing state" podUID="8ce46c99-ec66-4e61-8b6f-135d76c640e1" containerName="revoke-keys" Oct 30 21:00:23.649715 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:00:23.649464 5378 memory_manager.go:365] "RemoveStaleState removing state" podUID="0e8b014e-a0de-474e-a123-01775c2e4c83" containerName="cleanup" Oct 30 21:00:23.649715 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:00:23.649474 5378 memory_manager.go:365] "RemoveStaleState removing state" podUID="372b312c-da86-453f-8287-fe4603fd018c" containerName="cleanup" Oct 30 21:00:23.649715 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:00:23.649483 5378 memory_manager.go:365] "RemoveStaleState removing state" podUID="cbc0e737-1972-4f73-8457-7081264afdb7" containerName="cleanup" Oct 30 21:00:23.649715 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:00:23.649491 5378 memory_manager.go:365] "RemoveStaleState removing state" podUID="965b8faf-458e-499d-8cf2-4001dc4f97ef" containerName="revoke-keys" Oct 30 21:00:23.654475 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:00:23.654442 5378 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ea167fdf-769pg" Oct 30 21:00:23.708489 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:00:23.708463 5378 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 21:00:23.709827 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:00:23.709472 5378 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ea167fdf-769pg"] Oct 30 21:00:23.743899 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:00:23.743865 5378 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="372b312c-da86-453f-8287-fe4603fd018c" path="/var/lib/kubelet/pods/372b312c-da86-453f-8287-fe4603fd018c/volumes" Oct 30 21:00:23.754911 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:00:23.754885 5378 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-q2mgc\"" Oct 30 21:00:23.758959 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:00:23.758915 5378 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j2jkm\" (UniqueName: \"kubernetes.io/projected/4b8072df-d2bb-4709-82c5-6e6a9ccd9de6-kube-api-access-j2jkm\") pod \"maas-api-revoke-sub-ea167fdf-769pg\" (UID: \"4b8072df-d2bb-4709-82c5-6e6a9ccd9de6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ea167fdf-769pg" Oct 30 21:00:23.759098 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:00:23.758990 5378 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4b8072df-d2bb-4709-82c5-6e6a9ccd9de6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ea167fdf-769pg\" (UID: \"4b8072df-d2bb-4709-82c5-6e6a9ccd9de6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ea167fdf-769pg" Oct 30 21:00:23.759165 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:00:23.759100 5378 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4b8072df-d2bb-4709-82c5-6e6a9ccd9de6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ea167fdf-769pg\" (UID: \"4b8072df-d2bb-4709-82c5-6e6a9ccd9de6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ea167fdf-769pg" Oct 30 21:00:23.860347 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:00:23.860307 5378 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j2jkm\" (UniqueName: \"kubernetes.io/projected/4b8072df-d2bb-4709-82c5-6e6a9ccd9de6-kube-api-access-j2jkm\") pod \"maas-api-revoke-sub-ea167fdf-769pg\" (UID: \"4b8072df-d2bb-4709-82c5-6e6a9ccd9de6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ea167fdf-769pg" Oct 30 21:00:23.860347 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:00:23.860353 5378 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4b8072df-d2bb-4709-82c5-6e6a9ccd9de6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ea167fdf-769pg\" (UID: \"4b8072df-d2bb-4709-82c5-6e6a9ccd9de6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ea167fdf-769pg" Oct 30 21:00:23.860540 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:00:23.860525 5378 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4b8072df-d2bb-4709-82c5-6e6a9ccd9de6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ea167fdf-769pg\" (UID: \"4b8072df-d2bb-4709-82c5-6e6a9ccd9de6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ea167fdf-769pg" Oct 30 21:00:23.861042 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:00:23.861019 5378 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4b8072df-d2bb-4709-82c5-6e6a9ccd9de6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ea167fdf-769pg\" (UID: \"4b8072df-d2bb-4709-82c5-6e6a9ccd9de6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ea167fdf-769pg" Oct 30 21:00:23.862813 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:00:23.862790 5378 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4b8072df-d2bb-4709-82c5-6e6a9ccd9de6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ea167fdf-769pg\" (UID: \"4b8072df-d2bb-4709-82c5-6e6a9ccd9de6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ea167fdf-769pg" Oct 30 21:00:23.917195 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:00:23.917137 5378 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j2jkm\" (UniqueName: \"kubernetes.io/projected/4b8072df-d2bb-4709-82c5-6e6a9ccd9de6-kube-api-access-j2jkm\") pod \"maas-api-revoke-sub-ea167fdf-769pg\" (UID: \"4b8072df-d2bb-4709-82c5-6e6a9ccd9de6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ea167fdf-769pg" Oct 30 21:00:23.967814 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:00:23.967782 5378 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ea167fdf-769pg" Oct 30 21:00:24.362709 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:00:24.362672 5378 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ea167fdf-769pg"] Oct 30 21:00:24.363906 ip-10-0-138-149 kubenswrapper[5378]: W1030 21:00:24.363876 5378 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4b8072df_d2bb_4709_82c5_6e6a9ccd9de6.slice/crio-dbc548e6708afc88133662797245fd8ebdd55e33d61860fede6b0b9ac94b628b WatchSource:0}: Error finding container dbc548e6708afc88133662797245fd8ebdd55e33d61860fede6b0b9ac94b628b: Status 404 returned error can't find the container with id dbc548e6708afc88133662797245fd8ebdd55e33d61860fede6b0b9ac94b628b Oct 30 21:00:25.068333 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:00:25.068255 5378 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ea167fdf-769pg" event={"ID":"4b8072df-d2bb-4709-82c5-6e6a9ccd9de6","Type":"ContainerStarted","Data":"4d310153550a3bb25aca95f21baacaec9e19fcd98b0b127345a928c8ff837039"} Oct 30 21:00:25.068333 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:00:25.068315 5378 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ea167fdf-769pg" event={"ID":"4b8072df-d2bb-4709-82c5-6e6a9ccd9de6","Type":"ContainerStarted","Data":"dbc548e6708afc88133662797245fd8ebdd55e33d61860fede6b0b9ac94b628b"} Oct 30 21:00:25.182584 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:00:25.182526 5378 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ea167fdf-769pg" podStartSLOduration=2.182513012 podStartE2EDuration="2.182513012s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:00:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:00:25.181888062 +0000 UTC m=+1315.610001981" watchObservedRunningTime="2026-10-30 21:00:25.182513012 +0000 UTC m=+1315.610626930" Oct 30 21:00:26.076940 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:00:26.076904 5378 generic.go:356] "Generic (PLEG): container finished" podID="4b8072df-d2bb-4709-82c5-6e6a9ccd9de6" containerID="4d310153550a3bb25aca95f21baacaec9e19fcd98b0b127345a928c8ff837039" exitCode=0 Oct 30 21:00:26.077346 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:00:26.076997 5378 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ea167fdf-769pg" event={"ID":"4b8072df-d2bb-4709-82c5-6e6a9ccd9de6","Type":"ContainerDied","Data":"4d310153550a3bb25aca95f21baacaec9e19fcd98b0b127345a928c8ff837039"} Oct 30 21:00:27.272828 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:00:27.272800 5378 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ea167fdf-769pg" Oct 30 21:00:27.305010 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:00:27.304982 5378 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4b8072df-d2bb-4709-82c5-6e6a9ccd9de6-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4b8072df-d2bb-4709-82c5-6e6a9ccd9de6-maas-api-openshift-ca\") pod \"4b8072df-d2bb-4709-82c5-6e6a9ccd9de6\" (UID: \"4b8072df-d2bb-4709-82c5-6e6a9ccd9de6\") " Oct 30 21:00:27.305140 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:00:27.305086 5378 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4b8072df-d2bb-4709-82c5-6e6a9ccd9de6-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4b8072df-d2bb-4709-82c5-6e6a9ccd9de6-maas-api-xks-ca\") pod \"4b8072df-d2bb-4709-82c5-6e6a9ccd9de6\" (UID: \"4b8072df-d2bb-4709-82c5-6e6a9ccd9de6\") " Oct 30 21:00:27.305140 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:00:27.305121 5378 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4b8072df-d2bb-4709-82c5-6e6a9ccd9de6-kube-api-access-j2jkm\" (UniqueName: \"kubernetes.io/projected/4b8072df-d2bb-4709-82c5-6e6a9ccd9de6-kube-api-access-j2jkm\") pod \"4b8072df-d2bb-4709-82c5-6e6a9ccd9de6\" (UID: \"4b8072df-d2bb-4709-82c5-6e6a9ccd9de6\") " Oct 30 21:00:27.305366 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:00:27.305342 5378 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4b8072df-d2bb-4709-82c5-6e6a9ccd9de6-maas-api-openshift-ca" pod "4b8072df-d2bb-4709-82c5-6e6a9ccd9de6" (UID: "4b8072df-d2bb-4709-82c5-6e6a9ccd9de6"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:00:27.305639 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:00:27.305623 5378 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4b8072df-d2bb-4709-82c5-6e6a9ccd9de6-maas-api-openshift-ca\") on node \"ip-10-0-138-149.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:00:27.307269 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:00:27.307248 5378 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4b8072df-d2bb-4709-82c5-6e6a9ccd9de6-kube-api-access-j2jkm" pod "4b8072df-d2bb-4709-82c5-6e6a9ccd9de6" (UID: "4b8072df-d2bb-4709-82c5-6e6a9ccd9de6"). InnerVolumeSpecName "kube-api-access-j2jkm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:00:27.307353 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:00:27.307324 5378 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4b8072df-d2bb-4709-82c5-6e6a9ccd9de6-maas-api-xks-ca" pod "4b8072df-d2bb-4709-82c5-6e6a9ccd9de6" (UID: "4b8072df-d2bb-4709-82c5-6e6a9ccd9de6"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:00:27.407167 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:00:27.407138 5378 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j2jkm\" (UniqueName: \"kubernetes.io/projected/4b8072df-d2bb-4709-82c5-6e6a9ccd9de6-kube-api-access-j2jkm\") on node \"ip-10-0-138-149.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:00:27.407167 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:00:27.407168 5378 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4b8072df-d2bb-4709-82c5-6e6a9ccd9de6-maas-api-xks-ca\") on node \"ip-10-0-138-149.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:00:28.093395 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:00:28.093316 5378 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ea167fdf-769pg" Oct 30 21:00:28.093395 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:00:28.093340 5378 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ea167fdf-769pg" event={"ID":"4b8072df-d2bb-4709-82c5-6e6a9ccd9de6","Type":"ContainerDied","Data":"dbc548e6708afc88133662797245fd8ebdd55e33d61860fede6b0b9ac94b628b"} Oct 30 21:00:28.093395 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:00:28.093385 5378 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="dbc548e6708afc88133662797245fd8ebdd55e33d61860fede6b0b9ac94b628b" Oct 30 21:00:28.096603 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:00:28.096553 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:00:29.939001 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:00:29.938969 5378 scope.go:122] "RemoveContainer" containerID="6560457e1da2e780652f5972cf84904e5f3358c9e801458326fd6e5b7ded1736" Oct 30 21:00:29.948798 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:00:29.948779 5378 scope.go:122] "RemoveContainer" containerID="1f1196e486a3a5be083266069fbdacdb2adb74abb055d3457513abd9edae8aac" Oct 30 21:00:32.494269 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:00:32.494237 5378 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w5-2b425d-openshift-default-f547dbcbf-j99m9"] Oct 30 21:00:32.494853 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:00:32.494839 5378 cpu_manager.go:414] "containerMap: removing container" podUID="4b8072df-d2bb-4709-82c5-6e6a9ccd9de6" containerName="revoke-keys" Oct 30 21:00:32.494926 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:00:32.494853 5378 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4b8072df-d2bb-4709-82c5-6e6a9ccd9de6" containerName="revoke-keys" Oct 30 21:00:32.494926 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:00:32.494887 5378 cpu_manager.go:414] "containerMap: removing container" podUID="cbc0e737-1972-4f73-8457-7081264afdb7" containerName="cleanup" Oct 30 21:00:32.494926 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:00:32.494892 5378 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cbc0e737-1972-4f73-8457-7081264afdb7" containerName="cleanup" Oct 30 21:00:32.494926 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:00:32.494902 5378 cpu_manager.go:414] "containerMap: removing container" podUID="372b312c-da86-453f-8287-fe4603fd018c" containerName="cleanup" Oct 30 21:00:32.494926 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:00:32.494907 5378 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="372b312c-da86-453f-8287-fe4603fd018c" containerName="cleanup" Oct 30 21:00:32.494926 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:00:32.494916 5378 cpu_manager.go:414] "containerMap: removing container" podUID="0e8b014e-a0de-474e-a123-01775c2e4c83" containerName="cleanup" Oct 30 21:00:32.494926 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:00:32.494920 5378 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0e8b014e-a0de-474e-a123-01775c2e4c83" containerName="cleanup" Oct 30 21:00:32.495121 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:00:32.495023 5378 memory_manager.go:365] "RemoveStaleState removing state" podUID="4b8072df-d2bb-4709-82c5-6e6a9ccd9de6" containerName="revoke-keys" Oct 30 21:00:32.495121 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:00:32.495032 5378 memory_manager.go:365] "RemoveStaleState removing state" podUID="cbc0e737-1972-4f73-8457-7081264afdb7" containerName="cleanup" Oct 30 21:00:32.500176 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:00:32.500156 5378 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-2b425d-openshift-default-f547dbcbf-j99m9" Oct 30 21:00:32.593298 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:00:32.593267 5378 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-2b425d-openshift-default-f547dbcbf-j99m9"] Oct 30 21:00:32.609026 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:00:32.608998 5378 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rkgqv\" (UniqueName: \"kubernetes.io/projected/ad099f6e-3035-4a6d-8fef-aea2e4bd1f3b-kube-api-access-rkgqv\") pod \"e2e-worker-w5-2b425d-openshift-default-f547dbcbf-j99m9\" (UID: \"ad099f6e-3035-4a6d-8fef-aea2e4bd1f3b\") " pod="openshift-ingress/e2e-worker-w5-2b425d-openshift-default-f547dbcbf-j99m9" Oct 30 21:00:32.609202 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:00:32.609180 5378 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ad099f6e-3035-4a6d-8fef-aea2e4bd1f3b-istio-podinfo\") pod \"e2e-worker-w5-2b425d-openshift-default-f547dbcbf-j99m9\" (UID: \"ad099f6e-3035-4a6d-8fef-aea2e4bd1f3b\") " pod="openshift-ingress/e2e-worker-w5-2b425d-openshift-default-f547dbcbf-j99m9" Oct 30 21:00:32.609264 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:00:32.609227 5378 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ad099f6e-3035-4a6d-8fef-aea2e4bd1f3b-workload-socket\") pod \"e2e-worker-w5-2b425d-openshift-default-f547dbcbf-j99m9\" (UID: \"ad099f6e-3035-4a6d-8fef-aea2e4bd1f3b\") " pod="openshift-ingress/e2e-worker-w5-2b425d-openshift-default-f547dbcbf-j99m9" Oct 30 21:00:32.609349 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:00:32.609332 5378 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ad099f6e-3035-4a6d-8fef-aea2e4bd1f3b-istio-envoy\") pod \"e2e-worker-w5-2b425d-openshift-default-f547dbcbf-j99m9\" (UID: \"ad099f6e-3035-4a6d-8fef-aea2e4bd1f3b\") " pod="openshift-ingress/e2e-worker-w5-2b425d-openshift-default-f547dbcbf-j99m9" Oct 30 21:00:32.609399 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:00:32.609378 5378 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ad099f6e-3035-4a6d-8fef-aea2e4bd1f3b-credential-socket\") pod \"e2e-worker-w5-2b425d-openshift-default-f547dbcbf-j99m9\" (UID: \"ad099f6e-3035-4a6d-8fef-aea2e4bd1f3b\") " pod="openshift-ingress/e2e-worker-w5-2b425d-openshift-default-f547dbcbf-j99m9" Oct 30 21:00:32.609434 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:00:32.609419 5378 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ad099f6e-3035-4a6d-8fef-aea2e4bd1f3b-istio-data\") pod \"e2e-worker-w5-2b425d-openshift-default-f547dbcbf-j99m9\" (UID: \"ad099f6e-3035-4a6d-8fef-aea2e4bd1f3b\") " pod="openshift-ingress/e2e-worker-w5-2b425d-openshift-default-f547dbcbf-j99m9" Oct 30 21:00:32.609475 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:00:32.609434 5378 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ad099f6e-3035-4a6d-8fef-aea2e4bd1f3b-istio-token\") pod \"e2e-worker-w5-2b425d-openshift-default-f547dbcbf-j99m9\" (UID: \"ad099f6e-3035-4a6d-8fef-aea2e4bd1f3b\") " pod="openshift-ingress/e2e-worker-w5-2b425d-openshift-default-f547dbcbf-j99m9" Oct 30 21:00:32.609475 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:00:32.609465 5378 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ad099f6e-3035-4a6d-8fef-aea2e4bd1f3b-istiod-ca-cert\") pod \"e2e-worker-w5-2b425d-openshift-default-f547dbcbf-j99m9\" (UID: \"ad099f6e-3035-4a6d-8fef-aea2e4bd1f3b\") " pod="openshift-ingress/e2e-worker-w5-2b425d-openshift-default-f547dbcbf-j99m9" Oct 30 21:00:32.609574 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:00:32.609550 5378 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ad099f6e-3035-4a6d-8fef-aea2e4bd1f3b-workload-certs\") pod \"e2e-worker-w5-2b425d-openshift-default-f547dbcbf-j99m9\" (UID: \"ad099f6e-3035-4a6d-8fef-aea2e4bd1f3b\") " pod="openshift-ingress/e2e-worker-w5-2b425d-openshift-default-f547dbcbf-j99m9" Oct 30 21:00:32.710741 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:00:32.710702 5378 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ad099f6e-3035-4a6d-8fef-aea2e4bd1f3b-istio-podinfo\") pod \"e2e-worker-w5-2b425d-openshift-default-f547dbcbf-j99m9\" (UID: \"ad099f6e-3035-4a6d-8fef-aea2e4bd1f3b\") " pod="openshift-ingress/e2e-worker-w5-2b425d-openshift-default-f547dbcbf-j99m9" Oct 30 21:00:32.710893 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:00:32.710767 5378 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ad099f6e-3035-4a6d-8fef-aea2e4bd1f3b-workload-socket\") pod \"e2e-worker-w5-2b425d-openshift-default-f547dbcbf-j99m9\" (UID: \"ad099f6e-3035-4a6d-8fef-aea2e4bd1f3b\") " pod="openshift-ingress/e2e-worker-w5-2b425d-openshift-default-f547dbcbf-j99m9" Oct 30 21:00:32.710893 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:00:32.710823 5378 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ad099f6e-3035-4a6d-8fef-aea2e4bd1f3b-istio-envoy\") pod \"e2e-worker-w5-2b425d-openshift-default-f547dbcbf-j99m9\" (UID: \"ad099f6e-3035-4a6d-8fef-aea2e4bd1f3b\") " pod="openshift-ingress/e2e-worker-w5-2b425d-openshift-default-f547dbcbf-j99m9" Oct 30 21:00:32.710893 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:00:32.710882 5378 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ad099f6e-3035-4a6d-8fef-aea2e4bd1f3b-credential-socket\") pod \"e2e-worker-w5-2b425d-openshift-default-f547dbcbf-j99m9\" (UID: \"ad099f6e-3035-4a6d-8fef-aea2e4bd1f3b\") " pod="openshift-ingress/e2e-worker-w5-2b425d-openshift-default-f547dbcbf-j99m9" Oct 30 21:00:32.711076 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:00:32.710933 5378 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ad099f6e-3035-4a6d-8fef-aea2e4bd1f3b-istio-data\") pod \"e2e-worker-w5-2b425d-openshift-default-f547dbcbf-j99m9\" (UID: \"ad099f6e-3035-4a6d-8fef-aea2e4bd1f3b\") " pod="openshift-ingress/e2e-worker-w5-2b425d-openshift-default-f547dbcbf-j99m9" Oct 30 21:00:32.711076 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:00:32.710960 5378 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ad099f6e-3035-4a6d-8fef-aea2e4bd1f3b-istio-token\") pod \"e2e-worker-w5-2b425d-openshift-default-f547dbcbf-j99m9\" (UID: \"ad099f6e-3035-4a6d-8fef-aea2e4bd1f3b\") " pod="openshift-ingress/e2e-worker-w5-2b425d-openshift-default-f547dbcbf-j99m9" Oct 30 21:00:32.711076 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:00:32.711012 5378 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ad099f6e-3035-4a6d-8fef-aea2e4bd1f3b-istiod-ca-cert\") pod \"e2e-worker-w5-2b425d-openshift-default-f547dbcbf-j99m9\" (UID: \"ad099f6e-3035-4a6d-8fef-aea2e4bd1f3b\") " pod="openshift-ingress/e2e-worker-w5-2b425d-openshift-default-f547dbcbf-j99m9" Oct 30 21:00:32.711248 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:00:32.711082 5378 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ad099f6e-3035-4a6d-8fef-aea2e4bd1f3b-workload-certs\") pod \"e2e-worker-w5-2b425d-openshift-default-f547dbcbf-j99m9\" (UID: \"ad099f6e-3035-4a6d-8fef-aea2e4bd1f3b\") " pod="openshift-ingress/e2e-worker-w5-2b425d-openshift-default-f547dbcbf-j99m9" Oct 30 21:00:32.711248 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:00:32.711139 5378 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rkgqv\" (UniqueName: \"kubernetes.io/projected/ad099f6e-3035-4a6d-8fef-aea2e4bd1f3b-kube-api-access-rkgqv\") pod \"e2e-worker-w5-2b425d-openshift-default-f547dbcbf-j99m9\" (UID: \"ad099f6e-3035-4a6d-8fef-aea2e4bd1f3b\") " pod="openshift-ingress/e2e-worker-w5-2b425d-openshift-default-f547dbcbf-j99m9" Oct 30 21:00:32.711248 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:00:32.711184 5378 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ad099f6e-3035-4a6d-8fef-aea2e4bd1f3b-workload-socket\") pod \"e2e-worker-w5-2b425d-openshift-default-f547dbcbf-j99m9\" (UID: \"ad099f6e-3035-4a6d-8fef-aea2e4bd1f3b\") " pod="openshift-ingress/e2e-worker-w5-2b425d-openshift-default-f547dbcbf-j99m9" Oct 30 21:00:32.711401 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:00:32.711327 5378 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ad099f6e-3035-4a6d-8fef-aea2e4bd1f3b-istio-data\") pod \"e2e-worker-w5-2b425d-openshift-default-f547dbcbf-j99m9\" (UID: \"ad099f6e-3035-4a6d-8fef-aea2e4bd1f3b\") " pod="openshift-ingress/e2e-worker-w5-2b425d-openshift-default-f547dbcbf-j99m9" Oct 30 21:00:32.711464 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:00:32.711450 5378 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ad099f6e-3035-4a6d-8fef-aea2e4bd1f3b-workload-certs\") pod \"e2e-worker-w5-2b425d-openshift-default-f547dbcbf-j99m9\" (UID: \"ad099f6e-3035-4a6d-8fef-aea2e4bd1f3b\") " pod="openshift-ingress/e2e-worker-w5-2b425d-openshift-default-f547dbcbf-j99m9" Oct 30 21:00:32.711729 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:00:32.711701 5378 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ad099f6e-3035-4a6d-8fef-aea2e4bd1f3b-credential-socket\") pod \"e2e-worker-w5-2b425d-openshift-default-f547dbcbf-j99m9\" (UID: \"ad099f6e-3035-4a6d-8fef-aea2e4bd1f3b\") " pod="openshift-ingress/e2e-worker-w5-2b425d-openshift-default-f547dbcbf-j99m9" Oct 30 21:00:32.712237 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:00:32.712210 5378 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ad099f6e-3035-4a6d-8fef-aea2e4bd1f3b-istiod-ca-cert\") pod \"e2e-worker-w5-2b425d-openshift-default-f547dbcbf-j99m9\" (UID: \"ad099f6e-3035-4a6d-8fef-aea2e4bd1f3b\") " pod="openshift-ingress/e2e-worker-w5-2b425d-openshift-default-f547dbcbf-j99m9" Oct 30 21:00:32.713343 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:00:32.713313 5378 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ad099f6e-3035-4a6d-8fef-aea2e4bd1f3b-istio-envoy\") pod \"e2e-worker-w5-2b425d-openshift-default-f547dbcbf-j99m9\" (UID: \"ad099f6e-3035-4a6d-8fef-aea2e4bd1f3b\") " pod="openshift-ingress/e2e-worker-w5-2b425d-openshift-default-f547dbcbf-j99m9" Oct 30 21:00:32.714211 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:00:32.714185 5378 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ad099f6e-3035-4a6d-8fef-aea2e4bd1f3b-istio-podinfo\") pod \"e2e-worker-w5-2b425d-openshift-default-f547dbcbf-j99m9\" (UID: \"ad099f6e-3035-4a6d-8fef-aea2e4bd1f3b\") " pod="openshift-ingress/e2e-worker-w5-2b425d-openshift-default-f547dbcbf-j99m9" Oct 30 21:00:32.772456 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:00:32.772384 5378 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ad099f6e-3035-4a6d-8fef-aea2e4bd1f3b-istio-token\") pod \"e2e-worker-w5-2b425d-openshift-default-f547dbcbf-j99m9\" (UID: \"ad099f6e-3035-4a6d-8fef-aea2e4bd1f3b\") " pod="openshift-ingress/e2e-worker-w5-2b425d-openshift-default-f547dbcbf-j99m9" Oct 30 21:00:32.772594 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:00:32.772524 5378 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rkgqv\" (UniqueName: \"kubernetes.io/projected/ad099f6e-3035-4a6d-8fef-aea2e4bd1f3b-kube-api-access-rkgqv\") pod \"e2e-worker-w5-2b425d-openshift-default-f547dbcbf-j99m9\" (UID: \"ad099f6e-3035-4a6d-8fef-aea2e4bd1f3b\") " pod="openshift-ingress/e2e-worker-w5-2b425d-openshift-default-f547dbcbf-j99m9" Oct 30 21:00:32.813961 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:00:32.813936 5378 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-2b425d-openshift-default-f547dbcbf-j99m9" Oct 30 21:00:32.846112 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:00:32.846077 5378 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-781058-openshift-default-55fb495b7-2c7gl"] Oct 30 21:00:32.856959 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:00:32.856931 5378 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-781058-openshift-default-55fb495b7-2c7gl" Oct 30 21:00:32.914602 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:00:32.913618 5378 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/507cd7ee-c139-489c-a520-292d8ebd91e4-istio-token\") pod \"e2e-worker-w0-781058-openshift-default-55fb495b7-2c7gl\" (UID: \"507cd7ee-c139-489c-a520-292d8ebd91e4\") " pod="openshift-ingress/e2e-worker-w0-781058-openshift-default-55fb495b7-2c7gl" Oct 30 21:00:32.914602 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:00:32.913840 5378 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/507cd7ee-c139-489c-a520-292d8ebd91e4-credential-socket\") pod \"e2e-worker-w0-781058-openshift-default-55fb495b7-2c7gl\" (UID: \"507cd7ee-c139-489c-a520-292d8ebd91e4\") " pod="openshift-ingress/e2e-worker-w0-781058-openshift-default-55fb495b7-2c7gl" Oct 30 21:00:32.914602 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:00:32.913878 5378 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/507cd7ee-c139-489c-a520-292d8ebd91e4-istio-envoy\") pod \"e2e-worker-w0-781058-openshift-default-55fb495b7-2c7gl\" (UID: \"507cd7ee-c139-489c-a520-292d8ebd91e4\") " pod="openshift-ingress/e2e-worker-w0-781058-openshift-default-55fb495b7-2c7gl" Oct 30 21:00:32.914602 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:00:32.913934 5378 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/507cd7ee-c139-489c-a520-292d8ebd91e4-workload-certs\") pod \"e2e-worker-w0-781058-openshift-default-55fb495b7-2c7gl\" (UID: \"507cd7ee-c139-489c-a520-292d8ebd91e4\") " pod="openshift-ingress/e2e-worker-w0-781058-openshift-default-55fb495b7-2c7gl" Oct 30 21:00:32.914602 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:00:32.913959 5378 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/507cd7ee-c139-489c-a520-292d8ebd91e4-istiod-ca-cert\") pod \"e2e-worker-w0-781058-openshift-default-55fb495b7-2c7gl\" (UID: \"507cd7ee-c139-489c-a520-292d8ebd91e4\") " pod="openshift-ingress/e2e-worker-w0-781058-openshift-default-55fb495b7-2c7gl" Oct 30 21:00:32.914602 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:00:32.913994 5378 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/507cd7ee-c139-489c-a520-292d8ebd91e4-istio-podinfo\") pod \"e2e-worker-w0-781058-openshift-default-55fb495b7-2c7gl\" (UID: \"507cd7ee-c139-489c-a520-292d8ebd91e4\") " pod="openshift-ingress/e2e-worker-w0-781058-openshift-default-55fb495b7-2c7gl" Oct 30 21:00:32.914602 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:00:32.914025 5378 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5tzd9\" (UniqueName: \"kubernetes.io/projected/507cd7ee-c139-489c-a520-292d8ebd91e4-kube-api-access-5tzd9\") pod \"e2e-worker-w0-781058-openshift-default-55fb495b7-2c7gl\" (UID: \"507cd7ee-c139-489c-a520-292d8ebd91e4\") " pod="openshift-ingress/e2e-worker-w0-781058-openshift-default-55fb495b7-2c7gl" Oct 30 21:00:32.914602 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:00:32.914053 5378 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/507cd7ee-c139-489c-a520-292d8ebd91e4-istio-data\") pod \"e2e-worker-w0-781058-openshift-default-55fb495b7-2c7gl\" (UID: \"507cd7ee-c139-489c-a520-292d8ebd91e4\") " pod="openshift-ingress/e2e-worker-w0-781058-openshift-default-55fb495b7-2c7gl" Oct 30 21:00:32.914602 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:00:32.914145 5378 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/507cd7ee-c139-489c-a520-292d8ebd91e4-workload-socket\") pod \"e2e-worker-w0-781058-openshift-default-55fb495b7-2c7gl\" (UID: \"507cd7ee-c139-489c-a520-292d8ebd91e4\") " pod="openshift-ingress/e2e-worker-w0-781058-openshift-default-55fb495b7-2c7gl" Oct 30 21:00:32.918995 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:00:32.918977 5378 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-781058-openshift-default-55fb495b7-2c7gl"] Oct 30 21:00:32.990456 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:00:32.990425 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:00:33.015944 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:00:33.015914 5378 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/507cd7ee-c139-489c-a520-292d8ebd91e4-credential-socket\") pod \"e2e-worker-w0-781058-openshift-default-55fb495b7-2c7gl\" (UID: \"507cd7ee-c139-489c-a520-292d8ebd91e4\") " pod="openshift-ingress/e2e-worker-w0-781058-openshift-default-55fb495b7-2c7gl" Oct 30 21:00:33.016106 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:00:33.015954 5378 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/507cd7ee-c139-489c-a520-292d8ebd91e4-istio-envoy\") pod \"e2e-worker-w0-781058-openshift-default-55fb495b7-2c7gl\" (UID: \"507cd7ee-c139-489c-a520-292d8ebd91e4\") " pod="openshift-ingress/e2e-worker-w0-781058-openshift-default-55fb495b7-2c7gl" Oct 30 21:00:33.016106 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:00:33.015993 5378 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/507cd7ee-c139-489c-a520-292d8ebd91e4-workload-certs\") pod \"e2e-worker-w0-781058-openshift-default-55fb495b7-2c7gl\" (UID: \"507cd7ee-c139-489c-a520-292d8ebd91e4\") " pod="openshift-ingress/e2e-worker-w0-781058-openshift-default-55fb495b7-2c7gl" Oct 30 21:00:33.016106 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:00:33.016010 5378 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/507cd7ee-c139-489c-a520-292d8ebd91e4-istiod-ca-cert\") pod \"e2e-worker-w0-781058-openshift-default-55fb495b7-2c7gl\" (UID: \"507cd7ee-c139-489c-a520-292d8ebd91e4\") " pod="openshift-ingress/e2e-worker-w0-781058-openshift-default-55fb495b7-2c7gl" Oct 30 21:00:33.016106 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:00:33.016033 5378 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/507cd7ee-c139-489c-a520-292d8ebd91e4-istio-podinfo\") pod \"e2e-worker-w0-781058-openshift-default-55fb495b7-2c7gl\" (UID: \"507cd7ee-c139-489c-a520-292d8ebd91e4\") " pod="openshift-ingress/e2e-worker-w0-781058-openshift-default-55fb495b7-2c7gl" Oct 30 21:00:33.016106 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:00:33.016061 5378 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5tzd9\" (UniqueName: \"kubernetes.io/projected/507cd7ee-c139-489c-a520-292d8ebd91e4-kube-api-access-5tzd9\") pod \"e2e-worker-w0-781058-openshift-default-55fb495b7-2c7gl\" (UID: \"507cd7ee-c139-489c-a520-292d8ebd91e4\") " pod="openshift-ingress/e2e-worker-w0-781058-openshift-default-55fb495b7-2c7gl" Oct 30 21:00:33.016106 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:00:33.016088 5378 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/507cd7ee-c139-489c-a520-292d8ebd91e4-istio-data\") pod \"e2e-worker-w0-781058-openshift-default-55fb495b7-2c7gl\" (UID: \"507cd7ee-c139-489c-a520-292d8ebd91e4\") " pod="openshift-ingress/e2e-worker-w0-781058-openshift-default-55fb495b7-2c7gl" Oct 30 21:00:33.016412 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:00:33.016179 5378 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/507cd7ee-c139-489c-a520-292d8ebd91e4-workload-socket\") pod \"e2e-worker-w0-781058-openshift-default-55fb495b7-2c7gl\" (UID: \"507cd7ee-c139-489c-a520-292d8ebd91e4\") " pod="openshift-ingress/e2e-worker-w0-781058-openshift-default-55fb495b7-2c7gl" Oct 30 21:00:33.016412 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:00:33.016238 5378 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/507cd7ee-c139-489c-a520-292d8ebd91e4-istio-token\") pod \"e2e-worker-w0-781058-openshift-default-55fb495b7-2c7gl\" (UID: \"507cd7ee-c139-489c-a520-292d8ebd91e4\") " pod="openshift-ingress/e2e-worker-w0-781058-openshift-default-55fb495b7-2c7gl" Oct 30 21:00:33.016412 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:00:33.016354 5378 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/507cd7ee-c139-489c-a520-292d8ebd91e4-credential-socket\") pod \"e2e-worker-w0-781058-openshift-default-55fb495b7-2c7gl\" (UID: \"507cd7ee-c139-489c-a520-292d8ebd91e4\") " pod="openshift-ingress/e2e-worker-w0-781058-openshift-default-55fb495b7-2c7gl" Oct 30 21:00:33.016769 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:00:33.016746 5378 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/507cd7ee-c139-489c-a520-292d8ebd91e4-istiod-ca-cert\") pod \"e2e-worker-w0-781058-openshift-default-55fb495b7-2c7gl\" (UID: \"507cd7ee-c139-489c-a520-292d8ebd91e4\") " pod="openshift-ingress/e2e-worker-w0-781058-openshift-default-55fb495b7-2c7gl" Oct 30 21:00:33.016847 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:00:33.016756 5378 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/507cd7ee-c139-489c-a520-292d8ebd91e4-workload-socket\") pod \"e2e-worker-w0-781058-openshift-default-55fb495b7-2c7gl\" (UID: \"507cd7ee-c139-489c-a520-292d8ebd91e4\") " pod="openshift-ingress/e2e-worker-w0-781058-openshift-default-55fb495b7-2c7gl" Oct 30 21:00:33.016926 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:00:33.016907 5378 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/507cd7ee-c139-489c-a520-292d8ebd91e4-workload-certs\") pod \"e2e-worker-w0-781058-openshift-default-55fb495b7-2c7gl\" (UID: \"507cd7ee-c139-489c-a520-292d8ebd91e4\") " pod="openshift-ingress/e2e-worker-w0-781058-openshift-default-55fb495b7-2c7gl" Oct 30 21:00:33.016967 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:00:33.016928 5378 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/507cd7ee-c139-489c-a520-292d8ebd91e4-istio-data\") pod \"e2e-worker-w0-781058-openshift-default-55fb495b7-2c7gl\" (UID: \"507cd7ee-c139-489c-a520-292d8ebd91e4\") " pod="openshift-ingress/e2e-worker-w0-781058-openshift-default-55fb495b7-2c7gl" Oct 30 21:00:33.018484 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:00:33.018463 5378 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/507cd7ee-c139-489c-a520-292d8ebd91e4-istio-envoy\") pod \"e2e-worker-w0-781058-openshift-default-55fb495b7-2c7gl\" (UID: \"507cd7ee-c139-489c-a520-292d8ebd91e4\") " pod="openshift-ingress/e2e-worker-w0-781058-openshift-default-55fb495b7-2c7gl" Oct 30 21:00:33.018653 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:00:33.018634 5378 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/507cd7ee-c139-489c-a520-292d8ebd91e4-istio-podinfo\") pod \"e2e-worker-w0-781058-openshift-default-55fb495b7-2c7gl\" (UID: \"507cd7ee-c139-489c-a520-292d8ebd91e4\") " pod="openshift-ingress/e2e-worker-w0-781058-openshift-default-55fb495b7-2c7gl" Oct 30 21:00:33.083423 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:00:33.083296 5378 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f3113105-xn7nh"] Oct 30 21:00:33.092421 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:00:33.092348 5378 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f3113105-xn7nh"] Oct 30 21:00:33.107158 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:00:33.100098 5378 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5tzd9\" (UniqueName: \"kubernetes.io/projected/507cd7ee-c139-489c-a520-292d8ebd91e4-kube-api-access-5tzd9\") pod \"e2e-worker-w0-781058-openshift-default-55fb495b7-2c7gl\" (UID: \"507cd7ee-c139-489c-a520-292d8ebd91e4\") " pod="openshift-ingress/e2e-worker-w0-781058-openshift-default-55fb495b7-2c7gl" Oct 30 21:00:33.107447 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:00:33.107419 5378 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/507cd7ee-c139-489c-a520-292d8ebd91e4-istio-token\") pod \"e2e-worker-w0-781058-openshift-default-55fb495b7-2c7gl\" (UID: \"507cd7ee-c139-489c-a520-292d8ebd91e4\") " pod="openshift-ingress/e2e-worker-w0-781058-openshift-default-55fb495b7-2c7gl" Oct 30 21:00:33.175303 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:00:33.175275 5378 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-781058-openshift-default-55fb495b7-2c7gl" Oct 30 21:00:33.271225 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:00:33.271202 5378 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-2b425d-openshift-default-f547dbcbf-j99m9"] Oct 30 21:00:33.278730 ip-10-0-138-149 kubenswrapper[5378]: W1030 21:00:33.278693 5378 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podad099f6e_3035_4a6d_8fef_aea2e4bd1f3b.slice/crio-2ede2722fc5b8a46f879817b490ad7a6f2de00f3ae78584919d3579fd0b07bf8 WatchSource:0}: Error finding container 2ede2722fc5b8a46f879817b490ad7a6f2de00f3ae78584919d3579fd0b07bf8: Status 404 returned error can't find the container with id 2ede2722fc5b8a46f879817b490ad7a6f2de00f3ae78584919d3579fd0b07bf8 Oct 30 21:00:33.282327 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:00:33.282284 5378 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 21:00:33.282404 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:00:33.282360 5378 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 21:00:33.282404 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:00:33.282395 5378 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 21:00:33.563718 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:00:33.563687 5378 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-781058-openshift-default-55fb495b7-2c7gl"] Oct 30 21:00:33.564721 ip-10-0-138-149 kubenswrapper[5378]: W1030 21:00:33.564695 5378 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod507cd7ee_c139_489c_a520_292d8ebd91e4.slice/crio-2f027f9441458a04ae2b974849a14470830c98de45ace5fbdeaae974418a267c WatchSource:0}: Error finding container 2f027f9441458a04ae2b974849a14470830c98de45ace5fbdeaae974418a267c: Status 404 returned error can't find the container with id 2f027f9441458a04ae2b974849a14470830c98de45ace5fbdeaae974418a267c Oct 30 21:00:33.567813 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:00:33.567781 5378 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 21:00:33.567893 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:00:33.567856 5378 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 21:00:33.567893 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:00:33.567884 5378 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 21:00:33.744144 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:00:33.744101 5378 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b831b46a-433f-43a6-837f-dd353948593a" path="/var/lib/kubelet/pods/b831b46a-433f-43a6-837f-dd353948593a/volumes" Oct 30 21:00:34.163081 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:00:34.163044 5378 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-781058-openshift-default-55fb495b7-2c7gl" event={"ID":"507cd7ee-c139-489c-a520-292d8ebd91e4","Type":"ContainerStarted","Data":"6b92315664f5650233d2906cc50f29a6d230ad23d5fe304db0b0025ba3483c3e"} Oct 30 21:00:34.163261 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:00:34.163125 5378 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-781058-openshift-default-55fb495b7-2c7gl" event={"ID":"507cd7ee-c139-489c-a520-292d8ebd91e4","Type":"ContainerStarted","Data":"2f027f9441458a04ae2b974849a14470830c98de45ace5fbdeaae974418a267c"} Oct 30 21:00:34.164861 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:00:34.164833 5378 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-2b425d-openshift-default-f547dbcbf-j99m9" event={"ID":"ad099f6e-3035-4a6d-8fef-aea2e4bd1f3b","Type":"ContainerStarted","Data":"3119e4b6f8ac9712c0e43d657660719bcf26b0d85397424196cf757a1ad2f077"} Oct 30 21:00:34.164981 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:00:34.164875 5378 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-2b425d-openshift-default-f547dbcbf-j99m9" event={"ID":"ad099f6e-3035-4a6d-8fef-aea2e4bd1f3b","Type":"ContainerStarted","Data":"2ede2722fc5b8a46f879817b490ad7a6f2de00f3ae78584919d3579fd0b07bf8"} Oct 30 21:00:34.175805 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:00:34.175775 5378 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-781058-openshift-default-55fb495b7-2c7gl" Oct 30 21:00:34.177099 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:00:34.177073 5378 patch_prober.go:29] interesting pod/e2e-worker-w0-781058-openshift-default-55fb495b7-2c7gl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.92:15021/healthz/ready\": dial tcp 10.133.0.92:15021: connect: connection refused" start-of-body= Oct 30 21:00:34.177224 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:00:34.177122 5378 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-781058-openshift-default-55fb495b7-2c7gl" podUID="507cd7ee-c139-489c-a520-292d8ebd91e4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.92:15021/healthz/ready\": dial tcp 10.133.0.92:15021: connect: connection refused" Oct 30 21:00:34.289318 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:00:34.289246 5378 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-781058-openshift-default-55fb495b7-2c7gl" podStartSLOduration=2.28922558 podStartE2EDuration="2.28922558s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:00:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:00:34.283068545 +0000 UTC m=+1324.711182466" watchObservedRunningTime="2026-10-30 21:00:34.28922558 +0000 UTC m=+1324.717339501" Oct 30 21:00:34.403889 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:00:34.403816 5378 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w5-2b425d-openshift-default-f547dbcbf-j99m9" podStartSLOduration=2.403781951 podStartE2EDuration="2.403781951s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:00:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:00:34.403274746 +0000 UTC m=+1324.831388665" watchObservedRunningTime="2026-10-30 21:00:34.403781951 +0000 UTC m=+1324.831895871" Oct 30 21:00:34.814775 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:00:34.814743 5378 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w5-2b425d-openshift-default-f547dbcbf-j99m9" Oct 30 21:00:34.817014 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:00:34.816982 5378 patch_prober.go:29] interesting pod/e2e-worker-w5-2b425d-openshift-default-f547dbcbf-j99m9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.91:15021/healthz/ready\": dial tcp 10.133.0.91:15021: connect: connection refused" start-of-body= Oct 30 21:00:34.817119 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:00:34.817028 5378 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-2b425d-openshift-default-f547dbcbf-j99m9" podUID="ad099f6e-3035-4a6d-8fef-aea2e4bd1f3b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.91:15021/healthz/ready\": dial tcp 10.133.0.91:15021: connect: connection refused" Oct 30 21:00:35.194227 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:00:35.194189 5378 patch_prober.go:29] interesting pod/e2e-worker-w0-781058-openshift-default-55fb495b7-2c7gl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.92:15021/healthz/ready\": dial tcp 10.133.0.92:15021: connect: connection refused" start-of-body= Oct 30 21:00:35.194548 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:00:35.194504 5378 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-781058-openshift-default-55fb495b7-2c7gl" podUID="507cd7ee-c139-489c-a520-292d8ebd91e4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.92:15021/healthz/ready\": dial tcp 10.133.0.92:15021: connect: connection refused" Oct 30 21:00:35.815000 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:00:35.814968 5378 patch_prober.go:29] interesting pod/e2e-worker-w5-2b425d-openshift-default-f547dbcbf-j99m9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.91:15021/healthz/ready\": dial tcp 10.133.0.91:15021: connect: connection refused" start-of-body= Oct 30 21:00:35.815396 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:00:35.815016 5378 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-2b425d-openshift-default-f547dbcbf-j99m9" podUID="ad099f6e-3035-4a6d-8fef-aea2e4bd1f3b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.91:15021/healthz/ready\": dial tcp 10.133.0.91:15021: connect: connection refused" Oct 30 21:00:36.177016 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:00:36.176978 5378 patch_prober.go:29] interesting pod/e2e-worker-w0-781058-openshift-default-55fb495b7-2c7gl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.92:15021/healthz/ready\": dial tcp 10.133.0.92:15021: connect: connection refused" start-of-body= Oct 30 21:00:36.203590 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:00:36.202699 5378 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-781058-openshift-default-55fb495b7-2c7gl" podUID="507cd7ee-c139-489c-a520-292d8ebd91e4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.92:15021/healthz/ready\": dial tcp 10.133.0.92:15021: connect: connection refused" Oct 30 21:00:36.814858 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:00:36.814825 5378 patch_prober.go:29] interesting pod/e2e-worker-w5-2b425d-openshift-default-f547dbcbf-j99m9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.91:15021/healthz/ready\": dial tcp 10.133.0.91:15021: connect: connection refused" start-of-body= Oct 30 21:00:36.815022 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:00:36.814880 5378 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-2b425d-openshift-default-f547dbcbf-j99m9" podUID="ad099f6e-3035-4a6d-8fef-aea2e4bd1f3b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.91:15021/healthz/ready\": dial tcp 10.133.0.91:15021: connect: connection refused" Oct 30 21:00:37.197591 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:00:37.181976 5378 patch_prober.go:29] interesting pod/e2e-worker-w0-781058-openshift-default-55fb495b7-2c7gl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.92:15021/healthz/ready\": dial tcp 10.133.0.92:15021: connect: connection refused" start-of-body= Oct 30 21:00:37.197591 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:00:37.182041 5378 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-781058-openshift-default-55fb495b7-2c7gl" podUID="507cd7ee-c139-489c-a520-292d8ebd91e4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.92:15021/healthz/ready\": dial tcp 10.133.0.92:15021: connect: connection refused" Oct 30 21:00:37.814728 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:00:37.814692 5378 patch_prober.go:29] interesting pod/e2e-worker-w5-2b425d-openshift-default-f547dbcbf-j99m9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.91:15021/healthz/ready\": dial tcp 10.133.0.91:15021: connect: connection refused" start-of-body= Oct 30 21:00:37.815005 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:00:37.814972 5378 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-2b425d-openshift-default-f547dbcbf-j99m9" podUID="ad099f6e-3035-4a6d-8fef-aea2e4bd1f3b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.91:15021/healthz/ready\": dial tcp 10.133.0.91:15021: connect: connection refused" Oct 30 21:00:38.183589 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:00:38.181950 5378 patch_prober.go:29] interesting pod/e2e-worker-w0-781058-openshift-default-55fb495b7-2c7gl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.92:15021/healthz/ready\": dial tcp 10.133.0.92:15021: connect: connection refused" start-of-body= Oct 30 21:00:38.183589 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:00:38.182045 5378 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-781058-openshift-default-55fb495b7-2c7gl" podUID="507cd7ee-c139-489c-a520-292d8ebd91e4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.92:15021/healthz/ready\": dial tcp 10.133.0.92:15021: connect: connection refused" Oct 30 21:00:38.225989 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:00:38.225959 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:00:38.721257 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:00:38.721222 5378 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-r9g59"] Oct 30 21:00:38.814923 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:00:38.814879 5378 patch_prober.go:29] interesting pod/e2e-worker-w5-2b425d-openshift-default-f547dbcbf-j99m9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.91:15021/healthz/ready\": dial tcp 10.133.0.91:15021: connect: connection refused" start-of-body= Oct 30 21:00:38.815104 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:00:38.814935 5378 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-2b425d-openshift-default-f547dbcbf-j99m9" podUID="ad099f6e-3035-4a6d-8fef-aea2e4bd1f3b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.91:15021/healthz/ready\": dial tcp 10.133.0.91:15021: connect: connection refused" Oct 30 21:00:39.179036 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:00:39.179000 5378 patch_prober.go:29] interesting pod/e2e-worker-w0-781058-openshift-default-55fb495b7-2c7gl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.92:15021/healthz/ready\": dial tcp 10.133.0.92:15021: connect: connection refused" start-of-body= Oct 30 21:00:39.186048 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:00:39.185625 5378 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-781058-openshift-default-55fb495b7-2c7gl" podUID="507cd7ee-c139-489c-a520-292d8ebd91e4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.92:15021/healthz/ready\": dial tcp 10.133.0.92:15021: connect: connection refused" Oct 30 21:00:39.840591 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:00:39.819370 5378 patch_prober.go:29] interesting pod/e2e-worker-w5-2b425d-openshift-default-f547dbcbf-j99m9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.91:15021/healthz/ready\": dial tcp 10.133.0.91:15021: connect: connection refused" start-of-body= Oct 30 21:00:39.840591 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:00:39.819500 5378 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-2b425d-openshift-default-f547dbcbf-j99m9" podUID="ad099f6e-3035-4a6d-8fef-aea2e4bd1f3b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.91:15021/healthz/ready\": dial tcp 10.133.0.91:15021: connect: connection refused" Oct 30 21:00:40.181585 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:00:40.181034 5378 patch_prober.go:29] interesting pod/e2e-worker-w0-781058-openshift-default-55fb495b7-2c7gl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.92:15021/healthz/ready\": dial tcp 10.133.0.92:15021: connect: connection refused" start-of-body= Oct 30 21:00:40.181585 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:00:40.181112 5378 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-781058-openshift-default-55fb495b7-2c7gl" podUID="507cd7ee-c139-489c-a520-292d8ebd91e4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.92:15021/healthz/ready\": dial tcp 10.133.0.92:15021: connect: connection refused" Oct 30 21:00:40.822102 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:00:40.822064 5378 patch_prober.go:29] interesting pod/e2e-worker-w5-2b425d-openshift-default-f547dbcbf-j99m9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.91:15021/healthz/ready\": dial tcp 10.133.0.91:15021: connect: connection refused" start-of-body= Oct 30 21:00:40.822784 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:00:40.822749 5378 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-2b425d-openshift-default-f547dbcbf-j99m9" podUID="ad099f6e-3035-4a6d-8fef-aea2e4bd1f3b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.91:15021/healthz/ready\": dial tcp 10.133.0.91:15021: connect: connection refused" Oct 30 21:00:40.867125 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:00:40.866680 5378 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-f9f277/distinct-2-w1-f9f277-kserve-6c49cf5dcb-h4x74" podUID="9eb6ff54-b77b-453c-a166-988b98cf8f49" containerName="main" probeResult="failure" output="Get \"https://10.133.0.76:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 21:00:41.179427 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:00:41.178971 5378 patch_prober.go:29] interesting pod/e2e-worker-w0-781058-openshift-default-55fb495b7-2c7gl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.92:15021/healthz/ready\": dial tcp 10.133.0.92:15021: connect: connection refused" start-of-body= Oct 30 21:00:41.179427 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:00:41.179038 5378 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-781058-openshift-default-55fb495b7-2c7gl" podUID="507cd7ee-c139-489c-a520-292d8ebd91e4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.92:15021/healthz/ready\": dial tcp 10.133.0.92:15021: connect: connection refused" Oct 30 21:00:41.230602 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:00:41.230137 5378 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-8fa2a7/simulator-w6-8fa2a7-kserve-6cbf5bd894-lfpcq" podUID="f409f469-71fc-445b-bdc8-b8e84a4d1dc6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:00:41.242162 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:00:41.239757 5378 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-f9f277/simulator-w1-f9f277-kserve-5d9685f85c-2h78x" podUID="dc662e9d-914c-4dfb-a86f-0fb048f7b690" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": context deadline exceeded" Oct 30 21:00:41.256978 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:00:41.256642 5378 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-8fa2a7/premium-w6-8fa2a7-kserve-64bf4bf6c-dxrdr" podUID="484cba5f-b415-430a-b52d-50b1293be948" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:00:41.315606 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:00:41.314053 5378 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-d061d0/simulator-w2-d061d0-kserve-8484f9678b-lnszm" podUID="525fac02-b21c-4dd5-b553-72bcc36062c5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/ready\": context deadline exceeded" Oct 30 21:00:41.360737 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:00:41.360689 5378 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-781058/distinct-w0-781058-kserve-8bdf9bfcd-hzdfk" podUID="a9678fc2-1c56-4d45-9007-eee5049e0f48" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 21:00:41.785590 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:00:41.784678 5378 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-d061d0/distinct-w2-d061d0-kserve-679c7cd7c4-8czrv" podUID="2c139b9f-c7df-4675-b431-95d4136095e5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:00:41.815591 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:00:41.815057 5378 patch_prober.go:29] interesting pod/e2e-worker-w5-2b425d-openshift-default-f547dbcbf-j99m9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.91:15021/healthz/ready\": dial tcp 10.133.0.91:15021: connect: connection refused" start-of-body= Oct 30 21:00:41.815591 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:00:41.815134 5378 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-2b425d-openshift-default-f547dbcbf-j99m9" podUID="ad099f6e-3035-4a6d-8fef-aea2e4bd1f3b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.91:15021/healthz/ready\": dial tcp 10.133.0.91:15021: connect: connection refused" Oct 30 21:00:41.873590 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:00:41.872654 5378 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-f9f277/distinct-2-w1-f9f277-kserve-6c49cf5dcb-h4x74" podUID="9eb6ff54-b77b-453c-a166-988b98cf8f49" containerName="main" probeResult="failure" output="Get \"https://10.133.0.76:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:00:41.873590 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:00:41.873098 5378 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-f9f277/distinct-2-w1-f9f277-kserve-6c49cf5dcb-h4x74" Oct 30 21:00:41.985590 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:00:41.980983 5378 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-1fdaa7/distinct-w4-1fdaa7-kserve-86dc9774b9-6x2gf" podUID="2e162c95-0527-4cb3-85a3-795aae212d29" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:00:42.030589 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:00:42.027730 5378 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-781058/distinct-w0-781058-kserve-8bdf9bfcd-hzdfk" podUID="a9678fc2-1c56-4d45-9007-eee5049e0f48" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:00:42.215592 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:00:42.210716 5378 patch_prober.go:29] interesting pod/e2e-worker-w0-781058-openshift-default-55fb495b7-2c7gl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.92:15021/healthz/ready\": dial tcp 10.133.0.92:15021: connect: connection refused" start-of-body= Oct 30 21:00:42.215592 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:00:42.210783 5378 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-781058-openshift-default-55fb495b7-2c7gl" podUID="507cd7ee-c139-489c-a520-292d8ebd91e4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.92:15021/healthz/ready\": dial tcp 10.133.0.92:15021: connect: connection refused" Oct 30 21:00:42.388595 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:00:42.248789 5378 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-1fdaa7/simulator-w4-1fdaa7-kserve-9866d468c-fwmsk" podUID="09219832-010e-4ba8-bfb9-5a2884b358e1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": context deadline exceeded" Oct 30 21:00:42.388595 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:00:42.249225 5378 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-f9f277/simulator-w1-f9f277-kserve-5d9685f85c-2h78x" podUID="dc662e9d-914c-4dfb-a86f-0fb048f7b690" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": context deadline exceeded" Oct 30 21:00:42.388595 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:00:42.249606 5378 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-f9f277/simulator-w1-f9f277-kserve-5d9685f85c-2h78x" Oct 30 21:00:42.388595 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:00:42.250746 5378 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-8fa2a7/simulator-w6-8fa2a7-kserve-6cbf5bd894-lfpcq" podUID="f409f469-71fc-445b-bdc8-b8e84a4d1dc6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:00:42.388595 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:00:42.251033 5378 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-8fa2a7/simulator-w6-8fa2a7-kserve-6cbf5bd894-lfpcq" Oct 30 21:00:42.388595 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:00:42.257632 5378 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-8fa2a7/premium-w6-8fa2a7-kserve-64bf4bf6c-dxrdr" podUID="484cba5f-b415-430a-b52d-50b1293be948" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:00:42.388595 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:00:42.257978 5378 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-8fa2a7/premium-w6-8fa2a7-kserve-64bf4bf6c-dxrdr" Oct 30 21:00:42.428591 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:00:42.428000 5378 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-781058/distinct-w0-781058-kserve-8bdf9bfcd-hzdfk" podUID="a9678fc2-1c56-4d45-9007-eee5049e0f48" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/ready\": context deadline exceeded" Oct 30 21:00:42.467647 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:00:42.429901 5378 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-781058/distinct-w0-781058-kserve-8bdf9bfcd-hzdfk" Oct 30 21:00:42.491589 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:00:42.487839 5378 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-d061d0/simulator-w2-d061d0-kserve-8484f9678b-lnszm" podUID="525fac02-b21c-4dd5-b553-72bcc36062c5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/ready\": context deadline exceeded" Oct 30 21:00:42.491589 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:00:42.488866 5378 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-d061d0/simulator-w2-d061d0-kserve-8484f9678b-lnszm" Oct 30 21:00:42.661591 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:00:42.645940 5378 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-1fdaa7/premium-w4-1fdaa7-kserve-55dffbd5cd-gvpxq" podUID="ec282a4b-742f-4382-a296-335dcd2e9636" containerName="main" probeResult="failure" output="Get \"https://10.133.0.59:8000/ready\": context deadline exceeded" Oct 30 21:00:42.804966 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:00:42.804863 5378 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-d061d0/distinct-w2-d061d0-kserve-679c7cd7c4-8czrv" podUID="2c139b9f-c7df-4675-b431-95d4136095e5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/ready\": context deadline exceeded" Oct 30 21:00:42.805971 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:00:42.805885 5378 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-d061d0/distinct-w2-d061d0-kserve-679c7cd7c4-8czrv" Oct 30 21:00:42.883621 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:00:42.877823 5378 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-f9f277/distinct-2-w1-f9f277-kserve-6c49cf5dcb-h4x74" podUID="9eb6ff54-b77b-453c-a166-988b98cf8f49" containerName="main" probeResult="failure" output="Get \"https://10.133.0.76:8000/ready\": context deadline exceeded" Oct 30 21:00:42.981733 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:00:42.981675 5378 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-1fdaa7/distinct-w4-1fdaa7-kserve-86dc9774b9-6x2gf" podUID="2e162c95-0527-4cb3-85a3-795aae212d29" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:00:43.013590 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:00:42.982404 5378 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-1fdaa7/distinct-w4-1fdaa7-kserve-86dc9774b9-6x2gf" Oct 30 21:00:43.270598 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:00:43.265414 5378 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-8fa2a7/premium-w6-8fa2a7-kserve-64bf4bf6c-dxrdr" podUID="484cba5f-b415-430a-b52d-50b1293be948" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:00:43.270598 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:00:43.265746 5378 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-8fa2a7/simulator-w6-8fa2a7-kserve-6cbf5bd894-lfpcq" podUID="f409f469-71fc-445b-bdc8-b8e84a4d1dc6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:00:43.270598 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:00:43.265948 5378 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-1fdaa7/simulator-w4-1fdaa7-kserve-9866d468c-fwmsk" podUID="09219832-010e-4ba8-bfb9-5a2884b358e1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:00:43.270598 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:00:43.266110 5378 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-f9f277/simulator-w1-f9f277-kserve-5d9685f85c-2h78x" podUID="dc662e9d-914c-4dfb-a86f-0fb048f7b690" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": context deadline exceeded" Oct 30 21:00:43.270598 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:00:43.266321 5378 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-1fdaa7/simulator-w4-1fdaa7-kserve-9866d468c-fwmsk" Oct 30 21:00:43.428546 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:00:43.428502 5378 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-781058/distinct-w0-781058-kserve-8bdf9bfcd-hzdfk" podUID="a9678fc2-1c56-4d45-9007-eee5049e0f48" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:00:43.509421 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:00:43.508792 5378 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-d061d0/simulator-w2-d061d0-kserve-8484f9678b-lnszm" podUID="525fac02-b21c-4dd5-b553-72bcc36062c5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:00:43.619587 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:00:43.599003 5378 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-1fdaa7/simulator-w4-1fdaa7-kserve-9866d468c-fwmsk" Oct 30 21:00:43.900817 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:00:43.871923 5378 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-d061d0/distinct-w2-d061d0-kserve-679c7cd7c4-8czrv" podUID="2c139b9f-c7df-4675-b431-95d4136095e5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/ready\": context deadline exceeded" Oct 30 21:00:43.901635 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:00:43.901590 5378 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-1fdaa7/premium-w4-1fdaa7-kserve-55dffbd5cd-gvpxq" podUID="ec282a4b-742f-4382-a296-335dcd2e9636" containerName="main" probeResult="failure" output="Get \"https://10.133.0.59:8000/ready\": context deadline exceeded" Oct 30 21:00:43.928861 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:00:43.901979 5378 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-f9f277/distinct-2-w1-f9f277-kserve-6c49cf5dcb-h4x74" podUID="9eb6ff54-b77b-453c-a166-988b98cf8f49" containerName="main" probeResult="failure" output="Get \"https://10.133.0.76:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:00:43.989682 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:00:43.989052 5378 patch_prober.go:29] interesting pod/e2e-worker-w5-2b425d-openshift-default-f547dbcbf-j99m9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.91:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 21:00:43.989682 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:00:43.989113 5378 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-2b425d-openshift-default-f547dbcbf-j99m9" podUID="ad099f6e-3035-4a6d-8fef-aea2e4bd1f3b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.91:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 21:00:43.989682 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:00:43.989380 5378 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-1fdaa7/distinct-w4-1fdaa7-kserve-86dc9774b9-6x2gf" podUID="2e162c95-0527-4cb3-85a3-795aae212d29" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/ready\": context deadline exceeded" Oct 30 21:00:44.233652 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:00:44.229707 5378 patch_prober.go:29] interesting pod/e2e-worker-w2-d061d0-openshift-default-86b87f495-k5sd4 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.85:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 21:00:44.233652 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:00:44.229785 5378 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-d061d0-openshift-default-86b87f495-k5sd4" podUID="ddde4251-43ac-4def-b4d4-e1f9a2a7dc30" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.85:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 21:00:44.233652 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:00:44.230245 5378 patch_prober.go:29] interesting pod/e2e-worker-w0-781058-openshift-default-55fb495b7-2c7gl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.92:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 21:00:44.233652 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:00:44.230285 5378 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-781058-openshift-default-55fb495b7-2c7gl" podUID="507cd7ee-c139-489c-a520-292d8ebd91e4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.92:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 21:00:44.341842 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:00:44.330398 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:00:44.341842 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:00:44.330498 5378 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-f9f277/simulator-w1-f9f277-kserve-5d9685f85c-2h78x" podUID="dc662e9d-914c-4dfb-a86f-0fb048f7b690" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:00:44.341842 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:00:44.330793 5378 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-8fa2a7/simulator-w6-8fa2a7-kserve-6cbf5bd894-lfpcq" podUID="f409f469-71fc-445b-bdc8-b8e84a4d1dc6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:00:44.341842 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:00:44.331631 5378 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-8fa2a7/premium-w6-8fa2a7-kserve-64bf4bf6c-dxrdr" podUID="484cba5f-b415-430a-b52d-50b1293be948" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/ready\": context deadline exceeded" Oct 30 21:00:44.362590 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:00:44.359068 5378 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-1fdaa7/premium-w4-1fdaa7-kserve-55dffbd5cd-gvpxq" Oct 30 21:00:44.644595 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:00:44.412614 5378 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-1fdaa7/premium-w4-1fdaa7-kserve-55dffbd5cd-gvpxq" podUID="ec282a4b-742f-4382-a296-335dcd2e9636" containerName="main" probeResult="failure" output="Get \"https://10.133.0.59:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:00:44.644595 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:00:44.455962 5378 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-781058/distinct-w0-781058-kserve-8bdf9bfcd-hzdfk" podUID="a9678fc2-1c56-4d45-9007-eee5049e0f48" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/ready\": context deadline exceeded" Oct 30 21:00:44.644595 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:00:44.526740 5378 patch_prober.go:29] interesting pod/e2e-worker-w3-682999-openshift-default-54dcb797b5-r27vc container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.83:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 21:00:44.644595 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:00:44.638380 5378 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-682999-openshift-default-54dcb797b5-r27vc" podUID="feae394a-bbbf-4624-8057-cf046f9ffb93" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.83:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 21:00:44.644595 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:00:44.605658 5378 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-1fdaa7/simulator-w4-1fdaa7-kserve-9866d468c-fwmsk" podUID="09219832-010e-4ba8-bfb9-5a2884b358e1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:00:44.644595 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:00:44.605700 5378 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-1fdaa7/simulator-w4-1fdaa7-kserve-9866d468c-fwmsk" podUID="09219832-010e-4ba8-bfb9-5a2884b358e1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": context deadline exceeded" Oct 30 21:00:44.644595 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:00:44.526810 5378 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-d061d0/simulator-w2-d061d0-kserve-8484f9678b-lnszm" podUID="525fac02-b21c-4dd5-b553-72bcc36062c5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:00:44.683837 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:00:44.671124 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:00:44.871730 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:00:44.871669 5378 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-1fdaa7/distinct-w4-1fdaa7-kserve-86dc9774b9-6x2gf" podUID="2e162c95-0527-4cb3-85a3-795aae212d29" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/health\": context deadline exceeded" Oct 30 21:00:44.905344 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:00:44.904936 5378 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-f9f277/distinct-2-w1-f9f277-kserve-6c49cf5dcb-h4x74" podUID="9eb6ff54-b77b-453c-a166-988b98cf8f49" containerName="main" probeResult="failure" output="Get \"https://10.133.0.76:8000/ready\": context deadline exceeded" Oct 30 21:00:44.911783 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:00:44.911728 5378 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-d061d0/distinct-w2-d061d0-kserve-679c7cd7c4-8czrv" podUID="2c139b9f-c7df-4675-b431-95d4136095e5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:00:44.958405 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:00:44.919860 5378 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-1fdaa7/premium-w4-1fdaa7-kserve-55dffbd5cd-gvpxq" podUID="ec282a4b-742f-4382-a296-335dcd2e9636" containerName="main" probeResult="failure" output="Get \"https://10.133.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:00:44.979944 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:00:44.979914 5378 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-1fdaa7/distinct-w4-1fdaa7-kserve-86dc9774b9-6x2gf" Oct 30 21:00:45.010453 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:00:45.010425 5378 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-781058/distinct-w0-781058-kserve-8bdf9bfcd-hzdfk" Oct 30 21:00:45.015170 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:00:45.015140 5378 patch_prober.go:29] interesting pod/e2e-worker-w5-2b425d-openshift-default-f547dbcbf-j99m9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.91:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 21:00:45.015815 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:00:45.015779 5378 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-2b425d-openshift-default-f547dbcbf-j99m9" podUID="ad099f6e-3035-4a6d-8fef-aea2e4bd1f3b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.91:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 21:00:45.041245 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:00:45.041212 5378 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-1fdaa7/premium-w4-1fdaa7-kserve-55dffbd5cd-gvpxq" Oct 30 21:00:45.127589 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:00:45.103881 5378 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-781058-openshift-default-55fb495b7-2c7gl" Oct 30 21:00:45.147706 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:00:45.140083 5378 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-781058-openshift-default-55fb495b7-2c7gl" Oct 30 21:00:45.210678 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:00:45.147758 5378 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-d061d0/distinct-w2-d061d0-kserve-679c7cd7c4-8czrv" podUID="2c139b9f-c7df-4675-b431-95d4136095e5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:00:45.210678 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:00:45.168447 5378 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-781058-openshift-default-55fb495b7-2c7gl" Oct 30 21:00:45.319590 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:00:45.303223 5378 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-d061d0/distinct-w2-d061d0-kserve-679c7cd7c4-8czrv" Oct 30 21:00:45.362258 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:00:45.362217 5378 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-8fa2a7/simulator-w6-8fa2a7-kserve-6cbf5bd894-lfpcq" podUID="f409f469-71fc-445b-bdc8-b8e84a4d1dc6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:00:45.423990 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:00:45.362624 5378 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-f9f277/simulator-w1-f9f277-kserve-5d9685f85c-2h78x" Oct 30 21:00:45.438702 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:00:45.376546 5378 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-8fa2a7/premium-w6-8fa2a7-kserve-64bf4bf6c-dxrdr" podUID="484cba5f-b415-430a-b52d-50b1293be948" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/ready\": context deadline exceeded" Oct 30 21:00:45.504656 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:00:45.493956 5378 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-f9f277/distinct-2-w1-f9f277-kserve-6c49cf5dcb-h4x74" Oct 30 21:00:45.504656 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:00:45.501982 5378 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-d061d0/simulator-w2-d061d0-kserve-8484f9678b-lnszm" Oct 30 21:00:45.504656 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:00:45.502019 5378 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-8fa2a7/simulator-w6-8fa2a7-kserve-6cbf5bd894-lfpcq" Oct 30 21:00:45.523947 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:00:45.523918 5378 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-8fa2a7/premium-w6-8fa2a7-kserve-64bf4bf6c-dxrdr" Oct 30 21:00:46.057748 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:00:46.057705 5378 patch_prober.go:29] interesting pod/e2e-worker-w5-2b425d-openshift-default-f547dbcbf-j99m9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.91:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 21:00:46.058247 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:00:46.057768 5378 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-2b425d-openshift-default-f547dbcbf-j99m9" podUID="ad099f6e-3035-4a6d-8fef-aea2e4bd1f3b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.91:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 21:00:46.062647 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:00:46.062619 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:00:46.642372 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:00:46.642332 5378 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w5-2b425d-openshift-default-f547dbcbf-j99m9" Oct 30 21:00:46.642797 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:00:46.642773 5378 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w5-2b425d-openshift-default-f547dbcbf-j99m9" Oct 30 21:00:46.643969 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:00:46.643945 5378 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w5-2b425d-openshift-default-f547dbcbf-j99m9" Oct 30 21:00:48.463512 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:00:48.463480 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:00:48.609806 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:00:48.609771 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:00:48.837601 ip-10-0-138-149 kubenswrapper[5378]: E1030 21:00:48.837475 5378 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q6mx7" containerName="extractor" Oct 30 21:00:49.257298 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:00:49.257259 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:00:49.610085 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:00:49.610053 5378 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-23c672dc-5cccz"] Oct 30 21:00:49.614285 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:00:49.614268 5378 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-23c672dc-5cccz" Oct 30 21:00:49.671476 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:00:49.671451 5378 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 21:00:49.676556 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:00:49.676528 5378 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-23c672dc-5cccz"] Oct 30 21:00:49.701149 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:00:49.701123 5378 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d5c2205b-0665-490d-83c9-203f04e51d93-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-23c672dc-5cccz\" (UID: \"d5c2205b-0665-490d-83c9-203f04e51d93\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-23c672dc-5cccz" Oct 30 21:00:49.701415 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:00:49.701394 5378 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d5c2205b-0665-490d-83c9-203f04e51d93-maas-api-xks-ca\") pod \"maas-api-revoke-sub-23c672dc-5cccz\" (UID: \"d5c2205b-0665-490d-83c9-203f04e51d93\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-23c672dc-5cccz" Oct 30 21:00:49.701479 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:00:49.701423 5378 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5xtqt\" (UniqueName: \"kubernetes.io/projected/d5c2205b-0665-490d-83c9-203f04e51d93-kube-api-access-5xtqt\") pod \"maas-api-revoke-sub-23c672dc-5cccz\" (UID: \"d5c2205b-0665-490d-83c9-203f04e51d93\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-23c672dc-5cccz" Oct 30 21:00:49.721703 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:00:49.721682 5378 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-q2mgc\"" Oct 30 21:00:49.802654 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:00:49.802623 5378 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d5c2205b-0665-490d-83c9-203f04e51d93-maas-api-xks-ca\") pod \"maas-api-revoke-sub-23c672dc-5cccz\" (UID: \"d5c2205b-0665-490d-83c9-203f04e51d93\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-23c672dc-5cccz" Oct 30 21:00:49.802799 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:00:49.802661 5378 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5xtqt\" (UniqueName: \"kubernetes.io/projected/d5c2205b-0665-490d-83c9-203f04e51d93-kube-api-access-5xtqt\") pod \"maas-api-revoke-sub-23c672dc-5cccz\" (UID: \"d5c2205b-0665-490d-83c9-203f04e51d93\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-23c672dc-5cccz" Oct 30 21:00:49.802799 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:00:49.802703 5378 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d5c2205b-0665-490d-83c9-203f04e51d93-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-23c672dc-5cccz\" (UID: \"d5c2205b-0665-490d-83c9-203f04e51d93\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-23c672dc-5cccz" Oct 30 21:00:49.803502 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:00:49.803476 5378 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d5c2205b-0665-490d-83c9-203f04e51d93-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-23c672dc-5cccz\" (UID: \"d5c2205b-0665-490d-83c9-203f04e51d93\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-23c672dc-5cccz" Oct 30 21:00:49.804979 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:00:49.804961 5378 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d5c2205b-0665-490d-83c9-203f04e51d93-maas-api-xks-ca\") pod \"maas-api-revoke-sub-23c672dc-5cccz\" (UID: \"d5c2205b-0665-490d-83c9-203f04e51d93\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-23c672dc-5cccz" Oct 30 21:00:49.860131 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:00:49.860073 5378 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5xtqt\" (UniqueName: \"kubernetes.io/projected/d5c2205b-0665-490d-83c9-203f04e51d93-kube-api-access-5xtqt\") pod \"maas-api-revoke-sub-23c672dc-5cccz\" (UID: \"d5c2205b-0665-490d-83c9-203f04e51d93\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-23c672dc-5cccz" Oct 30 21:00:49.928401 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:00:49.928372 5378 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-23c672dc-5cccz" Oct 30 21:00:50.306616 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:00:50.306588 5378 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-23c672dc-5cccz"] Oct 30 21:00:50.307623 ip-10-0-138-149 kubenswrapper[5378]: W1030 21:00:50.307554 5378 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd5c2205b_0665_490d_83c9_203f04e51d93.slice/crio-737748f806c68d00a633a11de914f8ca4d0285f0ce59c0f69c54d37721085b40 WatchSource:0}: Error finding container 737748f806c68d00a633a11de914f8ca4d0285f0ce59c0f69c54d37721085b40: Status 404 returned error can't find the container with id 737748f806c68d00a633a11de914f8ca4d0285f0ce59c0f69c54d37721085b40 Oct 30 21:00:50.558342 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:00:50.558305 5378 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-23c672dc-5cccz" event={"ID":"d5c2205b-0665-490d-83c9-203f04e51d93","Type":"ContainerStarted","Data":"737748f806c68d00a633a11de914f8ca4d0285f0ce59c0f69c54d37721085b40"} Oct 30 21:00:50.971369 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:00:50.971331 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:00:51.567201 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:00:51.567122 5378 generic.go:356] "Generic (PLEG): container finished" podID="d5c2205b-0665-490d-83c9-203f04e51d93" containerID="3a7bffd1f19133cb813a22d614b54d28e11d8c3bf803b1a620bdfec52c6027dc" exitCode=0 Oct 30 21:00:51.567201 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:00:51.567163 5378 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-23c672dc-5cccz" event={"ID":"d5c2205b-0665-490d-83c9-203f04e51d93","Type":"ContainerDied","Data":"3a7bffd1f19133cb813a22d614b54d28e11d8c3bf803b1a620bdfec52c6027dc"} Oct 30 21:00:52.720376 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:00:52.720351 5378 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-23c672dc-5cccz" Oct 30 21:00:52.842136 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:00:52.842055 5378 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d5c2205b-0665-490d-83c9-203f04e51d93-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d5c2205b-0665-490d-83c9-203f04e51d93-maas-api-xks-ca\") pod \"d5c2205b-0665-490d-83c9-203f04e51d93\" (UID: \"d5c2205b-0665-490d-83c9-203f04e51d93\") " Oct 30 21:00:52.842305 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:00:52.842186 5378 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d5c2205b-0665-490d-83c9-203f04e51d93-kube-api-access-5xtqt\" (UniqueName: \"kubernetes.io/projected/d5c2205b-0665-490d-83c9-203f04e51d93-kube-api-access-5xtqt\") pod \"d5c2205b-0665-490d-83c9-203f04e51d93\" (UID: \"d5c2205b-0665-490d-83c9-203f04e51d93\") " Oct 30 21:00:52.842412 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:00:52.842397 5378 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d5c2205b-0665-490d-83c9-203f04e51d93-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d5c2205b-0665-490d-83c9-203f04e51d93-maas-api-openshift-ca\") pod \"d5c2205b-0665-490d-83c9-203f04e51d93\" (UID: \"d5c2205b-0665-490d-83c9-203f04e51d93\") " Oct 30 21:00:52.842768 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:00:52.842741 5378 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d5c2205b-0665-490d-83c9-203f04e51d93-maas-api-openshift-ca" pod "d5c2205b-0665-490d-83c9-203f04e51d93" (UID: "d5c2205b-0665-490d-83c9-203f04e51d93"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:00:52.842925 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:00:52.842910 5378 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d5c2205b-0665-490d-83c9-203f04e51d93-maas-api-openshift-ca\") on node \"ip-10-0-138-149.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:00:52.844142 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:00:52.844111 5378 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d5c2205b-0665-490d-83c9-203f04e51d93-maas-api-xks-ca" pod "d5c2205b-0665-490d-83c9-203f04e51d93" (UID: "d5c2205b-0665-490d-83c9-203f04e51d93"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:00:52.844237 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:00:52.844162 5378 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d5c2205b-0665-490d-83c9-203f04e51d93-kube-api-access-5xtqt" pod "d5c2205b-0665-490d-83c9-203f04e51d93" (UID: "d5c2205b-0665-490d-83c9-203f04e51d93"). InnerVolumeSpecName "kube-api-access-5xtqt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:00:52.943935 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:00:52.943905 5378 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d5c2205b-0665-490d-83c9-203f04e51d93-maas-api-xks-ca\") on node \"ip-10-0-138-149.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:00:52.943935 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:00:52.943932 5378 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5xtqt\" (UniqueName: \"kubernetes.io/projected/d5c2205b-0665-490d-83c9-203f04e51d93-kube-api-access-5xtqt\") on node \"ip-10-0-138-149.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:00:53.183430 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:00:53.183374 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:00:53.324379 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:00:53.324340 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:00:53.581594 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:00:53.581505 5378 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-23c672dc-5cccz" Oct 30 21:00:53.581594 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:00:53.581543 5378 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-23c672dc-5cccz" event={"ID":"d5c2205b-0665-490d-83c9-203f04e51d93","Type":"ContainerDied","Data":"737748f806c68d00a633a11de914f8ca4d0285f0ce59c0f69c54d37721085b40"} Oct 30 21:00:53.581818 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:00:53.581610 5378 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="737748f806c68d00a633a11de914f8ca4d0285f0ce59c0f69c54d37721085b40" Oct 30 21:00:53.636738 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:00:53.636699 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:00:54.282453 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:00:54.282417 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:00:54.527615 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:00:54.527543 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:00:55.430512 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:00:55.430475 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:00:55.565780 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:00:55.565748 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:00:55.753606 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:00:55.753511 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:00:55.986396 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:00:55.986363 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:00:56.721193 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:00:56.721116 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:00:56.929667 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:00:56.928632 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:00:58.624506 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:00:58.624472 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:00:58.766297 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:00:58.766260 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:00:58.976425 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:00:58.976342 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:00:59.260423 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:00:59.260344 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:01:00.417175 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:01:00.417140 5378 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-eac736c7-24cr2"] Oct 30 21:01:00.417784 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:01:00.417770 5378 cpu_manager.go:414] "containerMap: removing container" podUID="d5c2205b-0665-490d-83c9-203f04e51d93" containerName="revoke-keys" Oct 30 21:01:00.417846 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:01:00.417784 5378 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d5c2205b-0665-490d-83c9-203f04e51d93" containerName="revoke-keys" Oct 30 21:01:00.417903 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:01:00.417894 5378 memory_manager.go:365] "RemoveStaleState removing state" podUID="d5c2205b-0665-490d-83c9-203f04e51d93" containerName="revoke-keys" Oct 30 21:01:00.421715 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:01:00.421698 5378 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-eac736c7-24cr2" Oct 30 21:01:00.478069 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:01:00.478034 5378 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-eac736c7-24cr2"] Oct 30 21:01:00.478888 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:01:00.478852 5378 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m4j4t\" (UniqueName: \"kubernetes.io/projected/10b2d79a-d60b-4624-bf6d-6ac75c235de0-kube-api-access-m4j4t\") pod \"maas-api-revoke-sub-eac736c7-24cr2\" (UID: \"10b2d79a-d60b-4624-bf6d-6ac75c235de0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-eac736c7-24cr2" Oct 30 21:01:00.479041 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:01:00.479025 5378 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/10b2d79a-d60b-4624-bf6d-6ac75c235de0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-eac736c7-24cr2\" (UID: \"10b2d79a-d60b-4624-bf6d-6ac75c235de0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-eac736c7-24cr2" Oct 30 21:01:00.479115 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:01:00.479066 5378 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/10b2d79a-d60b-4624-bf6d-6ac75c235de0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-eac736c7-24cr2\" (UID: \"10b2d79a-d60b-4624-bf6d-6ac75c235de0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-eac736c7-24cr2" Oct 30 21:01:00.525000 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:01:00.524965 5378 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 21:01:00.525000 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:01:00.524995 5378 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-q2mgc\"" Oct 30 21:01:00.580235 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:01:00.580200 5378 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m4j4t\" (UniqueName: \"kubernetes.io/projected/10b2d79a-d60b-4624-bf6d-6ac75c235de0-kube-api-access-m4j4t\") pod \"maas-api-revoke-sub-eac736c7-24cr2\" (UID: \"10b2d79a-d60b-4624-bf6d-6ac75c235de0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-eac736c7-24cr2" Oct 30 21:01:00.580408 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:01:00.580326 5378 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/10b2d79a-d60b-4624-bf6d-6ac75c235de0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-eac736c7-24cr2\" (UID: \"10b2d79a-d60b-4624-bf6d-6ac75c235de0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-eac736c7-24cr2" Oct 30 21:01:00.580408 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:01:00.580360 5378 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/10b2d79a-d60b-4624-bf6d-6ac75c235de0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-eac736c7-24cr2\" (UID: \"10b2d79a-d60b-4624-bf6d-6ac75c235de0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-eac736c7-24cr2" Oct 30 21:01:00.581045 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:01:00.581027 5378 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/10b2d79a-d60b-4624-bf6d-6ac75c235de0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-eac736c7-24cr2\" (UID: \"10b2d79a-d60b-4624-bf6d-6ac75c235de0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-eac736c7-24cr2" Oct 30 21:01:00.582827 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:01:00.582799 5378 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/10b2d79a-d60b-4624-bf6d-6ac75c235de0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-eac736c7-24cr2\" (UID: \"10b2d79a-d60b-4624-bf6d-6ac75c235de0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-eac736c7-24cr2" Oct 30 21:01:00.637792 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:01:00.637760 5378 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m4j4t\" (UniqueName: \"kubernetes.io/projected/10b2d79a-d60b-4624-bf6d-6ac75c235de0-kube-api-access-m4j4t\") pod \"maas-api-revoke-sub-eac736c7-24cr2\" (UID: \"10b2d79a-d60b-4624-bf6d-6ac75c235de0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-eac736c7-24cr2" Oct 30 21:01:00.739053 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:01:00.738978 5378 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-eac736c7-24cr2" Oct 30 21:01:01.116283 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:01:01.116257 5378 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-eac736c7-24cr2"] Oct 30 21:01:01.117320 ip-10-0-138-149 kubenswrapper[5378]: W1030 21:01:01.117295 5378 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod10b2d79a_d60b_4624_bf6d_6ac75c235de0.slice/crio-c9411f89cca6219d21e8517bf7a084e7bfc34154dae45eca8d0bc774447146b0 WatchSource:0}: Error finding container c9411f89cca6219d21e8517bf7a084e7bfc34154dae45eca8d0bc774447146b0: Status 404 returned error can't find the container with id c9411f89cca6219d21e8517bf7a084e7bfc34154dae45eca8d0bc774447146b0 Oct 30 21:01:01.389237 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:01:01.389207 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:01:01.520276 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:01:01.520185 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:01:01.663928 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:01:01.663841 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:01:01.765381 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:01:01.765345 5378 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-eac736c7-24cr2" event={"ID":"10b2d79a-d60b-4624-bf6d-6ac75c235de0","Type":"ContainerStarted","Data":"f9515dce096602d9ad8d96b2c07ead6895651c7ccc1299d7116c10fac676013a"} Oct 30 21:01:01.765548 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:01:01.765403 5378 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-eac736c7-24cr2" event={"ID":"10b2d79a-d60b-4624-bf6d-6ac75c235de0","Type":"ContainerStarted","Data":"c9411f89cca6219d21e8517bf7a084e7bfc34154dae45eca8d0bc774447146b0"} Oct 30 21:01:01.792154 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:01:01.792122 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:01:01.989638 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:01:01.986117 5378 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-eac736c7-24cr2" podStartSLOduration=1.986095691 podStartE2EDuration="1.986095691s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:01:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:01:01.897929497 +0000 UTC m=+1352.326043421" watchObservedRunningTime="2026-10-30 21:01:01.986095691 +0000 UTC m=+1352.414209610" Oct 30 21:01:01.990387 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:01:01.990352 5378 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-f55142ef-5455c459ff-zsv57"] Oct 30 21:01:02.006766 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:01:02.006739 5378 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-f55142ef-5455c459ff-zsv57" Oct 30 21:01:02.062064 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:01:02.060942 5378 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-migrate-f55142ef\"" Oct 30 21:01:02.110954 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:01:02.110688 5378 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bdcf0b01-1a1b-4b24-a8fb-9bf730bd0636-plugins-config-volume\") pod \"payload-pre-processing-e2e-migrate-f55142ef-5455c459ff-zsv57\" (UID: \"bdcf0b01-1a1b-4b24-a8fb-9bf730bd0636\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-f55142ef-5455c459ff-zsv57" Oct 30 21:01:02.110954 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:01:02.110792 5378 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fs872\" (UniqueName: \"kubernetes.io/projected/bdcf0b01-1a1b-4b24-a8fb-9bf730bd0636-kube-api-access-fs872\") pod \"payload-pre-processing-e2e-migrate-f55142ef-5455c459ff-zsv57\" (UID: \"bdcf0b01-1a1b-4b24-a8fb-9bf730bd0636\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-f55142ef-5455c459ff-zsv57" Oct 30 21:01:02.140829 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:01:02.140773 5378 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-migrate-f55142ef-dockercfg-9vb9z\"" Oct 30 21:01:02.209238 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:01:02.191021 5378 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-f55142ef-5455c459ff-zsv57"] Oct 30 21:01:02.209238 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:01:02.191115 5378 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-f55142ef-77464b6464-z54fm"] Oct 30 21:01:02.308643 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:01:02.306113 5378 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bdcf0b01-1a1b-4b24-a8fb-9bf730bd0636-plugins-config-volume\") pod \"payload-pre-processing-e2e-migrate-f55142ef-5455c459ff-zsv57\" (UID: \"bdcf0b01-1a1b-4b24-a8fb-9bf730bd0636\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-f55142ef-5455c459ff-zsv57" Oct 30 21:01:02.308643 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:01:02.306247 5378 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fs872\" (UniqueName: \"kubernetes.io/projected/bdcf0b01-1a1b-4b24-a8fb-9bf730bd0636-kube-api-access-fs872\") pod \"payload-pre-processing-e2e-migrate-f55142ef-5455c459ff-zsv57\" (UID: \"bdcf0b01-1a1b-4b24-a8fb-9bf730bd0636\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-f55142ef-5455c459ff-zsv57" Oct 30 21:01:02.357635 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:01:02.353966 5378 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bdcf0b01-1a1b-4b24-a8fb-9bf730bd0636-plugins-config-volume\") pod \"payload-pre-processing-e2e-migrate-f55142ef-5455c459ff-zsv57\" (UID: \"bdcf0b01-1a1b-4b24-a8fb-9bf730bd0636\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-f55142ef-5455c459ff-zsv57" Oct 30 21:01:02.409369 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:01:02.409029 5378 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-f55142ef-77464b6464-z54fm"] Oct 30 21:01:02.409369 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:01:02.409159 5378 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-f55142ef-77464b6464-z54fm" Oct 30 21:01:02.413182 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:01:02.413149 5378 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fs872\" (UniqueName: \"kubernetes.io/projected/bdcf0b01-1a1b-4b24-a8fb-9bf730bd0636-kube-api-access-fs872\") pod \"payload-pre-processing-e2e-migrate-f55142ef-5455c459ff-zsv57\" (UID: \"bdcf0b01-1a1b-4b24-a8fb-9bf730bd0636\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-f55142ef-5455c459ff-zsv57" Oct 30 21:01:02.540387 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:01:02.540349 5378 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xp57m\" (UniqueName: \"kubernetes.io/projected/65353005-fa70-4a47-a789-27024c4c0a28-kube-api-access-xp57m\") pod \"payload-processing-e2e-migrate-f55142ef-77464b6464-z54fm\" (UID: \"65353005-fa70-4a47-a789-27024c4c0a28\") " pod="openshift-ingress/payload-processing-e2e-migrate-f55142ef-77464b6464-z54fm" Oct 30 21:01:02.540874 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:01:02.540525 5378 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/65353005-fa70-4a47-a789-27024c4c0a28-plugins-config-volume\") pod \"payload-processing-e2e-migrate-f55142ef-77464b6464-z54fm\" (UID: \"65353005-fa70-4a47-a789-27024c4c0a28\") " pod="openshift-ingress/payload-processing-e2e-migrate-f55142ef-77464b6464-z54fm" Oct 30 21:01:02.641786 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:01:02.641745 5378 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/65353005-fa70-4a47-a789-27024c4c0a28-plugins-config-volume\") pod \"payload-processing-e2e-migrate-f55142ef-77464b6464-z54fm\" (UID: \"65353005-fa70-4a47-a789-27024c4c0a28\") " pod="openshift-ingress/payload-processing-e2e-migrate-f55142ef-77464b6464-z54fm" Oct 30 21:01:02.641979 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:01:02.641960 5378 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xp57m\" (UniqueName: \"kubernetes.io/projected/65353005-fa70-4a47-a789-27024c4c0a28-kube-api-access-xp57m\") pod \"payload-processing-e2e-migrate-f55142ef-77464b6464-z54fm\" (UID: \"65353005-fa70-4a47-a789-27024c4c0a28\") " pod="openshift-ingress/payload-processing-e2e-migrate-f55142ef-77464b6464-z54fm" Oct 30 21:01:02.642432 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:01:02.642413 5378 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/65353005-fa70-4a47-a789-27024c4c0a28-plugins-config-volume\") pod \"payload-processing-e2e-migrate-f55142ef-77464b6464-z54fm\" (UID: \"65353005-fa70-4a47-a789-27024c4c0a28\") " pod="openshift-ingress/payload-processing-e2e-migrate-f55142ef-77464b6464-z54fm" Oct 30 21:01:02.644369 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:01:02.644337 5378 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-f55142ef-5455c459ff-zsv57" Oct 30 21:01:02.700975 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:01:02.700799 5378 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xp57m\" (UniqueName: \"kubernetes.io/projected/65353005-fa70-4a47-a789-27024c4c0a28-kube-api-access-xp57m\") pod \"payload-processing-e2e-migrate-f55142ef-77464b6464-z54fm\" (UID: \"65353005-fa70-4a47-a789-27024c4c0a28\") " pod="openshift-ingress/payload-processing-e2e-migrate-f55142ef-77464b6464-z54fm" Oct 30 21:01:02.735664 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:01:02.735635 5378 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-f55142ef-77464b6464-z54fm" Oct 30 21:01:02.776094 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:01:02.776061 5378 generic.go:356] "Generic (PLEG): container finished" podID="10b2d79a-d60b-4624-bf6d-6ac75c235de0" containerID="f9515dce096602d9ad8d96b2c07ead6895651c7ccc1299d7116c10fac676013a" exitCode=0 Oct 30 21:01:02.776259 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:01:02.776133 5378 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-eac736c7-24cr2" event={"ID":"10b2d79a-d60b-4624-bf6d-6ac75c235de0","Type":"ContainerDied","Data":"f9515dce096602d9ad8d96b2c07ead6895651c7ccc1299d7116c10fac676013a"} Oct 30 21:01:02.874166 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:01:02.874128 5378 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-1fdaa7-openshift-default-84586db7d5-zklmb"] Oct 30 21:01:02.881011 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:01:02.880982 5378 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-1fdaa7-openshift-default-84586db7d5-zklmb" Oct 30 21:01:02.890889 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:01:02.890866 5378 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-1fdaa7-openshift-default-84586db7d5-2svf5"] Oct 30 21:01:02.894856 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:01:02.894808 5378 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-1fdaa7-openshift-default-84586db7d5-2svf5" Oct 30 21:01:02.966345 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:01:02.966107 5378 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-1fdaa7-openshift-default-84586db7d5-zklmb"] Oct 30 21:01:03.021233 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:01:03.021208 5378 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-f55142ef-5455c459ff-zsv57"] Oct 30 21:01:03.021595 ip-10-0-138-149 kubenswrapper[5378]: W1030 21:01:03.021555 5378 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbdcf0b01_1a1b_4b24_a8fb_9bf730bd0636.slice/crio-36653640a3f0637958521e89f8c173ddb94fc8f137cb3e3e4ab31cf571855c49 WatchSource:0}: Error finding container 36653640a3f0637958521e89f8c173ddb94fc8f137cb3e3e4ab31cf571855c49: Status 404 returned error can't find the container with id 36653640a3f0637958521e89f8c173ddb94fc8f137cb3e3e4ab31cf571855c49 Oct 30 21:01:03.048636 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:01:03.048608 5378 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9jmjb\" (UniqueName: \"kubernetes.io/projected/f0685b8b-5dbc-4eb1-afcb-a390e5b464e7-kube-api-access-9jmjb\") pod \"e2e-worker-w4-1fdaa7-openshift-default-84586db7d5-zklmb\" (UID: \"f0685b8b-5dbc-4eb1-afcb-a390e5b464e7\") " pod="openshift-ingress/e2e-worker-w4-1fdaa7-openshift-default-84586db7d5-zklmb" Oct 30 21:01:03.048742 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:01:03.048660 5378 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/44227101-a97a-4bc9-a5b0-fd32cc961337-workload-certs\") pod \"e2e-worker-w4-1fdaa7-openshift-default-84586db7d5-2svf5\" (UID: \"44227101-a97a-4bc9-a5b0-fd32cc961337\") " pod="openshift-ingress/e2e-worker-w4-1fdaa7-openshift-default-84586db7d5-2svf5" Oct 30 21:01:03.048786 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:01:03.048757 5378 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c26vh\" (UniqueName: \"kubernetes.io/projected/44227101-a97a-4bc9-a5b0-fd32cc961337-kube-api-access-c26vh\") pod \"e2e-worker-w4-1fdaa7-openshift-default-84586db7d5-2svf5\" (UID: \"44227101-a97a-4bc9-a5b0-fd32cc961337\") " pod="openshift-ingress/e2e-worker-w4-1fdaa7-openshift-default-84586db7d5-2svf5" Oct 30 21:01:03.048829 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:01:03.048802 5378 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/44227101-a97a-4bc9-a5b0-fd32cc961337-istio-data\") pod \"e2e-worker-w4-1fdaa7-openshift-default-84586db7d5-2svf5\" (UID: \"44227101-a97a-4bc9-a5b0-fd32cc961337\") " pod="openshift-ingress/e2e-worker-w4-1fdaa7-openshift-default-84586db7d5-2svf5" Oct 30 21:01:03.048907 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:01:03.048883 5378 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f0685b8b-5dbc-4eb1-afcb-a390e5b464e7-workload-certs\") pod \"e2e-worker-w4-1fdaa7-openshift-default-84586db7d5-zklmb\" (UID: \"f0685b8b-5dbc-4eb1-afcb-a390e5b464e7\") " pod="openshift-ingress/e2e-worker-w4-1fdaa7-openshift-default-84586db7d5-zklmb" Oct 30 21:01:03.048957 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:01:03.048938 5378 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/44227101-a97a-4bc9-a5b0-fd32cc961337-istiod-ca-cert\") pod \"e2e-worker-w4-1fdaa7-openshift-default-84586db7d5-2svf5\" (UID: \"44227101-a97a-4bc9-a5b0-fd32cc961337\") " pod="openshift-ingress/e2e-worker-w4-1fdaa7-openshift-default-84586db7d5-2svf5" Oct 30 21:01:03.049077 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:01:03.049062 5378 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f0685b8b-5dbc-4eb1-afcb-a390e5b464e7-istio-token\") pod \"e2e-worker-w4-1fdaa7-openshift-default-84586db7d5-zklmb\" (UID: \"f0685b8b-5dbc-4eb1-afcb-a390e5b464e7\") " pod="openshift-ingress/e2e-worker-w4-1fdaa7-openshift-default-84586db7d5-zklmb" Oct 30 21:01:03.049132 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:01:03.049117 5378 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f0685b8b-5dbc-4eb1-afcb-a390e5b464e7-workload-socket\") pod \"e2e-worker-w4-1fdaa7-openshift-default-84586db7d5-zklmb\" (UID: \"f0685b8b-5dbc-4eb1-afcb-a390e5b464e7\") " pod="openshift-ingress/e2e-worker-w4-1fdaa7-openshift-default-84586db7d5-zklmb" Oct 30 21:01:03.049173 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:01:03.049146 5378 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f0685b8b-5dbc-4eb1-afcb-a390e5b464e7-istio-podinfo\") pod \"e2e-worker-w4-1fdaa7-openshift-default-84586db7d5-zklmb\" (UID: \"f0685b8b-5dbc-4eb1-afcb-a390e5b464e7\") " pod="openshift-ingress/e2e-worker-w4-1fdaa7-openshift-default-84586db7d5-zklmb" Oct 30 21:01:03.049216 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:01:03.049182 5378 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/44227101-a97a-4bc9-a5b0-fd32cc961337-istio-token\") pod \"e2e-worker-w4-1fdaa7-openshift-default-84586db7d5-2svf5\" (UID: \"44227101-a97a-4bc9-a5b0-fd32cc961337\") " pod="openshift-ingress/e2e-worker-w4-1fdaa7-openshift-default-84586db7d5-2svf5" Oct 30 21:01:03.049289 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:01:03.049273 5378 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/44227101-a97a-4bc9-a5b0-fd32cc961337-istio-podinfo\") pod \"e2e-worker-w4-1fdaa7-openshift-default-84586db7d5-2svf5\" (UID: \"44227101-a97a-4bc9-a5b0-fd32cc961337\") " pod="openshift-ingress/e2e-worker-w4-1fdaa7-openshift-default-84586db7d5-2svf5" Oct 30 21:01:03.049391 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:01:03.049339 5378 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f0685b8b-5dbc-4eb1-afcb-a390e5b464e7-credential-socket\") pod \"e2e-worker-w4-1fdaa7-openshift-default-84586db7d5-zklmb\" (UID: \"f0685b8b-5dbc-4eb1-afcb-a390e5b464e7\") " pod="openshift-ingress/e2e-worker-w4-1fdaa7-openshift-default-84586db7d5-zklmb" Oct 30 21:01:03.049391 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:01:03.049383 5378 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/44227101-a97a-4bc9-a5b0-fd32cc961337-istio-envoy\") pod \"e2e-worker-w4-1fdaa7-openshift-default-84586db7d5-2svf5\" (UID: \"44227101-a97a-4bc9-a5b0-fd32cc961337\") " pod="openshift-ingress/e2e-worker-w4-1fdaa7-openshift-default-84586db7d5-2svf5" Oct 30 21:01:03.049503 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:01:03.049454 5378 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f0685b8b-5dbc-4eb1-afcb-a390e5b464e7-istio-envoy\") pod \"e2e-worker-w4-1fdaa7-openshift-default-84586db7d5-zklmb\" (UID: \"f0685b8b-5dbc-4eb1-afcb-a390e5b464e7\") " pod="openshift-ingress/e2e-worker-w4-1fdaa7-openshift-default-84586db7d5-zklmb" Oct 30 21:01:03.049559 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:01:03.049502 5378 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f0685b8b-5dbc-4eb1-afcb-a390e5b464e7-istiod-ca-cert\") pod \"e2e-worker-w4-1fdaa7-openshift-default-84586db7d5-zklmb\" (UID: \"f0685b8b-5dbc-4eb1-afcb-a390e5b464e7\") " pod="openshift-ingress/e2e-worker-w4-1fdaa7-openshift-default-84586db7d5-zklmb" Oct 30 21:01:03.049559 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:01:03.049527 5378 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/44227101-a97a-4bc9-a5b0-fd32cc961337-credential-socket\") pod \"e2e-worker-w4-1fdaa7-openshift-default-84586db7d5-2svf5\" (UID: \"44227101-a97a-4bc9-a5b0-fd32cc961337\") " pod="openshift-ingress/e2e-worker-w4-1fdaa7-openshift-default-84586db7d5-2svf5" Oct 30 21:01:03.049730 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:01:03.049604 5378 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f0685b8b-5dbc-4eb1-afcb-a390e5b464e7-istio-data\") pod \"e2e-worker-w4-1fdaa7-openshift-default-84586db7d5-zklmb\" (UID: \"f0685b8b-5dbc-4eb1-afcb-a390e5b464e7\") " pod="openshift-ingress/e2e-worker-w4-1fdaa7-openshift-default-84586db7d5-zklmb" Oct 30 21:01:03.049730 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:01:03.049710 5378 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/44227101-a97a-4bc9-a5b0-fd32cc961337-workload-socket\") pod \"e2e-worker-w4-1fdaa7-openshift-default-84586db7d5-2svf5\" (UID: \"44227101-a97a-4bc9-a5b0-fd32cc961337\") " pod="openshift-ingress/e2e-worker-w4-1fdaa7-openshift-default-84586db7d5-2svf5" Oct 30 21:01:03.096190 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:01:03.096157 5378 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-1fdaa7-openshift-default-84586db7d5-2svf5"] Oct 30 21:01:03.121778 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:01:03.121745 5378 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-f55142ef-77464b6464-z54fm"] Oct 30 21:01:03.125933 ip-10-0-138-149 kubenswrapper[5378]: W1030 21:01:03.125894 5378 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod65353005_fa70_4a47_a789_27024c4c0a28.slice/crio-d71bdf330b9b68232164f4660227a4137105e1b22991e9d1dd7da0e810ea85c8 WatchSource:0}: Error finding container d71bdf330b9b68232164f4660227a4137105e1b22991e9d1dd7da0e810ea85c8: Status 404 returned error can't find the container with id d71bdf330b9b68232164f4660227a4137105e1b22991e9d1dd7da0e810ea85c8 Oct 30 21:01:03.150844 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:01:03.150784 5378 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/44227101-a97a-4bc9-a5b0-fd32cc961337-istio-token\") pod \"e2e-worker-w4-1fdaa7-openshift-default-84586db7d5-2svf5\" (UID: \"44227101-a97a-4bc9-a5b0-fd32cc961337\") " pod="openshift-ingress/e2e-worker-w4-1fdaa7-openshift-default-84586db7d5-2svf5" Oct 30 21:01:03.150844 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:01:03.150820 5378 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/44227101-a97a-4bc9-a5b0-fd32cc961337-istio-podinfo\") pod \"e2e-worker-w4-1fdaa7-openshift-default-84586db7d5-2svf5\" (UID: \"44227101-a97a-4bc9-a5b0-fd32cc961337\") " pod="openshift-ingress/e2e-worker-w4-1fdaa7-openshift-default-84586db7d5-2svf5" Oct 30 21:01:03.151016 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:01:03.150870 5378 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f0685b8b-5dbc-4eb1-afcb-a390e5b464e7-credential-socket\") pod \"e2e-worker-w4-1fdaa7-openshift-default-84586db7d5-zklmb\" (UID: \"f0685b8b-5dbc-4eb1-afcb-a390e5b464e7\") " pod="openshift-ingress/e2e-worker-w4-1fdaa7-openshift-default-84586db7d5-zklmb" Oct 30 21:01:03.151016 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:01:03.150899 5378 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/44227101-a97a-4bc9-a5b0-fd32cc961337-istio-envoy\") pod \"e2e-worker-w4-1fdaa7-openshift-default-84586db7d5-2svf5\" (UID: \"44227101-a97a-4bc9-a5b0-fd32cc961337\") " pod="openshift-ingress/e2e-worker-w4-1fdaa7-openshift-default-84586db7d5-2svf5" Oct 30 21:01:03.151016 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:01:03.150918 5378 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f0685b8b-5dbc-4eb1-afcb-a390e5b464e7-istio-envoy\") pod \"e2e-worker-w4-1fdaa7-openshift-default-84586db7d5-zklmb\" (UID: \"f0685b8b-5dbc-4eb1-afcb-a390e5b464e7\") " pod="openshift-ingress/e2e-worker-w4-1fdaa7-openshift-default-84586db7d5-zklmb" Oct 30 21:01:03.151016 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:01:03.150945 5378 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f0685b8b-5dbc-4eb1-afcb-a390e5b464e7-istiod-ca-cert\") pod \"e2e-worker-w4-1fdaa7-openshift-default-84586db7d5-zklmb\" (UID: \"f0685b8b-5dbc-4eb1-afcb-a390e5b464e7\") " pod="openshift-ingress/e2e-worker-w4-1fdaa7-openshift-default-84586db7d5-zklmb" Oct 30 21:01:03.151016 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:01:03.150975 5378 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/44227101-a97a-4bc9-a5b0-fd32cc961337-credential-socket\") pod \"e2e-worker-w4-1fdaa7-openshift-default-84586db7d5-2svf5\" (UID: \"44227101-a97a-4bc9-a5b0-fd32cc961337\") " pod="openshift-ingress/e2e-worker-w4-1fdaa7-openshift-default-84586db7d5-2svf5" Oct 30 21:01:03.151016 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:01:03.151005 5378 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f0685b8b-5dbc-4eb1-afcb-a390e5b464e7-istio-data\") pod \"e2e-worker-w4-1fdaa7-openshift-default-84586db7d5-zklmb\" (UID: \"f0685b8b-5dbc-4eb1-afcb-a390e5b464e7\") " pod="openshift-ingress/e2e-worker-w4-1fdaa7-openshift-default-84586db7d5-zklmb" Oct 30 21:01:03.151320 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:01:03.151088 5378 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/44227101-a97a-4bc9-a5b0-fd32cc961337-workload-socket\") pod \"e2e-worker-w4-1fdaa7-openshift-default-84586db7d5-2svf5\" (UID: \"44227101-a97a-4bc9-a5b0-fd32cc961337\") " pod="openshift-ingress/e2e-worker-w4-1fdaa7-openshift-default-84586db7d5-2svf5" Oct 30 21:01:03.151320 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:01:03.151256 5378 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9jmjb\" (UniqueName: \"kubernetes.io/projected/f0685b8b-5dbc-4eb1-afcb-a390e5b464e7-kube-api-access-9jmjb\") pod \"e2e-worker-w4-1fdaa7-openshift-default-84586db7d5-zklmb\" (UID: \"f0685b8b-5dbc-4eb1-afcb-a390e5b464e7\") " pod="openshift-ingress/e2e-worker-w4-1fdaa7-openshift-default-84586db7d5-zklmb" Oct 30 21:01:03.151320 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:01:03.151298 5378 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/44227101-a97a-4bc9-a5b0-fd32cc961337-workload-certs\") pod \"e2e-worker-w4-1fdaa7-openshift-default-84586db7d5-2svf5\" (UID: \"44227101-a97a-4bc9-a5b0-fd32cc961337\") " pod="openshift-ingress/e2e-worker-w4-1fdaa7-openshift-default-84586db7d5-2svf5" Oct 30 21:01:03.151515 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:01:03.151329 5378 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c26vh\" (UniqueName: \"kubernetes.io/projected/44227101-a97a-4bc9-a5b0-fd32cc961337-kube-api-access-c26vh\") pod \"e2e-worker-w4-1fdaa7-openshift-default-84586db7d5-2svf5\" (UID: \"44227101-a97a-4bc9-a5b0-fd32cc961337\") " pod="openshift-ingress/e2e-worker-w4-1fdaa7-openshift-default-84586db7d5-2svf5" Oct 30 21:01:03.151515 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:01:03.151366 5378 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/44227101-a97a-4bc9-a5b0-fd32cc961337-istio-data\") pod \"e2e-worker-w4-1fdaa7-openshift-default-84586db7d5-2svf5\" (UID: \"44227101-a97a-4bc9-a5b0-fd32cc961337\") " pod="openshift-ingress/e2e-worker-w4-1fdaa7-openshift-default-84586db7d5-2svf5" Oct 30 21:01:03.151515 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:01:03.151389 5378 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/44227101-a97a-4bc9-a5b0-fd32cc961337-credential-socket\") pod \"e2e-worker-w4-1fdaa7-openshift-default-84586db7d5-2svf5\" (UID: \"44227101-a97a-4bc9-a5b0-fd32cc961337\") " pod="openshift-ingress/e2e-worker-w4-1fdaa7-openshift-default-84586db7d5-2svf5" Oct 30 21:01:03.151515 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:01:03.151397 5378 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f0685b8b-5dbc-4eb1-afcb-a390e5b464e7-workload-certs\") pod \"e2e-worker-w4-1fdaa7-openshift-default-84586db7d5-zklmb\" (UID: \"f0685b8b-5dbc-4eb1-afcb-a390e5b464e7\") " pod="openshift-ingress/e2e-worker-w4-1fdaa7-openshift-default-84586db7d5-zklmb" Oct 30 21:01:03.151515 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:01:03.151400 5378 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f0685b8b-5dbc-4eb1-afcb-a390e5b464e7-credential-socket\") pod \"e2e-worker-w4-1fdaa7-openshift-default-84586db7d5-zklmb\" (UID: \"f0685b8b-5dbc-4eb1-afcb-a390e5b464e7\") " pod="openshift-ingress/e2e-worker-w4-1fdaa7-openshift-default-84586db7d5-zklmb" Oct 30 21:01:03.151515 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:01:03.151450 5378 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/44227101-a97a-4bc9-a5b0-fd32cc961337-istiod-ca-cert\") pod \"e2e-worker-w4-1fdaa7-openshift-default-84586db7d5-2svf5\" (UID: \"44227101-a97a-4bc9-a5b0-fd32cc961337\") " pod="openshift-ingress/e2e-worker-w4-1fdaa7-openshift-default-84586db7d5-2svf5" Oct 30 21:01:03.151515 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:01:03.151477 5378 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f0685b8b-5dbc-4eb1-afcb-a390e5b464e7-istio-data\") pod \"e2e-worker-w4-1fdaa7-openshift-default-84586db7d5-zklmb\" (UID: \"f0685b8b-5dbc-4eb1-afcb-a390e5b464e7\") " pod="openshift-ingress/e2e-worker-w4-1fdaa7-openshift-default-84586db7d5-zklmb" Oct 30 21:01:03.151902 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:01:03.151535 5378 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f0685b8b-5dbc-4eb1-afcb-a390e5b464e7-istio-token\") pod \"e2e-worker-w4-1fdaa7-openshift-default-84586db7d5-zklmb\" (UID: \"f0685b8b-5dbc-4eb1-afcb-a390e5b464e7\") " pod="openshift-ingress/e2e-worker-w4-1fdaa7-openshift-default-84586db7d5-zklmb" Oct 30 21:01:03.151902 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:01:03.151620 5378 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f0685b8b-5dbc-4eb1-afcb-a390e5b464e7-workload-socket\") pod \"e2e-worker-w4-1fdaa7-openshift-default-84586db7d5-zklmb\" (UID: \"f0685b8b-5dbc-4eb1-afcb-a390e5b464e7\") " pod="openshift-ingress/e2e-worker-w4-1fdaa7-openshift-default-84586db7d5-zklmb" Oct 30 21:01:03.151902 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:01:03.151649 5378 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f0685b8b-5dbc-4eb1-afcb-a390e5b464e7-istio-podinfo\") pod \"e2e-worker-w4-1fdaa7-openshift-default-84586db7d5-zklmb\" (UID: \"f0685b8b-5dbc-4eb1-afcb-a390e5b464e7\") " pod="openshift-ingress/e2e-worker-w4-1fdaa7-openshift-default-84586db7d5-zklmb" Oct 30 21:01:03.151902 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:01:03.151668 5378 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f0685b8b-5dbc-4eb1-afcb-a390e5b464e7-workload-certs\") pod \"e2e-worker-w4-1fdaa7-openshift-default-84586db7d5-zklmb\" (UID: \"f0685b8b-5dbc-4eb1-afcb-a390e5b464e7\") " pod="openshift-ingress/e2e-worker-w4-1fdaa7-openshift-default-84586db7d5-zklmb" Oct 30 21:01:03.151902 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:01:03.151793 5378 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f0685b8b-5dbc-4eb1-afcb-a390e5b464e7-istiod-ca-cert\") pod \"e2e-worker-w4-1fdaa7-openshift-default-84586db7d5-zklmb\" (UID: \"f0685b8b-5dbc-4eb1-afcb-a390e5b464e7\") " pod="openshift-ingress/e2e-worker-w4-1fdaa7-openshift-default-84586db7d5-zklmb" Oct 30 21:01:03.152201 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:01:03.152173 5378 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/44227101-a97a-4bc9-a5b0-fd32cc961337-workload-certs\") pod \"e2e-worker-w4-1fdaa7-openshift-default-84586db7d5-2svf5\" (UID: \"44227101-a97a-4bc9-a5b0-fd32cc961337\") " pod="openshift-ingress/e2e-worker-w4-1fdaa7-openshift-default-84586db7d5-2svf5" Oct 30 21:01:03.152311 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:01:03.152275 5378 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/44227101-a97a-4bc9-a5b0-fd32cc961337-workload-socket\") pod \"e2e-worker-w4-1fdaa7-openshift-default-84586db7d5-2svf5\" (UID: \"44227101-a97a-4bc9-a5b0-fd32cc961337\") " pod="openshift-ingress/e2e-worker-w4-1fdaa7-openshift-default-84586db7d5-2svf5" Oct 30 21:01:03.152372 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:01:03.152306 5378 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/44227101-a97a-4bc9-a5b0-fd32cc961337-istiod-ca-cert\") pod \"e2e-worker-w4-1fdaa7-openshift-default-84586db7d5-2svf5\" (UID: \"44227101-a97a-4bc9-a5b0-fd32cc961337\") " pod="openshift-ingress/e2e-worker-w4-1fdaa7-openshift-default-84586db7d5-2svf5" Oct 30 21:01:03.152618 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:01:03.152553 5378 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/44227101-a97a-4bc9-a5b0-fd32cc961337-istio-data\") pod \"e2e-worker-w4-1fdaa7-openshift-default-84586db7d5-2svf5\" (UID: \"44227101-a97a-4bc9-a5b0-fd32cc961337\") " pod="openshift-ingress/e2e-worker-w4-1fdaa7-openshift-default-84586db7d5-2svf5" Oct 30 21:01:03.152848 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:01:03.152822 5378 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f0685b8b-5dbc-4eb1-afcb-a390e5b464e7-workload-socket\") pod \"e2e-worker-w4-1fdaa7-openshift-default-84586db7d5-zklmb\" (UID: \"f0685b8b-5dbc-4eb1-afcb-a390e5b464e7\") " pod="openshift-ingress/e2e-worker-w4-1fdaa7-openshift-default-84586db7d5-zklmb" Oct 30 21:01:03.153638 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:01:03.153615 5378 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/44227101-a97a-4bc9-a5b0-fd32cc961337-istio-podinfo\") pod \"e2e-worker-w4-1fdaa7-openshift-default-84586db7d5-2svf5\" (UID: \"44227101-a97a-4bc9-a5b0-fd32cc961337\") " pod="openshift-ingress/e2e-worker-w4-1fdaa7-openshift-default-84586db7d5-2svf5" Oct 30 21:01:03.153773 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:01:03.153754 5378 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/44227101-a97a-4bc9-a5b0-fd32cc961337-istio-envoy\") pod \"e2e-worker-w4-1fdaa7-openshift-default-84586db7d5-2svf5\" (UID: \"44227101-a97a-4bc9-a5b0-fd32cc961337\") " pod="openshift-ingress/e2e-worker-w4-1fdaa7-openshift-default-84586db7d5-2svf5" Oct 30 21:01:03.153831 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:01:03.153787 5378 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f0685b8b-5dbc-4eb1-afcb-a390e5b464e7-istio-envoy\") pod \"e2e-worker-w4-1fdaa7-openshift-default-84586db7d5-zklmb\" (UID: \"f0685b8b-5dbc-4eb1-afcb-a390e5b464e7\") " pod="openshift-ingress/e2e-worker-w4-1fdaa7-openshift-default-84586db7d5-zklmb" Oct 30 21:01:03.154156 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:01:03.154139 5378 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f0685b8b-5dbc-4eb1-afcb-a390e5b464e7-istio-podinfo\") pod \"e2e-worker-w4-1fdaa7-openshift-default-84586db7d5-zklmb\" (UID: \"f0685b8b-5dbc-4eb1-afcb-a390e5b464e7\") " pod="openshift-ingress/e2e-worker-w4-1fdaa7-openshift-default-84586db7d5-zklmb" Oct 30 21:01:03.212458 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:01:03.212431 5378 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/44227101-a97a-4bc9-a5b0-fd32cc961337-istio-token\") pod \"e2e-worker-w4-1fdaa7-openshift-default-84586db7d5-2svf5\" (UID: \"44227101-a97a-4bc9-a5b0-fd32cc961337\") " pod="openshift-ingress/e2e-worker-w4-1fdaa7-openshift-default-84586db7d5-2svf5" Oct 30 21:01:03.212559 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:01:03.212546 5378 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9jmjb\" (UniqueName: \"kubernetes.io/projected/f0685b8b-5dbc-4eb1-afcb-a390e5b464e7-kube-api-access-9jmjb\") pod \"e2e-worker-w4-1fdaa7-openshift-default-84586db7d5-zklmb\" (UID: \"f0685b8b-5dbc-4eb1-afcb-a390e5b464e7\") " pod="openshift-ingress/e2e-worker-w4-1fdaa7-openshift-default-84586db7d5-zklmb" Oct 30 21:01:03.212896 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:01:03.212877 5378 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f0685b8b-5dbc-4eb1-afcb-a390e5b464e7-istio-token\") pod \"e2e-worker-w4-1fdaa7-openshift-default-84586db7d5-zklmb\" (UID: \"f0685b8b-5dbc-4eb1-afcb-a390e5b464e7\") " pod="openshift-ingress/e2e-worker-w4-1fdaa7-openshift-default-84586db7d5-zklmb" Oct 30 21:01:03.214160 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:01:03.214138 5378 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c26vh\" (UniqueName: \"kubernetes.io/projected/44227101-a97a-4bc9-a5b0-fd32cc961337-kube-api-access-c26vh\") pod \"e2e-worker-w4-1fdaa7-openshift-default-84586db7d5-2svf5\" (UID: \"44227101-a97a-4bc9-a5b0-fd32cc961337\") " pod="openshift-ingress/e2e-worker-w4-1fdaa7-openshift-default-84586db7d5-2svf5" Oct 30 21:01:03.381099 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:01:03.381069 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:01:03.498027 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:01:03.497945 5378 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-1fdaa7-openshift-default-84586db7d5-zklmb" Oct 30 21:01:03.508647 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:01:03.508620 5378 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-1fdaa7-openshift-default-84586db7d5-2svf5" Oct 30 21:01:03.582330 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:01:03.582285 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:01:03.785520 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:01:03.785424 5378 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-f55142ef-5455c459ff-zsv57" event={"ID":"bdcf0b01-1a1b-4b24-a8fb-9bf730bd0636","Type":"ContainerStarted","Data":"3c3318cd23cd82ceb105d3c720d8ad09a3b6e900685d9179d5c9233851e6acd6"} Oct 30 21:01:03.785520 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:01:03.785486 5378 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-f55142ef-5455c459ff-zsv57" event={"ID":"bdcf0b01-1a1b-4b24-a8fb-9bf730bd0636","Type":"ContainerStarted","Data":"36653640a3f0637958521e89f8c173ddb94fc8f137cb3e3e4ab31cf571855c49"} Oct 30 21:01:03.785520 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:01:03.785514 5378 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-migrate-f55142ef-5455c459ff-zsv57" Oct 30 21:01:03.787047 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:01:03.787019 5378 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-f55142ef-77464b6464-z54fm" event={"ID":"65353005-fa70-4a47-a789-27024c4c0a28","Type":"ContainerStarted","Data":"149b9383f1622944684f002bf041a4cc9c10d69ef1d0f7c4e8be491ae262b126"} Oct 30 21:01:03.787185 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:01:03.787068 5378 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-f55142ef-77464b6464-z54fm" event={"ID":"65353005-fa70-4a47-a789-27024c4c0a28","Type":"ContainerStarted","Data":"d71bdf330b9b68232164f4660227a4137105e1b22991e9d1dd7da0e810ea85c8"} Oct 30 21:01:03.787251 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:01:03.787235 5378 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-migrate-f55142ef-77464b6464-z54fm" Oct 30 21:01:03.912317 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:01:03.912281 5378 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-1fdaa7-openshift-default-84586db7d5-zklmb"] Oct 30 21:01:03.919998 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:01:03.919964 5378 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 21:01:03.920120 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:01:03.920043 5378 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 21:01:03.920120 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:01:03.920085 5378 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 21:01:03.939984 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:01:03.939930 5378 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-migrate-f55142ef-5455c459ff-zsv57" podStartSLOduration=2.939911983 podStartE2EDuration="2.939911983s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:01:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:01:03.933957714 +0000 UTC m=+1354.362071635" watchObservedRunningTime="2026-10-30 21:01:03.939911983 +0000 UTC m=+1354.368025903" Oct 30 21:01:03.944302 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:01:03.944273 5378 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-1fdaa7-openshift-default-84586db7d5-2svf5"] Oct 30 21:01:03.960458 ip-10-0-138-149 kubenswrapper[5378]: W1030 21:01:03.960433 5378 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod44227101_a97a_4bc9_a5b0_fd32cc961337.slice/crio-e563014471b67319a7023c03ec2a61ae98b2602ea3e6985cd000141e356ca4f8 WatchSource:0}: Error finding container e563014471b67319a7023c03ec2a61ae98b2602ea3e6985cd000141e356ca4f8: Status 404 returned error can't find the container with id e563014471b67319a7023c03ec2a61ae98b2602ea3e6985cd000141e356ca4f8 Oct 30 21:01:03.962924 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:01:03.962892 5378 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 21:01:03.963015 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:01:03.962957 5378 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 21:01:03.963015 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:01:03.962987 5378 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 21:01:04.078638 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:01:04.062378 5378 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-migrate-f55142ef-77464b6464-z54fm" podStartSLOduration=2.062356823 podStartE2EDuration="2.062356823s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:01:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:01:04.054906744 +0000 UTC m=+1354.483020665" watchObservedRunningTime="2026-10-30 21:01:04.062356823 +0000 UTC m=+1354.490470743" Oct 30 21:01:04.148603 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:01:04.144969 5378 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-eac736c7-24cr2" Oct 30 21:01:04.181510 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:01:04.180471 5378 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/10b2d79a-d60b-4624-bf6d-6ac75c235de0-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/10b2d79a-d60b-4624-bf6d-6ac75c235de0-maas-api-xks-ca\") pod \"10b2d79a-d60b-4624-bf6d-6ac75c235de0\" (UID: \"10b2d79a-d60b-4624-bf6d-6ac75c235de0\") " Oct 30 21:01:04.181510 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:01:04.180547 5378 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/10b2d79a-d60b-4624-bf6d-6ac75c235de0-kube-api-access-m4j4t\" (UniqueName: \"kubernetes.io/projected/10b2d79a-d60b-4624-bf6d-6ac75c235de0-kube-api-access-m4j4t\") pod \"10b2d79a-d60b-4624-bf6d-6ac75c235de0\" (UID: \"10b2d79a-d60b-4624-bf6d-6ac75c235de0\") " Oct 30 21:01:04.181510 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:01:04.180848 5378 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/10b2d79a-d60b-4624-bf6d-6ac75c235de0-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/10b2d79a-d60b-4624-bf6d-6ac75c235de0-maas-api-openshift-ca\") pod \"10b2d79a-d60b-4624-bf6d-6ac75c235de0\" (UID: \"10b2d79a-d60b-4624-bf6d-6ac75c235de0\") " Oct 30 21:01:04.181510 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:01:04.181211 5378 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/10b2d79a-d60b-4624-bf6d-6ac75c235de0-maas-api-openshift-ca" pod "10b2d79a-d60b-4624-bf6d-6ac75c235de0" (UID: "10b2d79a-d60b-4624-bf6d-6ac75c235de0"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:01:04.181968 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:01:04.181932 5378 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/10b2d79a-d60b-4624-bf6d-6ac75c235de0-maas-api-openshift-ca\") on node \"ip-10-0-138-149.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:01:04.183802 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:01:04.183768 5378 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/10b2d79a-d60b-4624-bf6d-6ac75c235de0-maas-api-xks-ca" pod "10b2d79a-d60b-4624-bf6d-6ac75c235de0" (UID: "10b2d79a-d60b-4624-bf6d-6ac75c235de0"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:01:04.185231 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:01:04.184804 5378 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/10b2d79a-d60b-4624-bf6d-6ac75c235de0-kube-api-access-m4j4t" pod "10b2d79a-d60b-4624-bf6d-6ac75c235de0" (UID: "10b2d79a-d60b-4624-bf6d-6ac75c235de0"). InnerVolumeSpecName "kube-api-access-m4j4t". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:01:04.300587 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:01:04.291963 5378 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/10b2d79a-d60b-4624-bf6d-6ac75c235de0-maas-api-xks-ca\") on node \"ip-10-0-138-149.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:01:04.300587 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:01:04.292005 5378 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m4j4t\" (UniqueName: \"kubernetes.io/projected/10b2d79a-d60b-4624-bf6d-6ac75c235de0-kube-api-access-m4j4t\") on node \"ip-10-0-138-149.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:01:04.813923 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:01:04.809456 5378 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-1fdaa7-openshift-default-84586db7d5-2svf5" event={"ID":"44227101-a97a-4bc9-a5b0-fd32cc961337","Type":"ContainerStarted","Data":"829c7b38cd116d38cc877ae98afebbc0baa8496ea21c8dc8ce8e1e43760ced3b"} Oct 30 21:01:04.813923 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:01:04.809525 5378 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-1fdaa7-openshift-default-84586db7d5-2svf5" event={"ID":"44227101-a97a-4bc9-a5b0-fd32cc961337","Type":"ContainerStarted","Data":"e563014471b67319a7023c03ec2a61ae98b2602ea3e6985cd000141e356ca4f8"} Oct 30 21:01:04.813923 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:01:04.812124 5378 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-1fdaa7-openshift-default-84586db7d5-zklmb" event={"ID":"f0685b8b-5dbc-4eb1-afcb-a390e5b464e7","Type":"ContainerStarted","Data":"71d242c0429a32ec01180957598143d6f2c6d3ab9c76ac0cf5f23a14dc188bcc"} Oct 30 21:01:04.813923 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:01:04.812189 5378 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-1fdaa7-openshift-default-84586db7d5-zklmb" event={"ID":"f0685b8b-5dbc-4eb1-afcb-a390e5b464e7","Type":"ContainerStarted","Data":"b5cde2dbb7d40b0676bb4a45ffa7742033470cdc6ede2057aafaa7584121480f"} Oct 30 21:01:04.814892 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:01:04.814859 5378 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-eac736c7-24cr2" event={"ID":"10b2d79a-d60b-4624-bf6d-6ac75c235de0","Type":"ContainerDied","Data":"c9411f89cca6219d21e8517bf7a084e7bfc34154dae45eca8d0bc774447146b0"} Oct 30 21:01:04.815074 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:01:04.815060 5378 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c9411f89cca6219d21e8517bf7a084e7bfc34154dae45eca8d0bc774447146b0" Oct 30 21:01:04.815228 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:01:04.815216 5378 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-eac736c7-24cr2" Oct 30 21:01:04.929127 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:01:04.929021 5378 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-1fdaa7-openshift-default-84586db7d5-2svf5" podStartSLOduration=2.929007801 podStartE2EDuration="2.929007801s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:01:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:01:04.927639825 +0000 UTC m=+1355.355753745" watchObservedRunningTime="2026-10-30 21:01:04.929007801 +0000 UTC m=+1355.357121719" Oct 30 21:01:05.132976 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:01:05.132920 5378 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-1fdaa7-openshift-default-84586db7d5-zklmb" podStartSLOduration=3.132905858 podStartE2EDuration="3.132905858s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:01:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:01:05.130228647 +0000 UTC m=+1355.558342567" watchObservedRunningTime="2026-10-30 21:01:05.132905858 +0000 UTC m=+1355.561019778" Oct 30 21:01:05.501676 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:01:05.500745 5378 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-1fdaa7-openshift-default-84586db7d5-zklmb" Oct 30 21:01:05.502659 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:01:05.502624 5378 patch_prober.go:29] interesting pod/e2e-worker-w4-1fdaa7-openshift-default-84586db7d5-zklmb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.97:15021/healthz/ready\": dial tcp 10.133.0.97:15021: connect: connection refused" start-of-body= Oct 30 21:01:05.502794 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:01:05.502682 5378 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-1fdaa7-openshift-default-84586db7d5-zklmb" podUID="f0685b8b-5dbc-4eb1-afcb-a390e5b464e7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.97:15021/healthz/ready\": dial tcp 10.133.0.97:15021: connect: connection refused" Oct 30 21:01:05.509761 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:01:05.509732 5378 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-1fdaa7-openshift-default-84586db7d5-2svf5" Oct 30 21:01:05.511014 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:01:05.510990 5378 patch_prober.go:29] interesting pod/e2e-worker-w4-1fdaa7-openshift-default-84586db7d5-2svf5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.98:15021/healthz/ready\": dial tcp 10.133.0.98:15021: connect: connection refused" start-of-body= Oct 30 21:01:05.511102 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:01:05.511078 5378 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-1fdaa7-openshift-default-84586db7d5-2svf5" podUID="44227101-a97a-4bc9-a5b0-fd32cc961337" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.98:15021/healthz/ready\": dial tcp 10.133.0.98:15021: connect: connection refused" Oct 30 21:01:06.246456 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:01:06.246421 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:01:06.397588 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:01:06.397177 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:01:06.503303 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:01:06.503173 5378 patch_prober.go:29] interesting pod/e2e-worker-w4-1fdaa7-openshift-default-84586db7d5-zklmb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.97:15021/healthz/ready\": dial tcp 10.133.0.97:15021: connect: connection refused" start-of-body= Oct 30 21:01:06.503303 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:01:06.503234 5378 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-1fdaa7-openshift-default-84586db7d5-zklmb" podUID="f0685b8b-5dbc-4eb1-afcb-a390e5b464e7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.97:15021/healthz/ready\": dial tcp 10.133.0.97:15021: connect: connection refused" Oct 30 21:01:06.509666 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:01:06.509631 5378 patch_prober.go:29] interesting pod/e2e-worker-w4-1fdaa7-openshift-default-84586db7d5-2svf5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.98:15021/healthz/ready\": dial tcp 10.133.0.98:15021: connect: connection refused" start-of-body= Oct 30 21:01:06.509811 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:01:06.509675 5378 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-1fdaa7-openshift-default-84586db7d5-2svf5" podUID="44227101-a97a-4bc9-a5b0-fd32cc961337" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.98:15021/healthz/ready\": dial tcp 10.133.0.98:15021: connect: connection refused" Oct 30 21:01:06.534053 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:01:06.534021 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:01:06.895588 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:01:06.894032 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:01:07.504737 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:01:07.504696 5378 patch_prober.go:29] interesting pod/e2e-worker-w4-1fdaa7-openshift-default-84586db7d5-zklmb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.97:15021/healthz/ready\": dial tcp 10.133.0.97:15021: connect: connection refused" start-of-body= Oct 30 21:01:07.505241 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:01:07.504757 5378 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-1fdaa7-openshift-default-84586db7d5-zklmb" podUID="f0685b8b-5dbc-4eb1-afcb-a390e5b464e7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.97:15021/healthz/ready\": dial tcp 10.133.0.97:15021: connect: connection refused" Oct 30 21:01:07.508910 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:01:07.508880 5378 patch_prober.go:29] interesting pod/e2e-worker-w4-1fdaa7-openshift-default-84586db7d5-2svf5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.98:15021/healthz/ready\": dial tcp 10.133.0.98:15021: connect: connection refused" start-of-body= Oct 30 21:01:07.509062 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:01:07.508929 5378 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-1fdaa7-openshift-default-84586db7d5-2svf5" podUID="44227101-a97a-4bc9-a5b0-fd32cc961337" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.98:15021/healthz/ready\": dial tcp 10.133.0.98:15021: connect: connection refused" Oct 30 21:01:07.599601 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:01:07.598558 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:01:07.744440 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:01:07.744411 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:01:08.508182 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:01:08.507969 5378 patch_prober.go:29] interesting pod/e2e-worker-w4-1fdaa7-openshift-default-84586db7d5-zklmb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.97:15021/healthz/ready\": dial tcp 10.133.0.97:15021: connect: connection refused" start-of-body= Oct 30 21:01:08.508688 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:01:08.508651 5378 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-1fdaa7-openshift-default-84586db7d5-zklmb" podUID="f0685b8b-5dbc-4eb1-afcb-a390e5b464e7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.97:15021/healthz/ready\": dial tcp 10.133.0.97:15021: connect: connection refused" Oct 30 21:01:08.509211 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:01:08.509184 5378 patch_prober.go:29] interesting pod/e2e-worker-w4-1fdaa7-openshift-default-84586db7d5-2svf5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.98:15021/healthz/ready\": dial tcp 10.133.0.98:15021: connect: connection refused" start-of-body= Oct 30 21:01:08.509371 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:01:08.509350 5378 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-1fdaa7-openshift-default-84586db7d5-2svf5" podUID="44227101-a97a-4bc9-a5b0-fd32cc961337" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.98:15021/healthz/ready\": dial tcp 10.133.0.98:15021: connect: connection refused" Oct 30 21:01:09.213966 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:01:09.213931 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:01:09.554658 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:01:09.554556 5378 patch_prober.go:29] interesting pod/e2e-worker-w4-1fdaa7-openshift-default-84586db7d5-zklmb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.97:15021/healthz/ready\": dial tcp 10.133.0.97:15021: connect: connection refused" start-of-body= Oct 30 21:01:09.568790 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:01:09.568752 5378 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-1fdaa7-openshift-default-84586db7d5-zklmb" podUID="f0685b8b-5dbc-4eb1-afcb-a390e5b464e7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.97:15021/healthz/ready\": dial tcp 10.133.0.97:15021: connect: connection refused" Oct 30 21:01:09.589829 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:01:09.562431 5378 patch_prober.go:29] interesting pod/e2e-worker-w4-1fdaa7-openshift-default-84586db7d5-2svf5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.98:15021/healthz/ready\": dial tcp 10.133.0.98:15021: connect: connection refused" start-of-body= Oct 30 21:01:09.590104 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:01:09.590067 5378 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-1fdaa7-openshift-default-84586db7d5-2svf5" podUID="44227101-a97a-4bc9-a5b0-fd32cc961337" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.98:15021/healthz/ready\": dial tcp 10.133.0.98:15021: connect: connection refused" Oct 30 21:01:09.590234 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:01:09.582941 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:01:09.721744 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:01:09.721711 5378 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-83851760-7c6fd5bb59-rdftz"] Oct 30 21:01:09.722369 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:01:09.722353 5378 cpu_manager.go:414] "containerMap: removing container" podUID="10b2d79a-d60b-4624-bf6d-6ac75c235de0" containerName="revoke-keys" Oct 30 21:01:09.722369 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:01:09.722369 5378 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="10b2d79a-d60b-4624-bf6d-6ac75c235de0" containerName="revoke-keys" Oct 30 21:01:09.722480 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:01:09.722467 5378 memory_manager.go:365] "RemoveStaleState removing state" podUID="10b2d79a-d60b-4624-bf6d-6ac75c235de0" containerName="revoke-keys" Oct 30 21:01:09.727421 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:01:09.727404 5378 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-83851760-7c6fd5bb59-rdftz" Oct 30 21:01:09.778589 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:01:09.777962 5378 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e9a8452e-59e0-47f0-8a96-098c18bc6424-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-83851760-7c6fd5bb59-rdftz\" (UID: \"e9a8452e-59e0-47f0-8a96-098c18bc6424\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-83851760-7c6fd5bb59-rdftz" Oct 30 21:01:09.778589 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:01:09.778038 5378 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9rpw4\" (UniqueName: \"kubernetes.io/projected/e9a8452e-59e0-47f0-8a96-098c18bc6424-kube-api-access-9rpw4\") pod \"payload-pre-processing-e2e-ait-83851760-7c6fd5bb59-rdftz\" (UID: \"e9a8452e-59e0-47f0-8a96-098c18bc6424\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-83851760-7c6fd5bb59-rdftz" Oct 30 21:01:09.812672 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:01:09.804718 5378 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-83851760-7c6fd5bb59-rdftz"] Oct 30 21:01:09.836956 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:01:09.836896 5378 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-83851760\"" Oct 30 21:01:09.837388 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:01:09.837360 5378 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-83851760-dockercfg-fxm2f\"" Oct 30 21:01:09.974588 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:01:09.969455 5378 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e9a8452e-59e0-47f0-8a96-098c18bc6424-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-83851760-7c6fd5bb59-rdftz\" (UID: \"e9a8452e-59e0-47f0-8a96-098c18bc6424\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-83851760-7c6fd5bb59-rdftz" Oct 30 21:01:09.974588 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:01:09.969540 5378 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9rpw4\" (UniqueName: \"kubernetes.io/projected/e9a8452e-59e0-47f0-8a96-098c18bc6424-kube-api-access-9rpw4\") pod \"payload-pre-processing-e2e-ait-83851760-7c6fd5bb59-rdftz\" (UID: \"e9a8452e-59e0-47f0-8a96-098c18bc6424\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-83851760-7c6fd5bb59-rdftz" Oct 30 21:01:09.974588 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:01:09.971135 5378 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e9a8452e-59e0-47f0-8a96-098c18bc6424-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-83851760-7c6fd5bb59-rdftz\" (UID: \"e9a8452e-59e0-47f0-8a96-098c18bc6424\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-83851760-7c6fd5bb59-rdftz" Oct 30 21:01:10.236605 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:01:10.225724 5378 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9rpw4\" (UniqueName: \"kubernetes.io/projected/e9a8452e-59e0-47f0-8a96-098c18bc6424-kube-api-access-9rpw4\") pod \"payload-pre-processing-e2e-ait-83851760-7c6fd5bb59-rdftz\" (UID: \"e9a8452e-59e0-47f0-8a96-098c18bc6424\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-83851760-7c6fd5bb59-rdftz" Oct 30 21:01:10.343546 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:01:10.343521 5378 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-83851760-7c6fd5bb59-rdftz" Oct 30 21:01:10.506724 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:01:10.506455 5378 patch_prober.go:29] interesting pod/e2e-worker-w4-1fdaa7-openshift-default-84586db7d5-zklmb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.97:15021/healthz/ready\": dial tcp 10.133.0.97:15021: connect: connection refused" start-of-body= Oct 30 21:01:10.506724 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:01:10.506500 5378 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-1fdaa7-openshift-default-84586db7d5-zklmb" podUID="f0685b8b-5dbc-4eb1-afcb-a390e5b464e7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.97:15021/healthz/ready\": dial tcp 10.133.0.97:15021: connect: connection refused" Oct 30 21:01:10.514560 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:01:10.513998 5378 patch_prober.go:29] interesting pod/e2e-worker-w4-1fdaa7-openshift-default-84586db7d5-2svf5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.98:15021/healthz/ready\": dial tcp 10.133.0.98:15021: connect: connection refused" start-of-body= Oct 30 21:01:10.514560 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:01:10.514050 5378 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-1fdaa7-openshift-default-84586db7d5-2svf5" podUID="44227101-a97a-4bc9-a5b0-fd32cc961337" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.98:15021/healthz/ready\": dial tcp 10.133.0.98:15021: connect: connection refused" Oct 30 21:01:10.840279 ip-10-0-138-149 kubenswrapper[5378]: W1030 21:01:10.840246 5378 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode9a8452e_59e0_47f0_8a96_098c18bc6424.slice/crio-a134085a3f8c876b66d49c8252116ee3403277485a6934911e8c8f76bce73a58 WatchSource:0}: Error finding container a134085a3f8c876b66d49c8252116ee3403277485a6934911e8c8f76bce73a58: Status 404 returned error can't find the container with id a134085a3f8c876b66d49c8252116ee3403277485a6934911e8c8f76bce73a58 Oct 30 21:01:10.840772 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:01:10.840515 5378 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-83851760-7c6fd5bb59-rdftz"] Oct 30 21:01:10.953630 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:01:10.953559 5378 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-83851760-7c6fd5bb59-rdftz" event={"ID":"e9a8452e-59e0-47f0-8a96-098c18bc6424","Type":"ContainerStarted","Data":"6b95122cfd411e427d9db40f6b261e96a0e02a1a6e26a1e6f10da156d47f8f14"} Oct 30 21:01:10.953763 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:01:10.953672 5378 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-83851760-7c6fd5bb59-rdftz" Oct 30 21:01:10.953763 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:01:10.953698 5378 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-83851760-7c6fd5bb59-rdftz" event={"ID":"e9a8452e-59e0-47f0-8a96-098c18bc6424","Type":"ContainerStarted","Data":"a134085a3f8c876b66d49c8252116ee3403277485a6934911e8c8f76bce73a58"} Oct 30 21:01:11.073525 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:01:11.073464 5378 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ait-83851760-7c6fd5bb59-rdftz" podStartSLOduration=2.073446658 podStartE2EDuration="2.073446658s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:01:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:01:11.07096762 +0000 UTC m=+1361.499081540" watchObservedRunningTime="2026-10-30 21:01:11.073446658 +0000 UTC m=+1361.501560579" Oct 30 21:01:11.499187 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:01:11.499150 5378 patch_prober.go:29] interesting pod/e2e-worker-w4-1fdaa7-openshift-default-84586db7d5-zklmb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.97:15021/healthz/ready\": dial tcp 10.133.0.97:15021: connect: connection refused" start-of-body= Oct 30 21:01:11.499376 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:01:11.499211 5378 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-1fdaa7-openshift-default-84586db7d5-zklmb" podUID="f0685b8b-5dbc-4eb1-afcb-a390e5b464e7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.97:15021/healthz/ready\": dial tcp 10.133.0.97:15021: connect: connection refused" Oct 30 21:01:11.516591 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:01:11.511381 5378 patch_prober.go:29] interesting pod/e2e-worker-w4-1fdaa7-openshift-default-84586db7d5-2svf5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.98:15021/healthz/ready\": dial tcp 10.133.0.98:15021: connect: connection refused" start-of-body= Oct 30 21:01:11.516591 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:01:11.511434 5378 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-1fdaa7-openshift-default-84586db7d5-2svf5" podUID="44227101-a97a-4bc9-a5b0-fd32cc961337" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.98:15021/healthz/ready\": dial tcp 10.133.0.98:15021: connect: connection refused" Oct 30 21:01:12.313611 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:01:12.313545 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:01:12.507694 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:01:12.507662 5378 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-1fdaa7-openshift-default-84586db7d5-zklmb" Oct 30 21:01:12.508202 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:01:12.508182 5378 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-1fdaa7-openshift-default-84586db7d5-zklmb" Oct 30 21:01:12.509179 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:01:12.509155 5378 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-1fdaa7-openshift-default-84586db7d5-zklmb" Oct 30 21:01:12.513917 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:01:12.513897 5378 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-1fdaa7-openshift-default-84586db7d5-2svf5" Oct 30 21:01:12.514167 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:01:12.514146 5378 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-1fdaa7-openshift-default-84586db7d5-2svf5" Oct 30 21:01:12.515165 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:01:12.515149 5378 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-1fdaa7-openshift-default-84586db7d5-2svf5" Oct 30 21:01:12.970950 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:01:12.970919 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:01:13.118360 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:01:13.118328 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:01:13.388034 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:01:13.387973 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:01:14.173669 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:01:14.173639 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:01:14.299189 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:01:14.299155 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:01:14.818985 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:01:14.818959 5378 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-migrate-f55142ef-77464b6464-z54fm" Oct 30 21:01:14.819454 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:01:14.819027 5378 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-migrate-f55142ef-5455c459ff-zsv57" Oct 30 21:01:16.304055 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:01:16.304008 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:01:17.604587 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:01:17.603683 5378 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-1fdaa7/simulator-w4-1fdaa7-kserve-9866d468c-fwmsk" podUID="09219832-010e-4ba8-bfb9-5a2884b358e1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:01:17.863613 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:01:17.863487 5378 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-f9f277/simulator-w1-f9f277-kserve-5d9685f85c-2h78x" podUID="dc662e9d-914c-4dfb-a86f-0fb048f7b690" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:01:17.912720 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:01:17.912654 5378 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-f9f277/distinct-2-w1-f9f277-kserve-6c49cf5dcb-h4x74" podUID="9eb6ff54-b77b-453c-a166-988b98cf8f49" containerName="main" probeResult="failure" output="Get \"https://10.133.0.76:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:01:17.937589 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:01:17.926146 5378 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-d061d0/distinct-w2-d061d0-kserve-679c7cd7c4-8czrv" podUID="2c139b9f-c7df-4675-b431-95d4136095e5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 21:01:18.067597 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:01:18.065504 5378 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-1fdaa7/premium-w4-1fdaa7-kserve-55dffbd5cd-gvpxq" podUID="ec282a4b-742f-4382-a296-335dcd2e9636" containerName="main" probeResult="failure" output="Get \"https://10.133.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:01:18.282645 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:01:18.281799 5378 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-8fa2a7/simulator-w6-8fa2a7-kserve-6cbf5bd894-lfpcq" podUID="f409f469-71fc-445b-bdc8-b8e84a4d1dc6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": context deadline exceeded" Oct 30 21:01:18.455589 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:01:18.450690 5378 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-781058/distinct-w0-781058-kserve-8bdf9bfcd-hzdfk" podUID="a9678fc2-1c56-4d45-9007-eee5049e0f48" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:01:18.455793 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:01:18.455674 5378 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-8fa2a7/premium-w6-8fa2a7-kserve-64bf4bf6c-dxrdr" podUID="484cba5f-b415-430a-b52d-50b1293be948" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 21:01:18.514242 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:01:18.514196 5378 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-d061d0/simulator-w2-d061d0-kserve-8484f9678b-lnszm" podUID="525fac02-b21c-4dd5-b553-72bcc36062c5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:01:18.583774 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:01:18.583684 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:01:18.605833 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:01:18.605783 5378 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-1fdaa7/simulator-w4-1fdaa7-kserve-9866d468c-fwmsk" podUID="09219832-010e-4ba8-bfb9-5a2884b358e1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 21:01:18.606628 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:01:18.606604 5378 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-1fdaa7/simulator-w4-1fdaa7-kserve-9866d468c-fwmsk" Oct 30 21:01:18.920087 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:01:18.914140 5378 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-f9f277/distinct-2-w1-f9f277-kserve-6c49cf5dcb-h4x74" podUID="9eb6ff54-b77b-453c-a166-988b98cf8f49" containerName="main" probeResult="failure" output="Get \"https://10.133.0.76:8000/ready\": context deadline exceeded" Oct 30 21:01:19.052598 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:01:19.018127 5378 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-f9f277/distinct-2-w1-f9f277-kserve-6c49cf5dcb-h4x74" Oct 30 21:01:19.275641 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:01:19.271392 5378 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-1fdaa7/premium-w4-1fdaa7-kserve-55dffbd5cd-gvpxq" podUID="ec282a4b-742f-4382-a296-335dcd2e9636" containerName="main" probeResult="failure" output="Get \"https://10.133.0.59:8000/ready\": context deadline exceeded" Oct 30 21:01:19.284680 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:01:19.278104 5378 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-1fdaa7/premium-w4-1fdaa7-kserve-55dffbd5cd-gvpxq" Oct 30 21:01:19.349039 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:01:19.322676 5378 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-d061d0/distinct-w2-d061d0-kserve-679c7cd7c4-8czrv" podUID="2c139b9f-c7df-4675-b431-95d4136095e5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 21:01:19.404602 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:01:19.351546 5378 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-d061d0/distinct-w2-d061d0-kserve-679c7cd7c4-8czrv" Oct 30 21:01:19.536830 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:01:19.536717 5378 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-d061d0/simulator-w2-d061d0-kserve-8484f9678b-lnszm" podUID="525fac02-b21c-4dd5-b553-72bcc36062c5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:01:19.538840 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:01:19.538814 5378 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-d061d0/simulator-w2-d061d0-kserve-8484f9678b-lnszm" Oct 30 21:01:19.558614 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:01:19.555799 5378 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-d061d0/distinct-w2-d061d0-kserve-679c7cd7c4-8czrv" Oct 30 21:01:19.558614 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:01:19.555926 5378 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-f9f277/distinct-2-w1-f9f277-kserve-6c49cf5dcb-h4x74" Oct 30 21:01:19.609928 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:01:19.609892 5378 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-1fdaa7/simulator-w4-1fdaa7-kserve-9866d468c-fwmsk" podUID="09219832-010e-4ba8-bfb9-5a2884b358e1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": context deadline exceeded" Oct 30 21:01:19.661957 ip-10-0-138-149 kubenswrapper[5378]: E1030 21:01:19.623777 5378 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q6mx7" containerName="extractor" Oct 30 21:01:19.874693 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:01:19.874657 5378 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-d061d0/simulator-w2-d061d0-kserve-8484f9678b-lnszm" Oct 30 21:01:19.874892 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:01:19.874720 5378 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-1fdaa7/premium-w4-1fdaa7-kserve-55dffbd5cd-gvpxq" Oct 30 21:01:19.892600 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:01:19.881990 5378 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-1fdaa7/simulator-w4-1fdaa7-kserve-9866d468c-fwmsk" Oct 30 21:01:20.072680 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:01:20.072652 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:01:20.751688 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:01:20.751547 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:01:20.903879 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:01:20.903845 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:01:21.055553 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:01:21.055448 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:01:21.456492 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:01:21.456458 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:01:21.968392 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:01:21.968357 5378 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-83851760-7c6fd5bb59-rdftz" Oct 30 21:01:22.318309 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:01:22.318207 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:01:23.025104 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:01:23.025063 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:01:23.943455 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:01:23.943415 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:01:24.089833 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:01:24.089787 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:01:26.435255 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:01:26.435217 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:01:26.849644 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:01:26.849606 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:01:27.043371 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:01:27.043328 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:01:27.209039 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:01:27.208955 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:01:27.791697 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:01:27.791662 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:01:27.938927 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:01:27.938888 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:01:28.095165 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:01:28.095133 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:01:29.114836 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:01:29.114801 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:01:29.993815 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:01:29.993788 5378 scope.go:122] "RemoveContainer" containerID="378176a6c61d5325db1dab1d835d3f89e4ef00780dcd3f1227615023f9a6326e" Oct 30 21:01:30.003981 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:01:30.003961 5378 scope.go:122] "RemoveContainer" containerID="6a666916611bf7e2a5928fd7680f91ad2a0830d8df9a435ed444316e1cb957ce" Oct 30 21:01:30.013065 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:01:30.013043 5378 scope.go:122] "RemoveContainer" containerID="e338ba4b17595477700ccc4e9ab9f44cf8bb0b0caeb9770fbb0916196d337360" Oct 30 21:01:30.025607 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:01:30.025559 5378 scope.go:122] "RemoveContainer" containerID="e6eccc026db7218dfcdb1eeb5a21b06fad3ff0ca690afc354045563983f6143a" Oct 30 21:01:30.034239 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:01:30.034221 5378 scope.go:122] "RemoveContainer" containerID="e0936b412ac0ad9f042a7be6dcdc2d2683f9173b65d7c870419ff04e7ca54bda" Oct 30 21:01:30.094863 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:01:30.094835 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:01:30.284165 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:01:30.284071 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:01:30.808588 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:01:30.806154 5378 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-bc88b34e-55b79996d-kfqbm"] Oct 30 21:01:30.821583 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:01:30.820952 5378 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-bc88b34e-55b79996d-kfqbm" Oct 30 21:01:30.847244 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:01:30.845837 5378 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-bc88b34e-55b79996d-kfqbm"] Oct 30 21:01:30.927338 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:01:30.927312 5378 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-bc88b34e\"" Oct 30 21:01:30.927338 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:01:30.927320 5378 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-bc88b34e-dockercfg-r5mtf\"" Oct 30 21:01:30.945512 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:01:30.945470 5378 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/727b7af5-e485-42f3-8bb5-a6b5136a3e99-plugins-config-volume\") pod \"payload-processing-e2e-ait-bc88b34e-55b79996d-kfqbm\" (UID: \"727b7af5-e485-42f3-8bb5-a6b5136a3e99\") " pod="openshift-ingress/payload-processing-e2e-ait-bc88b34e-55b79996d-kfqbm" Oct 30 21:01:30.945734 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:01:30.945694 5378 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lcjtn\" (UniqueName: \"kubernetes.io/projected/727b7af5-e485-42f3-8bb5-a6b5136a3e99-kube-api-access-lcjtn\") pod \"payload-processing-e2e-ait-bc88b34e-55b79996d-kfqbm\" (UID: \"727b7af5-e485-42f3-8bb5-a6b5136a3e99\") " pod="openshift-ingress/payload-processing-e2e-ait-bc88b34e-55b79996d-kfqbm" Oct 30 21:01:31.047004 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:01:31.046970 5378 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/727b7af5-e485-42f3-8bb5-a6b5136a3e99-plugins-config-volume\") pod \"payload-processing-e2e-ait-bc88b34e-55b79996d-kfqbm\" (UID: \"727b7af5-e485-42f3-8bb5-a6b5136a3e99\") " pod="openshift-ingress/payload-processing-e2e-ait-bc88b34e-55b79996d-kfqbm" Oct 30 21:01:31.047152 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:01:31.047067 5378 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lcjtn\" (UniqueName: \"kubernetes.io/projected/727b7af5-e485-42f3-8bb5-a6b5136a3e99-kube-api-access-lcjtn\") pod \"payload-processing-e2e-ait-bc88b34e-55b79996d-kfqbm\" (UID: \"727b7af5-e485-42f3-8bb5-a6b5136a3e99\") " pod="openshift-ingress/payload-processing-e2e-ait-bc88b34e-55b79996d-kfqbm" Oct 30 21:01:31.047724 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:01:31.047707 5378 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/727b7af5-e485-42f3-8bb5-a6b5136a3e99-plugins-config-volume\") pod \"payload-processing-e2e-ait-bc88b34e-55b79996d-kfqbm\" (UID: \"727b7af5-e485-42f3-8bb5-a6b5136a3e99\") " pod="openshift-ingress/payload-processing-e2e-ait-bc88b34e-55b79996d-kfqbm" Oct 30 21:01:31.105446 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:01:31.105423 5378 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lcjtn\" (UniqueName: \"kubernetes.io/projected/727b7af5-e485-42f3-8bb5-a6b5136a3e99-kube-api-access-lcjtn\") pod \"payload-processing-e2e-ait-bc88b34e-55b79996d-kfqbm\" (UID: \"727b7af5-e485-42f3-8bb5-a6b5136a3e99\") " pod="openshift-ingress/payload-processing-e2e-ait-bc88b34e-55b79996d-kfqbm" Oct 30 21:01:31.224177 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:01:31.224143 5378 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-bc88b34e-55b79996d-kfqbm" Oct 30 21:01:31.611204 ip-10-0-138-149 kubenswrapper[5378]: W1030 21:01:31.611168 5378 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod727b7af5_e485_42f3_8bb5_a6b5136a3e99.slice/crio-c2d2ef2b8e418a7d4d6581e8a1344d142c2344ae79e4d013e089e33ff529ea71 WatchSource:0}: Error finding container c2d2ef2b8e418a7d4d6581e8a1344d142c2344ae79e4d013e089e33ff529ea71: Status 404 returned error can't find the container with id c2d2ef2b8e418a7d4d6581e8a1344d142c2344ae79e4d013e089e33ff529ea71 Oct 30 21:01:31.611853 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:01:31.611823 5378 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-bc88b34e-55b79996d-kfqbm"] Oct 30 21:01:32.576207 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:01:32.576168 5378 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-bc88b34e-55b79996d-kfqbm" event={"ID":"727b7af5-e485-42f3-8bb5-a6b5136a3e99","Type":"ContainerStarted","Data":"fade0178ccccfd995880e856a7839a0fd8e82f8252f680989f244ae4930306d6"} Oct 30 21:01:32.576414 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:01:32.576225 5378 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-bc88b34e-55b79996d-kfqbm" event={"ID":"727b7af5-e485-42f3-8bb5-a6b5136a3e99","Type":"ContainerStarted","Data":"c2d2ef2b8e418a7d4d6581e8a1344d142c2344ae79e4d013e089e33ff529ea71"} Oct 30 21:01:32.576414 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:01:32.576257 5378 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ait-bc88b34e-55b79996d-kfqbm" Oct 30 21:01:32.692986 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:01:32.692927 5378 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ait-bc88b34e-55b79996d-kfqbm" podStartSLOduration=2.692910119 podStartE2EDuration="2.692910119s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:01:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:01:32.691953516 +0000 UTC m=+1383.120067436" watchObservedRunningTime="2026-10-30 21:01:32.692910119 +0000 UTC m=+1383.121024042" Oct 30 21:01:34.702555 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:01:34.702505 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:01:39.809694 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:01:39.809657 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:01:43.095674 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:01:43.095639 5378 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-769df7846c-swlpw"] Oct 30 21:01:43.101328 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:01:43.101307 5378 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-swlpw" Oct 30 21:01:43.234699 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:01:43.234660 5378 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c5b6e9d2-ca83-4956-819e-d7ec7997acd8-workload-certs\") pod \"maas-default-gateway-openshift-default-769df7846c-swlpw\" (UID: \"c5b6e9d2-ca83-4956-819e-d7ec7997acd8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-swlpw" Oct 30 21:01:43.234867 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:01:43.234708 5378 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c5b6e9d2-ca83-4956-819e-d7ec7997acd8-istio-podinfo\") pod \"maas-default-gateway-openshift-default-769df7846c-swlpw\" (UID: \"c5b6e9d2-ca83-4956-819e-d7ec7997acd8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-swlpw" Oct 30 21:01:43.234867 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:01:43.234729 5378 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c5b6e9d2-ca83-4956-819e-d7ec7997acd8-istio-data\") pod \"maas-default-gateway-openshift-default-769df7846c-swlpw\" (UID: \"c5b6e9d2-ca83-4956-819e-d7ec7997acd8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-swlpw" Oct 30 21:01:43.235002 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:01:43.234907 5378 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c5b6e9d2-ca83-4956-819e-d7ec7997acd8-istio-envoy\") pod \"maas-default-gateway-openshift-default-769df7846c-swlpw\" (UID: \"c5b6e9d2-ca83-4956-819e-d7ec7997acd8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-swlpw" Oct 30 21:01:43.235002 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:01:43.234962 5378 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c5b6e9d2-ca83-4956-819e-d7ec7997acd8-istio-token\") pod \"maas-default-gateway-openshift-default-769df7846c-swlpw\" (UID: \"c5b6e9d2-ca83-4956-819e-d7ec7997acd8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-swlpw" Oct 30 21:01:43.235112 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:01:43.235090 5378 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cwpcg\" (UniqueName: \"kubernetes.io/projected/c5b6e9d2-ca83-4956-819e-d7ec7997acd8-kube-api-access-cwpcg\") pod \"maas-default-gateway-openshift-default-769df7846c-swlpw\" (UID: \"c5b6e9d2-ca83-4956-819e-d7ec7997acd8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-swlpw" Oct 30 21:01:43.235151 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:01:43.235125 5378 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c5b6e9d2-ca83-4956-819e-d7ec7997acd8-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-769df7846c-swlpw\" (UID: \"c5b6e9d2-ca83-4956-819e-d7ec7997acd8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-swlpw" Oct 30 21:01:43.235250 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:01:43.235239 5378 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c5b6e9d2-ca83-4956-819e-d7ec7997acd8-credential-socket\") pod \"maas-default-gateway-openshift-default-769df7846c-swlpw\" (UID: \"c5b6e9d2-ca83-4956-819e-d7ec7997acd8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-swlpw" Oct 30 21:01:43.235287 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:01:43.235277 5378 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c5b6e9d2-ca83-4956-819e-d7ec7997acd8-workload-socket\") pod \"maas-default-gateway-openshift-default-769df7846c-swlpw\" (UID: \"c5b6e9d2-ca83-4956-819e-d7ec7997acd8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-swlpw" Oct 30 21:01:43.306133 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:01:43.306101 5378 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-769df7846c-swlpw"] Oct 30 21:01:43.337228 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:01:43.337192 5378 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c5b6e9d2-ca83-4956-819e-d7ec7997acd8-workload-certs\") pod \"maas-default-gateway-openshift-default-769df7846c-swlpw\" (UID: \"c5b6e9d2-ca83-4956-819e-d7ec7997acd8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-swlpw" Oct 30 21:01:43.337436 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:01:43.337245 5378 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c5b6e9d2-ca83-4956-819e-d7ec7997acd8-istio-podinfo\") pod \"maas-default-gateway-openshift-default-769df7846c-swlpw\" (UID: \"c5b6e9d2-ca83-4956-819e-d7ec7997acd8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-swlpw" Oct 30 21:01:43.337436 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:01:43.337276 5378 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c5b6e9d2-ca83-4956-819e-d7ec7997acd8-istio-data\") pod \"maas-default-gateway-openshift-default-769df7846c-swlpw\" (UID: \"c5b6e9d2-ca83-4956-819e-d7ec7997acd8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-swlpw" Oct 30 21:01:43.337436 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:01:43.337328 5378 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c5b6e9d2-ca83-4956-819e-d7ec7997acd8-istio-envoy\") pod \"maas-default-gateway-openshift-default-769df7846c-swlpw\" (UID: \"c5b6e9d2-ca83-4956-819e-d7ec7997acd8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-swlpw" Oct 30 21:01:43.337436 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:01:43.337363 5378 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c5b6e9d2-ca83-4956-819e-d7ec7997acd8-istio-token\") pod \"maas-default-gateway-openshift-default-769df7846c-swlpw\" (UID: \"c5b6e9d2-ca83-4956-819e-d7ec7997acd8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-swlpw" Oct 30 21:01:43.337695 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:01:43.337470 5378 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cwpcg\" (UniqueName: \"kubernetes.io/projected/c5b6e9d2-ca83-4956-819e-d7ec7997acd8-kube-api-access-cwpcg\") pod \"maas-default-gateway-openshift-default-769df7846c-swlpw\" (UID: \"c5b6e9d2-ca83-4956-819e-d7ec7997acd8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-swlpw" Oct 30 21:01:43.337695 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:01:43.337502 5378 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c5b6e9d2-ca83-4956-819e-d7ec7997acd8-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-769df7846c-swlpw\" (UID: \"c5b6e9d2-ca83-4956-819e-d7ec7997acd8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-swlpw" Oct 30 21:01:43.337695 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:01:43.337650 5378 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c5b6e9d2-ca83-4956-819e-d7ec7997acd8-credential-socket\") pod \"maas-default-gateway-openshift-default-769df7846c-swlpw\" (UID: \"c5b6e9d2-ca83-4956-819e-d7ec7997acd8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-swlpw" Oct 30 21:01:43.337695 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:01:43.337655 5378 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c5b6e9d2-ca83-4956-819e-d7ec7997acd8-workload-certs\") pod \"maas-default-gateway-openshift-default-769df7846c-swlpw\" (UID: \"c5b6e9d2-ca83-4956-819e-d7ec7997acd8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-swlpw" Oct 30 21:01:43.337898 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:01:43.337727 5378 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c5b6e9d2-ca83-4956-819e-d7ec7997acd8-workload-socket\") pod \"maas-default-gateway-openshift-default-769df7846c-swlpw\" (UID: \"c5b6e9d2-ca83-4956-819e-d7ec7997acd8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-swlpw" Oct 30 21:01:43.337980 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:01:43.337959 5378 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c5b6e9d2-ca83-4956-819e-d7ec7997acd8-credential-socket\") pod \"maas-default-gateway-openshift-default-769df7846c-swlpw\" (UID: \"c5b6e9d2-ca83-4956-819e-d7ec7997acd8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-swlpw" Oct 30 21:01:43.338138 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:01:43.338116 5378 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c5b6e9d2-ca83-4956-819e-d7ec7997acd8-workload-socket\") pod \"maas-default-gateway-openshift-default-769df7846c-swlpw\" (UID: \"c5b6e9d2-ca83-4956-819e-d7ec7997acd8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-swlpw" Oct 30 21:01:43.338651 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:01:43.338624 5378 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c5b6e9d2-ca83-4956-819e-d7ec7997acd8-istio-data\") pod \"maas-default-gateway-openshift-default-769df7846c-swlpw\" (UID: \"c5b6e9d2-ca83-4956-819e-d7ec7997acd8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-swlpw" Oct 30 21:01:43.339253 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:01:43.339228 5378 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c5b6e9d2-ca83-4956-819e-d7ec7997acd8-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-769df7846c-swlpw\" (UID: \"c5b6e9d2-ca83-4956-819e-d7ec7997acd8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-swlpw" Oct 30 21:01:43.341677 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:01:43.341650 5378 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c5b6e9d2-ca83-4956-819e-d7ec7997acd8-istio-envoy\") pod \"maas-default-gateway-openshift-default-769df7846c-swlpw\" (UID: \"c5b6e9d2-ca83-4956-819e-d7ec7997acd8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-swlpw" Oct 30 21:01:43.342866 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:01:43.342843 5378 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c5b6e9d2-ca83-4956-819e-d7ec7997acd8-istio-podinfo\") pod \"maas-default-gateway-openshift-default-769df7846c-swlpw\" (UID: \"c5b6e9d2-ca83-4956-819e-d7ec7997acd8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-swlpw" Oct 30 21:01:43.396475 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:01:43.396410 5378 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c5b6e9d2-ca83-4956-819e-d7ec7997acd8-istio-token\") pod \"maas-default-gateway-openshift-default-769df7846c-swlpw\" (UID: \"c5b6e9d2-ca83-4956-819e-d7ec7997acd8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-swlpw" Oct 30 21:01:43.396608 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:01:43.396520 5378 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cwpcg\" (UniqueName: \"kubernetes.io/projected/c5b6e9d2-ca83-4956-819e-d7ec7997acd8-kube-api-access-cwpcg\") pod \"maas-default-gateway-openshift-default-769df7846c-swlpw\" (UID: \"c5b6e9d2-ca83-4956-819e-d7ec7997acd8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-swlpw" Oct 30 21:01:43.415380 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:01:43.415354 5378 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-swlpw" Oct 30 21:01:43.588274 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:01:43.588242 5378 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ait-bc88b34e-55b79996d-kfqbm" Oct 30 21:01:43.849699 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:01:43.849652 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:01:43.867971 ip-10-0-138-149 kubenswrapper[5378]: W1030 21:01:43.867946 5378 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc5b6e9d2_ca83_4956_819e_d7ec7997acd8.slice/crio-5acba50e417e6ac10721947d1cbcd5bf2bc0771ba58def9b3fa4112c65fa15c8 WatchSource:0}: Error finding container 5acba50e417e6ac10721947d1cbcd5bf2bc0771ba58def9b3fa4112c65fa15c8: Status 404 returned error can't find the container with id 5acba50e417e6ac10721947d1cbcd5bf2bc0771ba58def9b3fa4112c65fa15c8 Oct 30 21:01:43.868182 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:01:43.868163 5378 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-769df7846c-swlpw"] Oct 30 21:01:43.870452 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:01:43.870416 5378 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 21:01:43.870557 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:01:43.870479 5378 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 21:01:43.870557 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:01:43.870510 5378 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 21:01:44.684617 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:01:44.684559 5378 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-swlpw" event={"ID":"c5b6e9d2-ca83-4956-819e-d7ec7997acd8","Type":"ContainerStarted","Data":"1835c8de6c7c1ff14bbea419facdc67b74fc50cb023b3e6fea23f4d4a6d80c79"} Oct 30 21:01:44.685027 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:01:44.684644 5378 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-swlpw" event={"ID":"c5b6e9d2-ca83-4956-819e-d7ec7997acd8","Type":"ContainerStarted","Data":"5acba50e417e6ac10721947d1cbcd5bf2bc0771ba58def9b3fa4112c65fa15c8"} Oct 30 21:01:44.924551 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:01:44.924497 5378 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-swlpw" podStartSLOduration=1.924482878 podStartE2EDuration="1.924482878s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:01:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:01:44.873328336 +0000 UTC m=+1395.301442256" watchObservedRunningTime="2026-10-30 21:01:44.924482878 +0000 UTC m=+1395.352596796" Oct 30 21:01:44.930554 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:01:44.930531 5378 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-f55142ef-75c5db7868-2kvrk"] Oct 30 21:01:44.934681 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:01:44.934632 5378 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-f55142ef-75c5db7868-2kvrk" Oct 30 21:01:44.986702 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:01:44.986668 5378 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-f55142ef-75c5db7868-2kvrk"] Oct 30 21:01:44.987480 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:01:44.987451 5378 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-migrate-f55142ef-serving-cert\"" Oct 30 21:01:45.077590 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:01:45.072446 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:01:45.085005 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:01:45.083464 5378 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3c485363-d417-4168-a210-ca908192b1d6-metrics-certs\") pod \"maas-api-e2e-migrate-f55142ef-75c5db7868-2kvrk\" (UID: \"3c485363-d417-4168-a210-ca908192b1d6\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-f55142ef-75c5db7868-2kvrk" Oct 30 21:01:45.085005 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:01:45.083855 5378 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3c485363-d417-4168-a210-ca908192b1d6-openshift-service-ca\") pod \"maas-api-e2e-migrate-f55142ef-75c5db7868-2kvrk\" (UID: \"3c485363-d417-4168-a210-ca908192b1d6\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-f55142ef-75c5db7868-2kvrk" Oct 30 21:01:45.085005 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:01:45.083908 5378 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jldgd\" (UniqueName: \"kubernetes.io/projected/3c485363-d417-4168-a210-ca908192b1d6-kube-api-access-jldgd\") pod \"maas-api-e2e-migrate-f55142ef-75c5db7868-2kvrk\" (UID: \"3c485363-d417-4168-a210-ca908192b1d6\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-f55142ef-75c5db7868-2kvrk" Oct 30 21:01:45.085005 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:01:45.083971 5378 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3c485363-d417-4168-a210-ca908192b1d6-maas-api-tls\") pod \"maas-api-e2e-migrate-f55142ef-75c5db7868-2kvrk\" (UID: \"3c485363-d417-4168-a210-ca908192b1d6\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-f55142ef-75c5db7868-2kvrk" Oct 30 21:01:45.194644 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:01:45.193555 5378 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3c485363-d417-4168-a210-ca908192b1d6-maas-api-tls\") pod \"maas-api-e2e-migrate-f55142ef-75c5db7868-2kvrk\" (UID: \"3c485363-d417-4168-a210-ca908192b1d6\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-f55142ef-75c5db7868-2kvrk" Oct 30 21:01:45.194644 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:01:45.193832 5378 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3c485363-d417-4168-a210-ca908192b1d6-metrics-certs\") pod \"maas-api-e2e-migrate-f55142ef-75c5db7868-2kvrk\" (UID: \"3c485363-d417-4168-a210-ca908192b1d6\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-f55142ef-75c5db7868-2kvrk" Oct 30 21:01:45.194644 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:01:45.194009 5378 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3c485363-d417-4168-a210-ca908192b1d6-openshift-service-ca\") pod \"maas-api-e2e-migrate-f55142ef-75c5db7868-2kvrk\" (UID: \"3c485363-d417-4168-a210-ca908192b1d6\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-f55142ef-75c5db7868-2kvrk" Oct 30 21:01:45.194644 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:01:45.194052 5378 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jldgd\" (UniqueName: \"kubernetes.io/projected/3c485363-d417-4168-a210-ca908192b1d6-kube-api-access-jldgd\") pod \"maas-api-e2e-migrate-f55142ef-75c5db7868-2kvrk\" (UID: \"3c485363-d417-4168-a210-ca908192b1d6\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-f55142ef-75c5db7868-2kvrk" Oct 30 21:01:45.202589 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:01:45.199165 5378 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3c485363-d417-4168-a210-ca908192b1d6-openshift-service-ca\") pod \"maas-api-e2e-migrate-f55142ef-75c5db7868-2kvrk\" (UID: \"3c485363-d417-4168-a210-ca908192b1d6\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-f55142ef-75c5db7868-2kvrk" Oct 30 21:01:45.225587 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:01:45.209726 5378 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3c485363-d417-4168-a210-ca908192b1d6-metrics-certs\") pod \"maas-api-e2e-migrate-f55142ef-75c5db7868-2kvrk\" (UID: \"3c485363-d417-4168-a210-ca908192b1d6\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-f55142ef-75c5db7868-2kvrk" Oct 30 21:01:45.265593 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:01:45.249504 5378 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3c485363-d417-4168-a210-ca908192b1d6-maas-api-tls\") pod \"maas-api-e2e-migrate-f55142ef-75c5db7868-2kvrk\" (UID: \"3c485363-d417-4168-a210-ca908192b1d6\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-f55142ef-75c5db7868-2kvrk" Oct 30 21:01:45.265593 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:01:45.262883 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:01:45.282586 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:01:45.274232 5378 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jldgd\" (UniqueName: \"kubernetes.io/projected/3c485363-d417-4168-a210-ca908192b1d6-kube-api-access-jldgd\") pod \"maas-api-e2e-migrate-f55142ef-75c5db7868-2kvrk\" (UID: \"3c485363-d417-4168-a210-ca908192b1d6\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-f55142ef-75c5db7868-2kvrk" Oct 30 21:01:45.402589 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:01:45.396221 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:01:45.427589 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:01:45.418927 5378 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-swlpw" Oct 30 21:01:45.427589 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:01:45.426735 5378 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-769df7846c-swlpw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.101:15021/healthz/ready\": dial tcp 10.133.0.101:15021: connect: connection refused" start-of-body= Oct 30 21:01:45.427589 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:01:45.426791 5378 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-swlpw" podUID="c5b6e9d2-ca83-4956-819e-d7ec7997acd8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.101:15021/healthz/ready\": dial tcp 10.133.0.101:15021: connect: connection refused" Oct 30 21:01:45.549658 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:01:45.549556 5378 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-f55142ef-75c5db7868-2kvrk" Oct 30 21:01:45.980662 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:01:45.978981 5378 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-f55142ef-75c5db7868-2kvrk"] Oct 30 21:01:45.980662 ip-10-0-138-149 kubenswrapper[5378]: W1030 21:01:45.979923 5378 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3c485363_d417_4168_a210_ca908192b1d6.slice/crio-5ecb900f0531ae15f95f509485113476391ae660fd7bdfbcac2588ca25a52117 WatchSource:0}: Error finding container 5ecb900f0531ae15f95f509485113476391ae660fd7bdfbcac2588ca25a52117: Status 404 returned error can't find the container with id 5ecb900f0531ae15f95f509485113476391ae660fd7bdfbcac2588ca25a52117 Oct 30 21:01:45.983593 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:01:45.983540 5378 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 30 21:01:46.416438 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:01:46.416399 5378 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-769df7846c-swlpw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.101:15021/healthz/ready\": dial tcp 10.133.0.101:15021: connect: connection refused" start-of-body= Oct 30 21:01:46.416650 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:01:46.416457 5378 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-swlpw" podUID="c5b6e9d2-ca83-4956-819e-d7ec7997acd8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.101:15021/healthz/ready\": dial tcp 10.133.0.101:15021: connect: connection refused" Oct 30 21:01:46.701096 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:01:46.701012 5378 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-f55142ef-75c5db7868-2kvrk" event={"ID":"3c485363-d417-4168-a210-ca908192b1d6","Type":"ContainerStarted","Data":"5ecb900f0531ae15f95f509485113476391ae660fd7bdfbcac2588ca25a52117"} Oct 30 21:01:47.416198 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:01:47.416168 5378 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-769df7846c-swlpw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.101:15021/healthz/ready\": dial tcp 10.133.0.101:15021: connect: connection refused" start-of-body= Oct 30 21:01:47.416612 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:01:47.416216 5378 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-swlpw" podUID="c5b6e9d2-ca83-4956-819e-d7ec7997acd8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.101:15021/healthz/ready\": dial tcp 10.133.0.101:15021: connect: connection refused" Oct 30 21:01:47.709743 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:01:47.709654 5378 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-f55142ef-75c5db7868-2kvrk" event={"ID":"3c485363-d417-4168-a210-ca908192b1d6","Type":"ContainerStarted","Data":"710bce613fbe90cbec349bd5df35910e504674ea1411a61f25be704b2f5493a0"} Oct 30 21:01:47.709999 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:01:47.709976 5378 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-f55142ef-75c5db7868-2kvrk" Oct 30 21:01:48.416644 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:01:48.416605 5378 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-769df7846c-swlpw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.101:15021/healthz/ready\": dial tcp 10.133.0.101:15021: connect: connection refused" start-of-body= Oct 30 21:01:48.417025 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:01:48.416661 5378 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-swlpw" podUID="c5b6e9d2-ca83-4956-819e-d7ec7997acd8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.101:15021/healthz/ready\": dial tcp 10.133.0.101:15021: connect: connection refused" Oct 30 21:01:48.847295 ip-10-0-138-149 kubenswrapper[5378]: E1030 21:01:48.847265 5378 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q6mx7" containerName="extractor" Oct 30 21:01:49.415670 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:01:49.415638 5378 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-769df7846c-swlpw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.101:15021/healthz/ready\": dial tcp 10.133.0.101:15021: connect: connection refused" start-of-body= Oct 30 21:01:49.415846 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:01:49.415689 5378 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-swlpw" podUID="c5b6e9d2-ca83-4956-819e-d7ec7997acd8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.101:15021/healthz/ready\": dial tcp 10.133.0.101:15021: connect: connection refused" Oct 30 21:01:49.933610 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:01:49.933547 5378 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-f55142ef-75c5db7868-2kvrk" podStartSLOduration=4.937005408 podStartE2EDuration="5.933531717s" totalImagesPullingTime="996.526309ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:01:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:01:47.841087341 +0000 UTC m=+1398.269201260" watchObservedRunningTime="2026-10-30 21:01:49.933531717 +0000 UTC m=+1400.361645647" Oct 30 21:01:49.934618 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:01:49.934589 5378 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-83851760-854c79488f-xfj9r"] Oct 30 21:01:49.939245 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:01:49.939227 5378 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-83851760-854c79488f-xfj9r" Oct 30 21:01:50.026585 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:01:50.026537 5378 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2kwd4\" (UniqueName: \"kubernetes.io/projected/218105d7-352a-40b6-8940-274d2a0a7720-kube-api-access-2kwd4\") pod \"maas-api-e2e-ait-83851760-854c79488f-xfj9r\" (UID: \"218105d7-352a-40b6-8940-274d2a0a7720\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-83851760-854c79488f-xfj9r" Oct 30 21:01:50.026790 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:01:50.026604 5378 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/218105d7-352a-40b6-8940-274d2a0a7720-openshift-service-ca\") pod \"maas-api-e2e-ait-83851760-854c79488f-xfj9r\" (UID: \"218105d7-352a-40b6-8940-274d2a0a7720\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-83851760-854c79488f-xfj9r" Oct 30 21:01:50.026866 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:01:50.026785 5378 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/218105d7-352a-40b6-8940-274d2a0a7720-metrics-certs\") pod \"maas-api-e2e-ait-83851760-854c79488f-xfj9r\" (UID: \"218105d7-352a-40b6-8940-274d2a0a7720\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-83851760-854c79488f-xfj9r" Oct 30 21:01:50.026866 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:01:50.026845 5378 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/218105d7-352a-40b6-8940-274d2a0a7720-maas-api-tls\") pod \"maas-api-e2e-ait-83851760-854c79488f-xfj9r\" (UID: \"218105d7-352a-40b6-8940-274d2a0a7720\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-83851760-854c79488f-xfj9r" Oct 30 21:01:50.042697 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:01:50.042665 5378 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-83851760-serving-cert\"" Oct 30 21:01:50.048921 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:01:50.048893 5378 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-83851760-854c79488f-xfj9r"] Oct 30 21:01:50.121666 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:01:50.121632 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:01:50.128249 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:01:50.128218 5378 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2kwd4\" (UniqueName: \"kubernetes.io/projected/218105d7-352a-40b6-8940-274d2a0a7720-kube-api-access-2kwd4\") pod \"maas-api-e2e-ait-83851760-854c79488f-xfj9r\" (UID: \"218105d7-352a-40b6-8940-274d2a0a7720\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-83851760-854c79488f-xfj9r" Oct 30 21:01:50.128249 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:01:50.128251 5378 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/218105d7-352a-40b6-8940-274d2a0a7720-openshift-service-ca\") pod \"maas-api-e2e-ait-83851760-854c79488f-xfj9r\" (UID: \"218105d7-352a-40b6-8940-274d2a0a7720\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-83851760-854c79488f-xfj9r" Oct 30 21:01:50.128457 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:01:50.128315 5378 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/218105d7-352a-40b6-8940-274d2a0a7720-metrics-certs\") pod \"maas-api-e2e-ait-83851760-854c79488f-xfj9r\" (UID: \"218105d7-352a-40b6-8940-274d2a0a7720\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-83851760-854c79488f-xfj9r" Oct 30 21:01:50.128457 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:01:50.128341 5378 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/218105d7-352a-40b6-8940-274d2a0a7720-maas-api-tls\") pod \"maas-api-e2e-ait-83851760-854c79488f-xfj9r\" (UID: \"218105d7-352a-40b6-8940-274d2a0a7720\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-83851760-854c79488f-xfj9r" Oct 30 21:01:50.129530 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:01:50.129502 5378 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/218105d7-352a-40b6-8940-274d2a0a7720-openshift-service-ca\") pod \"maas-api-e2e-ait-83851760-854c79488f-xfj9r\" (UID: \"218105d7-352a-40b6-8940-274d2a0a7720\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-83851760-854c79488f-xfj9r" Oct 30 21:01:50.129717 ip-10-0-138-149 kubenswrapper[5378]: E1030 21:01:50.129547 5378 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ait-83851760-serving-cert: secret "maas-api-e2e-ait-83851760-serving-cert" not found Oct 30 21:01:50.129878 ip-10-0-138-149 kubenswrapper[5378]: E1030 21:01:50.129866 5378 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/218105d7-352a-40b6-8940-274d2a0a7720-maas-api-tls podName:218105d7-352a-40b6-8940-274d2a0a7720 nodeName:}" failed. No retries permitted until 2026-10-30 21:01:50.629843521 +0000 UTC m=+1401.057957423 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/218105d7-352a-40b6-8940-274d2a0a7720-maas-api-tls") pod "maas-api-e2e-ait-83851760-854c79488f-xfj9r" (UID: "218105d7-352a-40b6-8940-274d2a0a7720") : secret "maas-api-e2e-ait-83851760-serving-cert" not found Oct 30 21:01:50.142467 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:01:50.142433 5378 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/218105d7-352a-40b6-8940-274d2a0a7720-metrics-certs\") pod \"maas-api-e2e-ait-83851760-854c79488f-xfj9r\" (UID: \"218105d7-352a-40b6-8940-274d2a0a7720\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-83851760-854c79488f-xfj9r" Oct 30 21:01:50.250709 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:01:50.250587 5378 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2kwd4\" (UniqueName: \"kubernetes.io/projected/218105d7-352a-40b6-8940-274d2a0a7720-kube-api-access-2kwd4\") pod \"maas-api-e2e-ait-83851760-854c79488f-xfj9r\" (UID: \"218105d7-352a-40b6-8940-274d2a0a7720\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-83851760-854c79488f-xfj9r" Oct 30 21:01:50.299264 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:01:50.299223 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:01:50.429689 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:01:50.429248 5378 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-769df7846c-swlpw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.101:15021/healthz/ready\": dial tcp 10.133.0.101:15021: connect: connection refused" start-of-body= Oct 30 21:01:50.429689 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:01:50.429310 5378 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-swlpw" podUID="c5b6e9d2-ca83-4956-819e-d7ec7997acd8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.101:15021/healthz/ready\": dial tcp 10.133.0.101:15021: connect: connection refused" Oct 30 21:01:50.431151 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:01:50.431032 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:01:50.685061 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:01:50.685020 5378 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/218105d7-352a-40b6-8940-274d2a0a7720-maas-api-tls\") pod \"maas-api-e2e-ait-83851760-854c79488f-xfj9r\" (UID: \"218105d7-352a-40b6-8940-274d2a0a7720\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-83851760-854c79488f-xfj9r" Oct 30 21:01:50.687448 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:01:50.687418 5378 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/218105d7-352a-40b6-8940-274d2a0a7720-maas-api-tls\") pod \"maas-api-e2e-ait-83851760-854c79488f-xfj9r\" (UID: \"218105d7-352a-40b6-8940-274d2a0a7720\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-83851760-854c79488f-xfj9r" Oct 30 21:01:50.870512 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:01:50.870483 5378 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-83851760-854c79488f-xfj9r" Oct 30 21:01:51.250324 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:01:51.250277 5378 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-83851760-854c79488f-xfj9r"] Oct 30 21:01:51.250808 ip-10-0-138-149 kubenswrapper[5378]: W1030 21:01:51.250549 5378 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod218105d7_352a_40b6_8940_274d2a0a7720.slice/crio-a026f60d685fea530db3206f21d8c0db5fb26e8f6c17ebea6e95adde8b485ed6 WatchSource:0}: Error finding container a026f60d685fea530db3206f21d8c0db5fb26e8f6c17ebea6e95adde8b485ed6: Status 404 returned error can't find the container with id a026f60d685fea530db3206f21d8c0db5fb26e8f6c17ebea6e95adde8b485ed6 Oct 30 21:01:51.416123 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:01:51.416090 5378 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-769df7846c-swlpw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.101:15021/healthz/ready\": dial tcp 10.133.0.101:15021: connect: connection refused" start-of-body= Oct 30 21:01:51.416278 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:01:51.416140 5378 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-swlpw" podUID="c5b6e9d2-ca83-4956-819e-d7ec7997acd8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.101:15021/healthz/ready\": dial tcp 10.133.0.101:15021: connect: connection refused" Oct 30 21:01:51.744086 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:01:51.744052 5378 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-83851760-854c79488f-xfj9r" event={"ID":"218105d7-352a-40b6-8940-274d2a0a7720","Type":"ContainerStarted","Data":"a026f60d685fea530db3206f21d8c0db5fb26e8f6c17ebea6e95adde8b485ed6"} Oct 30 21:01:52.687734 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:01:52.687701 5378 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-swlpw" Oct 30 21:01:52.688220 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:01:52.688073 5378 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-swlpw" Oct 30 21:01:52.688951 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:01:52.688931 5378 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-swlpw" Oct 30 21:01:52.751789 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:01:52.751739 5378 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-83851760-854c79488f-xfj9r" event={"ID":"218105d7-352a-40b6-8940-274d2a0a7720","Type":"ContainerStarted","Data":"fb4311eb069dccf57b23c5f58582cc96d98c61d689d546eea86f46f32775e1fd"} Oct 30 21:01:52.752006 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:01:52.751988 5378 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-83851760-854c79488f-xfj9r" Oct 30 21:01:52.987440 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:01:52.987344 5378 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-83851760-854c79488f-xfj9r" podStartSLOduration=3.15056476 podStartE2EDuration="3.987331116s" totalImagesPullingTime="836.766356ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:01:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:01:52.985756091 +0000 UTC m=+1403.413870023" watchObservedRunningTime="2026-10-30 21:01:52.987331116 +0000 UTC m=+1403.415445035" Oct 30 21:01:53.721486 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:01:53.721458 5378 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-f55142ef-75c5db7868-2kvrk" Oct 30 21:01:55.486282 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:01:55.486250 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:01:58.765132 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:01:58.765104 5378 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-83851760-854c79488f-xfj9r" Oct 30 21:02:00.264241 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:02:00.264200 5378 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-r9g59"] Oct 30 21:02:00.420430 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:02:00.420397 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:02:02.550521 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:02:02.549707 5378 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-d061d0/distinct-w2-d061d0-kserve-679c7cd7c4-8czrv" podUID="2c139b9f-c7df-4675-b431-95d4136095e5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:02:02.852713 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:02:02.852670 5378 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-1fdaa7/premium-w4-1fdaa7-kserve-55dffbd5cd-gvpxq" podUID="ec282a4b-742f-4382-a296-335dcd2e9636" containerName="main" probeResult="failure" output="Get \"https://10.133.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:02:02.871913 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:02:02.871879 5378 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-d061d0/simulator-w2-d061d0-kserve-8484f9678b-lnszm" podUID="525fac02-b21c-4dd5-b553-72bcc36062c5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 21:02:02.905589 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:02:02.899689 5378 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-1fdaa7/simulator-w4-1fdaa7-kserve-9866d468c-fwmsk" podUID="09219832-010e-4ba8-bfb9-5a2884b358e1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": context deadline exceeded" Oct 30 21:02:03.000589 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:02:02.994707 5378 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-1fdaa7/distinct-w4-1fdaa7-kserve-86dc9774b9-6x2gf" podUID="2e162c95-0527-4cb3-85a3-795aae212d29" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:02:03.267802 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:02:03.267687 5378 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-8fa2a7/simulator-w6-8fa2a7-kserve-6cbf5bd894-lfpcq" podUID="f409f469-71fc-445b-bdc8-b8e84a4d1dc6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:02:03.354718 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:02:03.354672 5378 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-8fa2a7/premium-w6-8fa2a7-kserve-64bf4bf6c-dxrdr" podUID="484cba5f-b415-430a-b52d-50b1293be948" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:02:03.431680 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:02:03.431635 5378 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-781058/distinct-w0-781058-kserve-8bdf9bfcd-hzdfk" podUID="a9678fc2-1c56-4d45-9007-eee5049e0f48" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/ready\": context deadline exceeded" Oct 30 21:02:03.512603 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:02:03.500388 5378 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-f9f277/simulator-w1-f9f277-kserve-5d9685f85c-2h78x" podUID="dc662e9d-914c-4dfb-a86f-0fb048f7b690" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": context deadline exceeded" Oct 30 21:02:03.554194 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:02:03.554091 5378 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-d061d0/distinct-w2-d061d0-kserve-679c7cd7c4-8czrv" podUID="2c139b9f-c7df-4675-b431-95d4136095e5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:02:03.555152 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:02:03.555129 5378 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-d061d0/distinct-w2-d061d0-kserve-679c7cd7c4-8czrv" Oct 30 21:02:03.563693 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:02:03.563661 5378 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-f9f277/distinct-2-w1-f9f277-kserve-6c49cf5dcb-h4x74" podUID="9eb6ff54-b77b-453c-a166-988b98cf8f49" containerName="main" probeResult="failure" output="Get \"https://10.133.0.76:8000/ready\": context deadline exceeded" Oct 30 21:02:03.857855 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:02:03.857799 5378 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-1fdaa7/premium-w4-1fdaa7-kserve-55dffbd5cd-gvpxq" podUID="ec282a4b-742f-4382-a296-335dcd2e9636" containerName="main" probeResult="failure" output="Get \"https://10.133.0.59:8000/ready\": context deadline exceeded" Oct 30 21:02:03.888362 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:02:03.888001 5378 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-d061d0/simulator-w2-d061d0-kserve-8484f9678b-lnszm" podUID="525fac02-b21c-4dd5-b553-72bcc36062c5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/ready\": context deadline exceeded" Oct 30 21:02:03.904603 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:02:03.903819 5378 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-1fdaa7/simulator-w4-1fdaa7-kserve-9866d468c-fwmsk" podUID="09219832-010e-4ba8-bfb9-5a2884b358e1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:02:03.937483 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:02:03.937442 5378 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-1fdaa7/premium-w4-1fdaa7-kserve-55dffbd5cd-gvpxq" Oct 30 21:02:03.937806 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:02:03.937787 5378 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-d061d0/simulator-w2-d061d0-kserve-8484f9678b-lnszm" Oct 30 21:02:03.938002 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:02:03.937979 5378 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-1fdaa7/simulator-w4-1fdaa7-kserve-9866d468c-fwmsk" Oct 30 21:02:04.013587 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:02:04.006641 5378 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-1fdaa7/distinct-w4-1fdaa7-kserve-86dc9774b9-6x2gf" podUID="2e162c95-0527-4cb3-85a3-795aae212d29" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/ready\": context deadline exceeded" Oct 30 21:02:04.013587 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:02:04.007013 5378 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-1fdaa7/distinct-w4-1fdaa7-kserve-86dc9774b9-6x2gf" Oct 30 21:02:04.277179 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:02:04.277081 5378 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-8fa2a7/simulator-w6-8fa2a7-kserve-6cbf5bd894-lfpcq" podUID="f409f469-71fc-445b-bdc8-b8e84a4d1dc6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:02:04.309805 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:02:04.289046 5378 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-8fa2a7/simulator-w6-8fa2a7-kserve-6cbf5bd894-lfpcq" Oct 30 21:02:04.353913 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:02:04.353870 5378 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-1fdaa7/premium-w4-1fdaa7-kserve-55dffbd5cd-gvpxq" podUID="ec282a4b-742f-4382-a296-335dcd2e9636" containerName="main" probeResult="failure" output="Get \"https://10.133.0.59:8000/health\": context deadline exceeded" Oct 30 21:02:04.422590 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:02:04.407885 5378 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-1fdaa7/premium-w4-1fdaa7-kserve-55dffbd5cd-gvpxq" Oct 30 21:02:04.455128 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:02:04.455097 5378 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-8fa2a7/simulator-w6-8fa2a7-kserve-6cbf5bd894-lfpcq" Oct 30 21:02:04.460803 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:02:04.460769 5378 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-d061d0/distinct-w2-d061d0-kserve-679c7cd7c4-8czrv" Oct 30 21:02:04.562921 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:02:04.562829 5378 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-1fdaa7/distinct-w4-1fdaa7-kserve-86dc9774b9-6x2gf" Oct 30 21:02:04.569680 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:02:04.569614 5378 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-f9f277/distinct-2-w1-f9f277-kserve-6c49cf5dcb-h4x74" podUID="9eb6ff54-b77b-453c-a166-988b98cf8f49" containerName="main" probeResult="failure" output="Get \"https://10.133.0.76:8000/ready\": context deadline exceeded" Oct 30 21:02:04.580684 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:02:04.574660 5378 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-f9f277/distinct-2-w1-f9f277-kserve-6c49cf5dcb-h4x74" Oct 30 21:02:04.580684 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:02:04.574739 5378 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-1fdaa7/simulator-w4-1fdaa7-kserve-9866d468c-fwmsk" podUID="09219832-010e-4ba8-bfb9-5a2884b358e1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:02:04.749663 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:02:04.749633 5378 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-f9f277/distinct-2-w1-f9f277-kserve-6c49cf5dcb-h4x74" Oct 30 21:02:04.795241 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:02:04.795213 5378 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-1fdaa7/simulator-w4-1fdaa7-kserve-9866d468c-fwmsk" Oct 30 21:02:04.823522 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:02:04.823418 5378 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-d061d0/simulator-w2-d061d0-kserve-8484f9678b-lnszm" Oct 30 21:02:06.114964 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:02:06.114916 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:02:10.967769 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:02:10.967710 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:02:15.775353 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:02:15.775317 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:02:18.837901 ip-10-0-138-149 kubenswrapper[5378]: E1030 21:02:18.837863 5378 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q6mx7" containerName="extractor" Oct 30 21:02:20.322799 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:02:20.322761 5378 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-bc88b34e-67f64959c8-5kndg"] Oct 30 21:02:20.327251 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:02:20.327231 5378 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-bc88b34e-67f64959c8-5kndg" Oct 30 21:02:20.383654 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:02:20.383627 5378 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-bc88b34e-67f64959c8-5kndg"] Oct 30 21:02:20.429752 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:02:20.429720 5378 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-bc88b34e-serving-cert\"" Oct 30 21:02:20.504599 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:02:20.504536 5378 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2fcd8f8c-42a7-4a72-9004-97c7f84fffac-maas-api-tls\") pod \"maas-api-e2e-ait-bc88b34e-67f64959c8-5kndg\" (UID: \"2fcd8f8c-42a7-4a72-9004-97c7f84fffac\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-bc88b34e-67f64959c8-5kndg" Oct 30 21:02:20.504778 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:02:20.504675 5378 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2fcd8f8c-42a7-4a72-9004-97c7f84fffac-openshift-service-ca\") pod \"maas-api-e2e-ait-bc88b34e-67f64959c8-5kndg\" (UID: \"2fcd8f8c-42a7-4a72-9004-97c7f84fffac\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-bc88b34e-67f64959c8-5kndg" Oct 30 21:02:20.504845 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:02:20.504796 5378 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2fcd8f8c-42a7-4a72-9004-97c7f84fffac-metrics-certs\") pod \"maas-api-e2e-ait-bc88b34e-67f64959c8-5kndg\" (UID: \"2fcd8f8c-42a7-4a72-9004-97c7f84fffac\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-bc88b34e-67f64959c8-5kndg" Oct 30 21:02:20.504845 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:02:20.504838 5378 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ghglj\" (UniqueName: \"kubernetes.io/projected/2fcd8f8c-42a7-4a72-9004-97c7f84fffac-kube-api-access-ghglj\") pod \"maas-api-e2e-ait-bc88b34e-67f64959c8-5kndg\" (UID: \"2fcd8f8c-42a7-4a72-9004-97c7f84fffac\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-bc88b34e-67f64959c8-5kndg" Oct 30 21:02:20.510450 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:02:20.510424 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:02:20.607737 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:02:20.607686 5378 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2fcd8f8c-42a7-4a72-9004-97c7f84fffac-metrics-certs\") pod \"maas-api-e2e-ait-bc88b34e-67f64959c8-5kndg\" (UID: \"2fcd8f8c-42a7-4a72-9004-97c7f84fffac\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-bc88b34e-67f64959c8-5kndg" Oct 30 21:02:20.608734 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:02:20.608708 5378 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ghglj\" (UniqueName: \"kubernetes.io/projected/2fcd8f8c-42a7-4a72-9004-97c7f84fffac-kube-api-access-ghglj\") pod \"maas-api-e2e-ait-bc88b34e-67f64959c8-5kndg\" (UID: \"2fcd8f8c-42a7-4a72-9004-97c7f84fffac\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-bc88b34e-67f64959c8-5kndg" Oct 30 21:02:20.609360 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:02:20.609338 5378 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2fcd8f8c-42a7-4a72-9004-97c7f84fffac-maas-api-tls\") pod \"maas-api-e2e-ait-bc88b34e-67f64959c8-5kndg\" (UID: \"2fcd8f8c-42a7-4a72-9004-97c7f84fffac\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-bc88b34e-67f64959c8-5kndg" Oct 30 21:02:20.610164 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:02:20.609829 5378 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2fcd8f8c-42a7-4a72-9004-97c7f84fffac-openshift-service-ca\") pod \"maas-api-e2e-ait-bc88b34e-67f64959c8-5kndg\" (UID: \"2fcd8f8c-42a7-4a72-9004-97c7f84fffac\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-bc88b34e-67f64959c8-5kndg" Oct 30 21:02:20.611310 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:02:20.611085 5378 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2fcd8f8c-42a7-4a72-9004-97c7f84fffac-openshift-service-ca\") pod \"maas-api-e2e-ait-bc88b34e-67f64959c8-5kndg\" (UID: \"2fcd8f8c-42a7-4a72-9004-97c7f84fffac\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-bc88b34e-67f64959c8-5kndg" Oct 30 21:02:20.611453 ip-10-0-138-149 kubenswrapper[5378]: E1030 21:02:20.609669 5378 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ait-bc88b34e-serving-cert: secret "maas-api-e2e-ait-bc88b34e-serving-cert" not found Oct 30 21:02:20.611619 ip-10-0-138-149 kubenswrapper[5378]: E1030 21:02:20.611607 5378 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2fcd8f8c-42a7-4a72-9004-97c7f84fffac-maas-api-tls podName:2fcd8f8c-42a7-4a72-9004-97c7f84fffac nodeName:}" failed. No retries permitted until 2026-10-30 21:02:21.11155854 +0000 UTC m=+1431.539672456 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/2fcd8f8c-42a7-4a72-9004-97c7f84fffac-maas-api-tls") pod "maas-api-e2e-ait-bc88b34e-67f64959c8-5kndg" (UID: "2fcd8f8c-42a7-4a72-9004-97c7f84fffac") : secret "maas-api-e2e-ait-bc88b34e-serving-cert" not found Oct 30 21:02:20.645847 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:02:20.645814 5378 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2fcd8f8c-42a7-4a72-9004-97c7f84fffac-metrics-certs\") pod \"maas-api-e2e-ait-bc88b34e-67f64959c8-5kndg\" (UID: \"2fcd8f8c-42a7-4a72-9004-97c7f84fffac\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-bc88b34e-67f64959c8-5kndg" Oct 30 21:02:20.679248 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:02:20.679022 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:02:20.679417 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:02:20.679296 5378 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ghglj\" (UniqueName: \"kubernetes.io/projected/2fcd8f8c-42a7-4a72-9004-97c7f84fffac-kube-api-access-ghglj\") pod \"maas-api-e2e-ait-bc88b34e-67f64959c8-5kndg\" (UID: \"2fcd8f8c-42a7-4a72-9004-97c7f84fffac\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-bc88b34e-67f64959c8-5kndg" Oct 30 21:02:20.825173 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:02:20.825133 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:02:21.132983 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:02:21.132951 5378 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2fcd8f8c-42a7-4a72-9004-97c7f84fffac-maas-api-tls\") pod \"maas-api-e2e-ait-bc88b34e-67f64959c8-5kndg\" (UID: \"2fcd8f8c-42a7-4a72-9004-97c7f84fffac\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-bc88b34e-67f64959c8-5kndg" Oct 30 21:02:21.135449 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:02:21.135422 5378 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2fcd8f8c-42a7-4a72-9004-97c7f84fffac-maas-api-tls\") pod \"maas-api-e2e-ait-bc88b34e-67f64959c8-5kndg\" (UID: \"2fcd8f8c-42a7-4a72-9004-97c7f84fffac\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-bc88b34e-67f64959c8-5kndg" Oct 30 21:02:21.242891 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:02:21.242859 5378 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-bc88b34e-67f64959c8-5kndg" Oct 30 21:02:21.621919 ip-10-0-138-149 kubenswrapper[5378]: W1030 21:02:21.621886 5378 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2fcd8f8c_42a7_4a72_9004_97c7f84fffac.slice/crio-e9fecc9a48e03dedefaeb67b2578566e5335f7295f4eea65d3d3443941acca5a WatchSource:0}: Error finding container e9fecc9a48e03dedefaeb67b2578566e5335f7295f4eea65d3d3443941acca5a: Status 404 returned error can't find the container with id e9fecc9a48e03dedefaeb67b2578566e5335f7295f4eea65d3d3443941acca5a Oct 30 21:02:21.622528 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:02:21.622422 5378 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-bc88b34e-67f64959c8-5kndg"] Oct 30 21:02:22.453723 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:02:22.453693 5378 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-bc88b34e-67f64959c8-5kndg" event={"ID":"2fcd8f8c-42a7-4a72-9004-97c7f84fffac","Type":"ContainerStarted","Data":"e9fecc9a48e03dedefaeb67b2578566e5335f7295f4eea65d3d3443941acca5a"} Oct 30 21:02:23.548758 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:02:23.548714 5378 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-bc88b34e-67f64959c8-5kndg" event={"ID":"2fcd8f8c-42a7-4a72-9004-97c7f84fffac","Type":"ContainerStarted","Data":"b3a953496eb8de8d9cd004730ff7563cc90d7e8f55b8b6c0addd459f12e184e0"} Oct 30 21:02:23.565539 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:02:23.565503 5378 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-bc88b34e-67f64959c8-5kndg" Oct 30 21:02:23.681902 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:02:23.681849 5378 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-bc88b34e-67f64959c8-5kndg" podStartSLOduration=2.860029196 podStartE2EDuration="3.681834516s" totalImagesPullingTime="821.80532ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:02:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:02:23.680124832 +0000 UTC m=+1434.108238751" watchObservedRunningTime="2026-10-30 21:02:23.681834516 +0000 UTC m=+1434.109948434" Oct 30 21:02:25.754899 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:02:25.754865 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:02:28.481880 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:02:28.481846 5378 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-769df7846c-b77jh"] Oct 30 21:02:28.486440 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:02:28.486416 5378 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-b77jh" Oct 30 21:02:28.565541 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:02:28.565510 5378 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2rjhh\" (UniqueName: \"kubernetes.io/projected/7bfe56ea-dd63-441e-a5c1-9b116f068419-kube-api-access-2rjhh\") pod \"maas-default-gateway-openshift-default-769df7846c-b77jh\" (UID: \"7bfe56ea-dd63-441e-a5c1-9b116f068419\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-b77jh" Oct 30 21:02:28.565744 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:02:28.565649 5378 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7bfe56ea-dd63-441e-a5c1-9b116f068419-istio-token\") pod \"maas-default-gateway-openshift-default-769df7846c-b77jh\" (UID: \"7bfe56ea-dd63-441e-a5c1-9b116f068419\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-b77jh" Oct 30 21:02:28.565744 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:02:28.565693 5378 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7bfe56ea-dd63-441e-a5c1-9b116f068419-istio-data\") pod \"maas-default-gateway-openshift-default-769df7846c-b77jh\" (UID: \"7bfe56ea-dd63-441e-a5c1-9b116f068419\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-b77jh" Oct 30 21:02:28.565855 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:02:28.565824 5378 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7bfe56ea-dd63-441e-a5c1-9b116f068419-workload-socket\") pod \"maas-default-gateway-openshift-default-769df7846c-b77jh\" (UID: \"7bfe56ea-dd63-441e-a5c1-9b116f068419\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-b77jh" Oct 30 21:02:28.565912 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:02:28.565882 5378 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7bfe56ea-dd63-441e-a5c1-9b116f068419-workload-certs\") pod \"maas-default-gateway-openshift-default-769df7846c-b77jh\" (UID: \"7bfe56ea-dd63-441e-a5c1-9b116f068419\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-b77jh" Oct 30 21:02:28.566023 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:02:28.566004 5378 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7bfe56ea-dd63-441e-a5c1-9b116f068419-istio-envoy\") pod \"maas-default-gateway-openshift-default-769df7846c-b77jh\" (UID: \"7bfe56ea-dd63-441e-a5c1-9b116f068419\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-b77jh" Oct 30 21:02:28.566141 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:02:28.566058 5378 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7bfe56ea-dd63-441e-a5c1-9b116f068419-credential-socket\") pod \"maas-default-gateway-openshift-default-769df7846c-b77jh\" (UID: \"7bfe56ea-dd63-441e-a5c1-9b116f068419\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-b77jh" Oct 30 21:02:28.566200 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:02:28.566170 5378 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7bfe56ea-dd63-441e-a5c1-9b116f068419-istio-podinfo\") pod \"maas-default-gateway-openshift-default-769df7846c-b77jh\" (UID: \"7bfe56ea-dd63-441e-a5c1-9b116f068419\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-b77jh" Oct 30 21:02:28.566269 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:02:28.566255 5378 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7bfe56ea-dd63-441e-a5c1-9b116f068419-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-769df7846c-b77jh\" (UID: \"7bfe56ea-dd63-441e-a5c1-9b116f068419\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-b77jh" Oct 30 21:02:28.600356 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:02:28.600319 5378 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-769df7846c-b77jh"] Oct 30 21:02:28.667334 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:02:28.667301 5378 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2rjhh\" (UniqueName: \"kubernetes.io/projected/7bfe56ea-dd63-441e-a5c1-9b116f068419-kube-api-access-2rjhh\") pod \"maas-default-gateway-openshift-default-769df7846c-b77jh\" (UID: \"7bfe56ea-dd63-441e-a5c1-9b116f068419\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-b77jh" Oct 30 21:02:28.667334 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:02:28.667340 5378 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7bfe56ea-dd63-441e-a5c1-9b116f068419-istio-token\") pod \"maas-default-gateway-openshift-default-769df7846c-b77jh\" (UID: \"7bfe56ea-dd63-441e-a5c1-9b116f068419\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-b77jh" Oct 30 21:02:28.667622 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:02:28.667358 5378 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7bfe56ea-dd63-441e-a5c1-9b116f068419-istio-data\") pod \"maas-default-gateway-openshift-default-769df7846c-b77jh\" (UID: \"7bfe56ea-dd63-441e-a5c1-9b116f068419\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-b77jh" Oct 30 21:02:28.667622 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:02:28.667402 5378 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7bfe56ea-dd63-441e-a5c1-9b116f068419-workload-socket\") pod \"maas-default-gateway-openshift-default-769df7846c-b77jh\" (UID: \"7bfe56ea-dd63-441e-a5c1-9b116f068419\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-b77jh" Oct 30 21:02:28.667622 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:02:28.667426 5378 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7bfe56ea-dd63-441e-a5c1-9b116f068419-workload-certs\") pod \"maas-default-gateway-openshift-default-769df7846c-b77jh\" (UID: \"7bfe56ea-dd63-441e-a5c1-9b116f068419\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-b77jh" Oct 30 21:02:28.667622 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:02:28.667498 5378 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7bfe56ea-dd63-441e-a5c1-9b116f068419-istio-envoy\") pod \"maas-default-gateway-openshift-default-769df7846c-b77jh\" (UID: \"7bfe56ea-dd63-441e-a5c1-9b116f068419\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-b77jh" Oct 30 21:02:28.667622 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:02:28.667547 5378 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7bfe56ea-dd63-441e-a5c1-9b116f068419-credential-socket\") pod \"maas-default-gateway-openshift-default-769df7846c-b77jh\" (UID: \"7bfe56ea-dd63-441e-a5c1-9b116f068419\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-b77jh" Oct 30 21:02:28.667906 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:02:28.667675 5378 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7bfe56ea-dd63-441e-a5c1-9b116f068419-istio-podinfo\") pod \"maas-default-gateway-openshift-default-769df7846c-b77jh\" (UID: \"7bfe56ea-dd63-441e-a5c1-9b116f068419\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-b77jh" Oct 30 21:02:28.667906 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:02:28.667743 5378 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7bfe56ea-dd63-441e-a5c1-9b116f068419-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-769df7846c-b77jh\" (UID: \"7bfe56ea-dd63-441e-a5c1-9b116f068419\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-b77jh" Oct 30 21:02:28.667906 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:02:28.667892 5378 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7bfe56ea-dd63-441e-a5c1-9b116f068419-workload-socket\") pod \"maas-default-gateway-openshift-default-769df7846c-b77jh\" (UID: \"7bfe56ea-dd63-441e-a5c1-9b116f068419\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-b77jh" Oct 30 21:02:28.668070 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:02:28.667988 5378 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7bfe56ea-dd63-441e-a5c1-9b116f068419-workload-certs\") pod \"maas-default-gateway-openshift-default-769df7846c-b77jh\" (UID: \"7bfe56ea-dd63-441e-a5c1-9b116f068419\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-b77jh" Oct 30 21:02:28.668070 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:02:28.668010 5378 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7bfe56ea-dd63-441e-a5c1-9b116f068419-credential-socket\") pod \"maas-default-gateway-openshift-default-769df7846c-b77jh\" (UID: \"7bfe56ea-dd63-441e-a5c1-9b116f068419\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-b77jh" Oct 30 21:02:28.668070 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:02:28.668052 5378 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7bfe56ea-dd63-441e-a5c1-9b116f068419-istio-data\") pod \"maas-default-gateway-openshift-default-769df7846c-b77jh\" (UID: \"7bfe56ea-dd63-441e-a5c1-9b116f068419\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-b77jh" Oct 30 21:02:28.668371 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:02:28.668351 5378 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7bfe56ea-dd63-441e-a5c1-9b116f068419-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-769df7846c-b77jh\" (UID: \"7bfe56ea-dd63-441e-a5c1-9b116f068419\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-b77jh" Oct 30 21:02:28.669833 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:02:28.669808 5378 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7bfe56ea-dd63-441e-a5c1-9b116f068419-istio-envoy\") pod \"maas-default-gateway-openshift-default-769df7846c-b77jh\" (UID: \"7bfe56ea-dd63-441e-a5c1-9b116f068419\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-b77jh" Oct 30 21:02:28.670298 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:02:28.670277 5378 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7bfe56ea-dd63-441e-a5c1-9b116f068419-istio-podinfo\") pod \"maas-default-gateway-openshift-default-769df7846c-b77jh\" (UID: \"7bfe56ea-dd63-441e-a5c1-9b116f068419\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-b77jh" Oct 30 21:02:28.732875 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:02:28.732816 5378 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7bfe56ea-dd63-441e-a5c1-9b116f068419-istio-token\") pod \"maas-default-gateway-openshift-default-769df7846c-b77jh\" (UID: \"7bfe56ea-dd63-441e-a5c1-9b116f068419\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-b77jh" Oct 30 21:02:28.733039 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:02:28.733015 5378 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2rjhh\" (UniqueName: \"kubernetes.io/projected/7bfe56ea-dd63-441e-a5c1-9b116f068419-kube-api-access-2rjhh\") pod \"maas-default-gateway-openshift-default-769df7846c-b77jh\" (UID: \"7bfe56ea-dd63-441e-a5c1-9b116f068419\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-b77jh" Oct 30 21:02:28.801436 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:02:28.801406 5378 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-b77jh" Oct 30 21:02:29.297459 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:02:29.297423 5378 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-769df7846c-b77jh"] Oct 30 21:02:29.297710 ip-10-0-138-149 kubenswrapper[5378]: W1030 21:02:29.297688 5378 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7bfe56ea_dd63_441e_a5c1_9b116f068419.slice/crio-f7e9192de7bbe4569bf25aa3ed298ce5c4a9d22bbd225dbfd7ea25c87780a3f6 WatchSource:0}: Error finding container f7e9192de7bbe4569bf25aa3ed298ce5c4a9d22bbd225dbfd7ea25c87780a3f6: Status 404 returned error can't find the container with id f7e9192de7bbe4569bf25aa3ed298ce5c4a9d22bbd225dbfd7ea25c87780a3f6 Oct 30 21:02:29.300223 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:02:29.300189 5378 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 21:02:29.300325 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:02:29.300254 5378 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 21:02:29.300325 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:02:29.300283 5378 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 21:02:29.561058 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:02:29.560984 5378 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-bc88b34e-67f64959c8-5kndg" Oct 30 21:02:29.593982 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:02:29.593951 5378 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-b77jh" event={"ID":"7bfe56ea-dd63-441e-a5c1-9b116f068419","Type":"ContainerStarted","Data":"3dd624ef0bf385fcbf1445ed0e49b444edc0d889d3f0d9b3dcf8a159308c4ecc"} Oct 30 21:02:29.594154 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:02:29.593998 5378 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-b77jh" event={"ID":"7bfe56ea-dd63-441e-a5c1-9b116f068419","Type":"ContainerStarted","Data":"f7e9192de7bbe4569bf25aa3ed298ce5c4a9d22bbd225dbfd7ea25c87780a3f6"} Oct 30 21:02:29.808134 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:02:29.808087 5378 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-b77jh" Oct 30 21:02:29.810741 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:02:29.810689 5378 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-b77jh" podStartSLOduration=1.810677043 podStartE2EDuration="1.810677043s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:02:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:02:29.80840314 +0000 UTC m=+1440.236517072" watchObservedRunningTime="2026-10-30 21:02:29.810677043 +0000 UTC m=+1440.238790962" Oct 30 21:02:30.684209 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:02:30.684169 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:02:30.827333 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:02:30.827293 5378 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-769df7846c-b77jh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.105:15021/healthz/ready\": dial tcp 10.133.0.105:15021: connect: connection refused" start-of-body= Oct 30 21:02:30.841063 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:02:30.837717 5378 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-b77jh" podUID="7bfe56ea-dd63-441e-a5c1-9b116f068419" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.105:15021/healthz/ready\": dial tcp 10.133.0.105:15021: connect: connection refused" Oct 30 21:02:31.802584 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:02:31.802539 5378 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-769df7846c-b77jh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.105:15021/healthz/ready\": dial tcp 10.133.0.105:15021: connect: connection refused" start-of-body= Oct 30 21:02:31.803006 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:02:31.802628 5378 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-b77jh" podUID="7bfe56ea-dd63-441e-a5c1-9b116f068419" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.105:15021/healthz/ready\": dial tcp 10.133.0.105:15021: connect: connection refused" Oct 30 21:02:32.801737 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:02:32.801702 5378 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-769df7846c-b77jh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.105:15021/healthz/ready\": dial tcp 10.133.0.105:15021: connect: connection refused" start-of-body= Oct 30 21:02:32.801919 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:02:32.801761 5378 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-b77jh" podUID="7bfe56ea-dd63-441e-a5c1-9b116f068419" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.105:15021/healthz/ready\": dial tcp 10.133.0.105:15021: connect: connection refused" Oct 30 21:02:33.802712 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:02:33.802681 5378 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-769df7846c-b77jh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.105:15021/healthz/ready\": dial tcp 10.133.0.105:15021: connect: connection refused" start-of-body= Oct 30 21:02:33.803101 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:02:33.802730 5378 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-b77jh" podUID="7bfe56ea-dd63-441e-a5c1-9b116f068419" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.105:15021/healthz/ready\": dial tcp 10.133.0.105:15021: connect: connection refused" Oct 30 21:02:34.802554 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:02:34.802512 5378 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-769df7846c-b77jh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.105:15021/healthz/ready\": dial tcp 10.133.0.105:15021: connect: connection refused" start-of-body= Oct 30 21:02:34.802765 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:02:34.802592 5378 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-b77jh" podUID="7bfe56ea-dd63-441e-a5c1-9b116f068419" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.105:15021/healthz/ready\": dial tcp 10.133.0.105:15021: connect: connection refused" Oct 30 21:02:35.578151 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:02:35.578112 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:02:35.802719 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:02:35.802683 5378 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-769df7846c-b77jh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.105:15021/healthz/ready\": dial tcp 10.133.0.105:15021: connect: connection refused" start-of-body= Oct 30 21:02:35.802996 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:02:35.802961 5378 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-b77jh" podUID="7bfe56ea-dd63-441e-a5c1-9b116f068419" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.105:15021/healthz/ready\": dial tcp 10.133.0.105:15021: connect: connection refused" Oct 30 21:02:36.801732 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:02:36.801693 5378 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-769df7846c-b77jh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.105:15021/healthz/ready\": dial tcp 10.133.0.105:15021: connect: connection refused" start-of-body= Oct 30 21:02:36.801928 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:02:36.801758 5378 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-b77jh" podUID="7bfe56ea-dd63-441e-a5c1-9b116f068419" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.105:15021/healthz/ready\": dial tcp 10.133.0.105:15021: connect: connection refused" Oct 30 21:02:38.802864 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:02:38.802765 5378 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-769df7846c-b77jh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.105:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 21:02:38.808844 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:02:38.808806 5378 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-b77jh" podUID="7bfe56ea-dd63-441e-a5c1-9b116f068419" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.105:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 21:02:39.015600 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:02:39.014578 5378 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-b77jh" Oct 30 21:02:39.018600 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:02:39.016109 5378 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-b77jh" Oct 30 21:02:39.040589 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:02:39.038661 5378 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-b77jh" Oct 30 21:02:40.567961 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:02:40.567928 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:02:44.192745 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:02:44.192715 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:02:45.016379 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:02:45.016332 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:02:45.157372 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:02:45.157342 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:02:48.840399 ip-10-0-138-149 kubenswrapper[5378]: E1030 21:02:48.840363 5378 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q6mx7" containerName="extractor" Oct 30 21:02:50.071865 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:02:50.071831 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:02:50.214460 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:02:50.214416 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:02:50.583245 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:02:50.583214 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:02:55.838360 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:02:55.838324 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:03:00.987410 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:03:00.987372 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:03:01.143693 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:03:01.143654 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:03:02.521236 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:03:02.521194 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:03:02.794822 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:03:02.794723 5378 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-f55142ef-77464b6464-z54fm"] Oct 30 21:03:02.912797 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:03:02.912757 5378 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-f55142ef-5455c459ff-zsv57"] Oct 30 21:03:03.149036 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:03:03.149005 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:03:03.351068 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:03:03.351035 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:03:03.852185 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:03:03.852151 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:03:05.564542 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:03:05.564506 5378 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-83851760-7c6fd5bb59-rdftz"] Oct 30 21:03:05.800440 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:03:05.800409 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:03:06.010630 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:03:05.998114 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:03:06.188289 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:03:06.188252 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:03:07.795064 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:03:07.794997 5378 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-migrate-f55142ef-77464b6464-z54fm" podUID="65353005-fa70-4a47-a789-27024c4c0a28" containerName="payload-processing" containerID="cri-o://149b9383f1622944684f002bf041a4cc9c10d69ef1d0f7c4e8be491ae262b126" gracePeriod=25 Oct 30 21:03:07.916323 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:03:07.916251 5378 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-migrate-f55142ef-5455c459ff-zsv57" podUID="bdcf0b01-1a1b-4b24-a8fb-9bf730bd0636" containerName="payload-pre-processing" containerID="cri-o://3c3318cd23cd82ceb105d3c720d8ad09a3b6e900685d9179d5c9233851e6acd6" gracePeriod=25 Oct 30 21:03:08.110928 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:03:08.110893 5378 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-f55142ef-77464b6464-z54fm" Oct 30 21:03:08.118120 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:03:08.118090 5378 generic.go:356] "Generic (PLEG): container finished" podID="bdcf0b01-1a1b-4b24-a8fb-9bf730bd0636" containerID="3c3318cd23cd82ceb105d3c720d8ad09a3b6e900685d9179d5c9233851e6acd6" exitCode=0 Oct 30 21:03:08.118257 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:03:08.118170 5378 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-f55142ef-5455c459ff-zsv57" event={"ID":"bdcf0b01-1a1b-4b24-a8fb-9bf730bd0636","Type":"ContainerDied","Data":"3c3318cd23cd82ceb105d3c720d8ad09a3b6e900685d9179d5c9233851e6acd6"} Oct 30 21:03:08.119839 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:03:08.119812 5378 generic.go:356] "Generic (PLEG): container finished" podID="65353005-fa70-4a47-a789-27024c4c0a28" containerID="149b9383f1622944684f002bf041a4cc9c10d69ef1d0f7c4e8be491ae262b126" exitCode=0 Oct 30 21:03:08.120006 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:03:08.119987 5378 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-f55142ef-77464b6464-z54fm" event={"ID":"65353005-fa70-4a47-a789-27024c4c0a28","Type":"ContainerDied","Data":"149b9383f1622944684f002bf041a4cc9c10d69ef1d0f7c4e8be491ae262b126"} Oct 30 21:03:08.120078 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:03:08.120036 5378 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-f55142ef-77464b6464-z54fm" event={"ID":"65353005-fa70-4a47-a789-27024c4c0a28","Type":"ContainerDied","Data":"d71bdf330b9b68232164f4660227a4137105e1b22991e9d1dd7da0e810ea85c8"} Oct 30 21:03:08.120078 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:03:08.120069 5378 scope.go:122] "RemoveContainer" containerID="149b9383f1622944684f002bf041a4cc9c10d69ef1d0f7c4e8be491ae262b126" Oct 30 21:03:08.120237 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:03:08.120222 5378 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-f55142ef-77464b6464-z54fm" Oct 30 21:03:08.132040 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:03:08.132021 5378 scope.go:122] "RemoveContainer" containerID="149b9383f1622944684f002bf041a4cc9c10d69ef1d0f7c4e8be491ae262b126" Oct 30 21:03:08.132462 ip-10-0-138-149 kubenswrapper[5378]: E1030 21:03:08.132432 5378 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"149b9383f1622944684f002bf041a4cc9c10d69ef1d0f7c4e8be491ae262b126\": container with ID starting with 149b9383f1622944684f002bf041a4cc9c10d69ef1d0f7c4e8be491ae262b126 not found: ID does not exist" containerID="149b9383f1622944684f002bf041a4cc9c10d69ef1d0f7c4e8be491ae262b126" Oct 30 21:03:08.132613 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:03:08.132472 5378 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"149b9383f1622944684f002bf041a4cc9c10d69ef1d0f7c4e8be491ae262b126"} err="failed to get container status \"149b9383f1622944684f002bf041a4cc9c10d69ef1d0f7c4e8be491ae262b126\": rpc error: code = NotFound desc = could not find container \"149b9383f1622944684f002bf041a4cc9c10d69ef1d0f7c4e8be491ae262b126\": container with ID starting with 149b9383f1622944684f002bf041a4cc9c10d69ef1d0f7c4e8be491ae262b126 not found: ID does not exist" Oct 30 21:03:08.211092 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:03:08.211069 5378 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-f55142ef-5455c459ff-zsv57" Oct 30 21:03:08.214436 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:03:08.214412 5378 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/65353005-fa70-4a47-a789-27024c4c0a28-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/65353005-fa70-4a47-a789-27024c4c0a28-plugins-config-volume\") pod \"65353005-fa70-4a47-a789-27024c4c0a28\" (UID: \"65353005-fa70-4a47-a789-27024c4c0a28\") " Oct 30 21:03:08.214546 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:03:08.214507 5378 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/65353005-fa70-4a47-a789-27024c4c0a28-kube-api-access-xp57m\" (UniqueName: \"kubernetes.io/projected/65353005-fa70-4a47-a789-27024c4c0a28-kube-api-access-xp57m\") pod \"65353005-fa70-4a47-a789-27024c4c0a28\" (UID: \"65353005-fa70-4a47-a789-27024c4c0a28\") " Oct 30 21:03:08.214790 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:03:08.214767 5378 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/65353005-fa70-4a47-a789-27024c4c0a28-plugins-config-volume" pod "65353005-fa70-4a47-a789-27024c4c0a28" (UID: "65353005-fa70-4a47-a789-27024c4c0a28"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:03:08.215062 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:03:08.215046 5378 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/65353005-fa70-4a47-a789-27024c4c0a28-plugins-config-volume\") on node \"ip-10-0-138-149.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:03:08.216411 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:03:08.216390 5378 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/65353005-fa70-4a47-a789-27024c4c0a28-kube-api-access-xp57m" pod "65353005-fa70-4a47-a789-27024c4c0a28" (UID: "65353005-fa70-4a47-a789-27024c4c0a28"). InnerVolumeSpecName "kube-api-access-xp57m". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:03:08.316068 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:03:08.316034 5378 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bdcf0b01-1a1b-4b24-a8fb-9bf730bd0636-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bdcf0b01-1a1b-4b24-a8fb-9bf730bd0636-plugins-config-volume\") pod \"bdcf0b01-1a1b-4b24-a8fb-9bf730bd0636\" (UID: \"bdcf0b01-1a1b-4b24-a8fb-9bf730bd0636\") " Oct 30 21:03:08.316231 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:03:08.316156 5378 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bdcf0b01-1a1b-4b24-a8fb-9bf730bd0636-kube-api-access-fs872\" (UniqueName: \"kubernetes.io/projected/bdcf0b01-1a1b-4b24-a8fb-9bf730bd0636-kube-api-access-fs872\") pod \"bdcf0b01-1a1b-4b24-a8fb-9bf730bd0636\" (UID: \"bdcf0b01-1a1b-4b24-a8fb-9bf730bd0636\") " Oct 30 21:03:08.316405 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:03:08.316380 5378 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bdcf0b01-1a1b-4b24-a8fb-9bf730bd0636-plugins-config-volume" pod "bdcf0b01-1a1b-4b24-a8fb-9bf730bd0636" (UID: "bdcf0b01-1a1b-4b24-a8fb-9bf730bd0636"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:03:08.316885 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:03:08.316868 5378 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xp57m\" (UniqueName: \"kubernetes.io/projected/65353005-fa70-4a47-a789-27024c4c0a28-kube-api-access-xp57m\") on node \"ip-10-0-138-149.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:03:08.316941 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:03:08.316887 5378 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bdcf0b01-1a1b-4b24-a8fb-9bf730bd0636-plugins-config-volume\") on node \"ip-10-0-138-149.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:03:08.318344 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:03:08.318324 5378 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bdcf0b01-1a1b-4b24-a8fb-9bf730bd0636-kube-api-access-fs872" pod "bdcf0b01-1a1b-4b24-a8fb-9bf730bd0636" (UID: "bdcf0b01-1a1b-4b24-a8fb-9bf730bd0636"). InnerVolumeSpecName "kube-api-access-fs872". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:03:08.418099 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:03:08.418071 5378 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fs872\" (UniqueName: \"kubernetes.io/projected/bdcf0b01-1a1b-4b24-a8fb-9bf730bd0636-kube-api-access-fs872\") on node \"ip-10-0-138-149.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:03:08.544390 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:03:08.544353 5378 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-f55142ef-77464b6464-z54fm"] Oct 30 21:03:08.552070 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:03:08.552040 5378 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-f55142ef-77464b6464-z54fm"] Oct 30 21:03:08.789886 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:03:08.789804 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:03:09.130943 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:03:09.130902 5378 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-f55142ef-5455c459ff-zsv57" Oct 30 21:03:09.130943 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:03:09.130935 5378 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-f55142ef-5455c459ff-zsv57" event={"ID":"bdcf0b01-1a1b-4b24-a8fb-9bf730bd0636","Type":"ContainerDied","Data":"36653640a3f0637958521e89f8c173ddb94fc8f137cb3e3e4ab31cf571855c49"} Oct 30 21:03:09.131436 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:03:09.130994 5378 scope.go:122] "RemoveContainer" containerID="3c3318cd23cd82ceb105d3c720d8ad09a3b6e900685d9179d5c9233851e6acd6" Oct 30 21:03:09.298366 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:03:09.298335 5378 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-f55142ef-5455c459ff-zsv57"] Oct 30 21:03:09.348594 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:03:09.348534 5378 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-f55142ef-5455c459ff-zsv57"] Oct 30 21:03:09.744738 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:03:09.744706 5378 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="65353005-fa70-4a47-a789-27024c4c0a28" path="/var/lib/kubelet/pods/65353005-fa70-4a47-a789-27024c4c0a28/volumes" Oct 30 21:03:09.745152 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:03:09.745138 5378 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bdcf0b01-1a1b-4b24-a8fb-9bf730bd0636" path="/var/lib/kubelet/pods/bdcf0b01-1a1b-4b24-a8fb-9bf730bd0636/volumes" Oct 30 21:03:10.564947 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:03:10.564909 5378 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ait-83851760-7c6fd5bb59-rdftz" podUID="e9a8452e-59e0-47f0-8a96-098c18bc6424" containerName="payload-pre-processing" containerID="cri-o://6b95122cfd411e427d9db40f6b261e96a0e02a1a6e26a1e6f10da156d47f8f14" gracePeriod=25 Oct 30 21:03:10.850795 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:03:10.850769 5378 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-83851760-7c6fd5bb59-rdftz" Oct 30 21:03:10.959199 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:03:10.959171 5378 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e9a8452e-59e0-47f0-8a96-098c18bc6424-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e9a8452e-59e0-47f0-8a96-098c18bc6424-plugins-config-volume\") pod \"e9a8452e-59e0-47f0-8a96-098c18bc6424\" (UID: \"e9a8452e-59e0-47f0-8a96-098c18bc6424\") " Oct 30 21:03:10.959359 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:03:10.959213 5378 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e9a8452e-59e0-47f0-8a96-098c18bc6424-kube-api-access-9rpw4\" (UniqueName: \"kubernetes.io/projected/e9a8452e-59e0-47f0-8a96-098c18bc6424-kube-api-access-9rpw4\") pod \"e9a8452e-59e0-47f0-8a96-098c18bc6424\" (UID: \"e9a8452e-59e0-47f0-8a96-098c18bc6424\") " Oct 30 21:03:10.959533 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:03:10.959510 5378 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e9a8452e-59e0-47f0-8a96-098c18bc6424-plugins-config-volume" pod "e9a8452e-59e0-47f0-8a96-098c18bc6424" (UID: "e9a8452e-59e0-47f0-8a96-098c18bc6424"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:03:10.959763 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:03:10.959749 5378 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e9a8452e-59e0-47f0-8a96-098c18bc6424-plugins-config-volume\") on node \"ip-10-0-138-149.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:03:10.961256 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:03:10.961238 5378 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e9a8452e-59e0-47f0-8a96-098c18bc6424-kube-api-access-9rpw4" pod "e9a8452e-59e0-47f0-8a96-098c18bc6424" (UID: "e9a8452e-59e0-47f0-8a96-098c18bc6424"). InnerVolumeSpecName "kube-api-access-9rpw4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:03:11.060932 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:03:11.060902 5378 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9rpw4\" (UniqueName: \"kubernetes.io/projected/e9a8452e-59e0-47f0-8a96-098c18bc6424-kube-api-access-9rpw4\") on node \"ip-10-0-138-149.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:03:11.082327 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:03:11.082253 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:03:11.237633 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:03:11.237553 5378 generic.go:356] "Generic (PLEG): container finished" podID="e9a8452e-59e0-47f0-8a96-098c18bc6424" containerID="6b95122cfd411e427d9db40f6b261e96a0e02a1a6e26a1e6f10da156d47f8f14" exitCode=0 Oct 30 21:03:11.237823 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:03:11.237706 5378 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-83851760-7c6fd5bb59-rdftz" event={"ID":"e9a8452e-59e0-47f0-8a96-098c18bc6424","Type":"ContainerDied","Data":"6b95122cfd411e427d9db40f6b261e96a0e02a1a6e26a1e6f10da156d47f8f14"} Oct 30 21:03:11.237823 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:03:11.237795 5378 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-83851760-7c6fd5bb59-rdftz" event={"ID":"e9a8452e-59e0-47f0-8a96-098c18bc6424","Type":"ContainerDied","Data":"a134085a3f8c876b66d49c8252116ee3403277485a6934911e8c8f76bce73a58"} Oct 30 21:03:11.237944 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:03:11.237825 5378 scope.go:122] "RemoveContainer" containerID="6b95122cfd411e427d9db40f6b261e96a0e02a1a6e26a1e6f10da156d47f8f14" Oct 30 21:03:11.238011 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:03:11.237971 5378 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-83851760-7c6fd5bb59-rdftz" Oct 30 21:03:11.252501 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:03:11.252480 5378 scope.go:122] "RemoveContainer" containerID="6b95122cfd411e427d9db40f6b261e96a0e02a1a6e26a1e6f10da156d47f8f14" Oct 30 21:03:11.252795 ip-10-0-138-149 kubenswrapper[5378]: E1030 21:03:11.252773 5378 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6b95122cfd411e427d9db40f6b261e96a0e02a1a6e26a1e6f10da156d47f8f14\": container with ID starting with 6b95122cfd411e427d9db40f6b261e96a0e02a1a6e26a1e6f10da156d47f8f14 not found: ID does not exist" containerID="6b95122cfd411e427d9db40f6b261e96a0e02a1a6e26a1e6f10da156d47f8f14" Oct 30 21:03:11.252846 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:03:11.252803 5378 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6b95122cfd411e427d9db40f6b261e96a0e02a1a6e26a1e6f10da156d47f8f14"} err="failed to get container status \"6b95122cfd411e427d9db40f6b261e96a0e02a1a6e26a1e6f10da156d47f8f14\": rpc error: code = NotFound desc = could not find container \"6b95122cfd411e427d9db40f6b261e96a0e02a1a6e26a1e6f10da156d47f8f14\": container with ID starting with 6b95122cfd411e427d9db40f6b261e96a0e02a1a6e26a1e6f10da156d47f8f14 not found: ID does not exist" Oct 30 21:03:11.406987 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:03:11.406955 5378 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-83851760-7c6fd5bb59-rdftz"] Oct 30 21:03:11.457486 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:03:11.457453 5378 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-83851760-7c6fd5bb59-rdftz"] Oct 30 21:03:11.742333 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:03:11.742254 5378 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e9a8452e-59e0-47f0-8a96-098c18bc6424" path="/var/lib/kubelet/pods/e9a8452e-59e0-47f0-8a96-098c18bc6424/volumes" Oct 30 21:03:12.100042 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:03:12.100013 5378 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-r9g59"] Oct 30 21:03:15.944591 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:03:15.944198 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:03:18.841765 ip-10-0-138-149 kubenswrapper[5378]: E1030 21:03:18.841727 5378 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q6mx7" containerName="extractor" Oct 30 21:03:20.889027 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:03:20.888988 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:03:26.157952 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:03:26.157919 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:03:29.917932 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:03:29.917899 5378 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-7x9jg_c2ff359c-cf92-4a5b-8610-f6eb41acbe73/ovn-acl-logging/0.log" Oct 30 21:03:29.924783 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:03:29.924760 5378 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-7x9jg_c2ff359c-cf92-4a5b-8610-f6eb41acbe73/ovn-acl-logging/0.log" Oct 30 21:03:31.262044 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:03:31.262014 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:03:35.518755 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:03:35.518725 5378 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-f55142ef-75c5db7868-2kvrk"] Oct 30 21:03:35.519161 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:03:35.518952 5378 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-f55142ef-75c5db7868-2kvrk" podUID="3c485363-d417-4168-a210-ca908192b1d6" containerName="maas-api" containerID="cri-o://710bce613fbe90cbec349bd5df35910e504674ea1411a61f25be704b2f5493a0" gracePeriod=30 Oct 30 21:03:35.613398 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:03:35.613369 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:03:35.934843 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:03:35.934818 5378 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-f55142ef-75c5db7868-2kvrk" Oct 30 21:03:36.028591 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:03:36.003357 5378 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3c485363-d417-4168-a210-ca908192b1d6-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3c485363-d417-4168-a210-ca908192b1d6-openshift-service-ca\") pod \"3c485363-d417-4168-a210-ca908192b1d6\" (UID: \"3c485363-d417-4168-a210-ca908192b1d6\") " Oct 30 21:03:36.028591 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:03:36.003628 5378 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3c485363-d417-4168-a210-ca908192b1d6-kube-api-access-jldgd\" (UniqueName: \"kubernetes.io/projected/3c485363-d417-4168-a210-ca908192b1d6-kube-api-access-jldgd\") pod \"3c485363-d417-4168-a210-ca908192b1d6\" (UID: \"3c485363-d417-4168-a210-ca908192b1d6\") " Oct 30 21:03:36.028591 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:03:36.003708 5378 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3c485363-d417-4168-a210-ca908192b1d6-metrics-certs\" (UniqueName: \"kubernetes.io/secret/3c485363-d417-4168-a210-ca908192b1d6-metrics-certs\") pod \"3c485363-d417-4168-a210-ca908192b1d6\" (UID: \"3c485363-d417-4168-a210-ca908192b1d6\") " Oct 30 21:03:36.028591 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:03:36.003969 5378 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3c485363-d417-4168-a210-ca908192b1d6-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3c485363-d417-4168-a210-ca908192b1d6-maas-api-tls\") pod \"3c485363-d417-4168-a210-ca908192b1d6\" (UID: \"3c485363-d417-4168-a210-ca908192b1d6\") " Oct 30 21:03:36.028591 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:03:36.018032 5378 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3c485363-d417-4168-a210-ca908192b1d6-openshift-service-ca" pod "3c485363-d417-4168-a210-ca908192b1d6" (UID: "3c485363-d417-4168-a210-ca908192b1d6"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:03:36.043256 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:03:36.041711 5378 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3c485363-d417-4168-a210-ca908192b1d6-maas-api-tls" pod "3c485363-d417-4168-a210-ca908192b1d6" (UID: "3c485363-d417-4168-a210-ca908192b1d6"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:03:36.057784 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:03:36.057744 5378 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3c485363-d417-4168-a210-ca908192b1d6-metrics-certs" pod "3c485363-d417-4168-a210-ca908192b1d6" (UID: "3c485363-d417-4168-a210-ca908192b1d6"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:03:36.060151 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:03:36.060104 5378 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3c485363-d417-4168-a210-ca908192b1d6-kube-api-access-jldgd" pod "3c485363-d417-4168-a210-ca908192b1d6" (UID: "3c485363-d417-4168-a210-ca908192b1d6"). InnerVolumeSpecName "kube-api-access-jldgd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:03:36.136597 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:03:36.131948 5378 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jldgd\" (UniqueName: \"kubernetes.io/projected/3c485363-d417-4168-a210-ca908192b1d6-kube-api-access-jldgd\") on node \"ip-10-0-138-149.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:03:36.136597 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:03:36.131988 5378 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3c485363-d417-4168-a210-ca908192b1d6-metrics-certs\") on node \"ip-10-0-138-149.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:03:36.136597 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:03:36.132040 5378 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3c485363-d417-4168-a210-ca908192b1d6-maas-api-tls\") on node \"ip-10-0-138-149.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:03:36.136597 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:03:36.132055 5378 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3c485363-d417-4168-a210-ca908192b1d6-openshift-service-ca\") on node \"ip-10-0-138-149.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:03:36.441588 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:03:36.439867 5378 generic.go:356] "Generic (PLEG): container finished" podID="3c485363-d417-4168-a210-ca908192b1d6" containerID="710bce613fbe90cbec349bd5df35910e504674ea1411a61f25be704b2f5493a0" exitCode=0 Oct 30 21:03:36.441588 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:03:36.439909 5378 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-f55142ef-75c5db7868-2kvrk" event={"ID":"3c485363-d417-4168-a210-ca908192b1d6","Type":"ContainerDied","Data":"710bce613fbe90cbec349bd5df35910e504674ea1411a61f25be704b2f5493a0"} Oct 30 21:03:36.441588 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:03:36.439976 5378 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-f55142ef-75c5db7868-2kvrk" event={"ID":"3c485363-d417-4168-a210-ca908192b1d6","Type":"ContainerDied","Data":"5ecb900f0531ae15f95f509485113476391ae660fd7bdfbcac2588ca25a52117"} Oct 30 21:03:36.441588 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:03:36.440008 5378 scope.go:122] "RemoveContainer" containerID="710bce613fbe90cbec349bd5df35910e504674ea1411a61f25be704b2f5493a0" Oct 30 21:03:36.441588 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:03:36.440095 5378 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-f55142ef-75c5db7868-2kvrk" Oct 30 21:03:36.450665 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:03:36.450644 5378 scope.go:122] "RemoveContainer" containerID="710bce613fbe90cbec349bd5df35910e504674ea1411a61f25be704b2f5493a0" Oct 30 21:03:36.450955 ip-10-0-138-149 kubenswrapper[5378]: E1030 21:03:36.450934 5378 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"710bce613fbe90cbec349bd5df35910e504674ea1411a61f25be704b2f5493a0\": container with ID starting with 710bce613fbe90cbec349bd5df35910e504674ea1411a61f25be704b2f5493a0 not found: ID does not exist" containerID="710bce613fbe90cbec349bd5df35910e504674ea1411a61f25be704b2f5493a0" Oct 30 21:03:36.451033 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:03:36.450965 5378 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"710bce613fbe90cbec349bd5df35910e504674ea1411a61f25be704b2f5493a0"} err="failed to get container status \"710bce613fbe90cbec349bd5df35910e504674ea1411a61f25be704b2f5493a0\": rpc error: code = NotFound desc = could not find container \"710bce613fbe90cbec349bd5df35910e504674ea1411a61f25be704b2f5493a0\": container with ID starting with 710bce613fbe90cbec349bd5df35910e504674ea1411a61f25be704b2f5493a0 not found: ID does not exist" Oct 30 21:03:36.610911 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:03:36.610877 5378 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-f55142ef-75c5db7868-2kvrk"] Oct 30 21:03:36.661477 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:03:36.661445 5378 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-f55142ef-75c5db7868-2kvrk"] Oct 30 21:03:36.700324 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:03:36.700254 5378 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-83851760-854c79488f-xfj9r"] Oct 30 21:03:36.700467 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:03:36.700447 5378 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-83851760-854c79488f-xfj9r" podUID="218105d7-352a-40b6-8940-274d2a0a7720" containerName="maas-api" containerID="cri-o://fb4311eb069dccf57b23c5f58582cc96d98c61d689d546eea86f46f32775e1fd" gracePeriod=30 Oct 30 21:03:36.791260 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:03:36.791229 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:03:37.084353 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:03:37.084312 5378 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-83851760-854c79488f-xfj9r" Oct 30 21:03:37.304495 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:03:37.302511 5378 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/218105d7-352a-40b6-8940-274d2a0a7720-kube-api-access-2kwd4\" (UniqueName: \"kubernetes.io/projected/218105d7-352a-40b6-8940-274d2a0a7720-kube-api-access-2kwd4\") pod \"218105d7-352a-40b6-8940-274d2a0a7720\" (UID: \"218105d7-352a-40b6-8940-274d2a0a7720\") " Oct 30 21:03:37.304495 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:03:37.302673 5378 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/218105d7-352a-40b6-8940-274d2a0a7720-metrics-certs\" (UniqueName: \"kubernetes.io/secret/218105d7-352a-40b6-8940-274d2a0a7720-metrics-certs\") pod \"218105d7-352a-40b6-8940-274d2a0a7720\" (UID: \"218105d7-352a-40b6-8940-274d2a0a7720\") " Oct 30 21:03:37.304495 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:03:37.302775 5378 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/218105d7-352a-40b6-8940-274d2a0a7720-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/218105d7-352a-40b6-8940-274d2a0a7720-openshift-service-ca\") pod \"218105d7-352a-40b6-8940-274d2a0a7720\" (UID: \"218105d7-352a-40b6-8940-274d2a0a7720\") " Oct 30 21:03:37.304495 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:03:37.303020 5378 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/218105d7-352a-40b6-8940-274d2a0a7720-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/218105d7-352a-40b6-8940-274d2a0a7720-maas-api-tls\") pod \"218105d7-352a-40b6-8940-274d2a0a7720\" (UID: \"218105d7-352a-40b6-8940-274d2a0a7720\") " Oct 30 21:03:37.306652 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:03:37.306298 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:03:37.308160 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:03:37.307801 5378 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/218105d7-352a-40b6-8940-274d2a0a7720-kube-api-access-2kwd4" pod "218105d7-352a-40b6-8940-274d2a0a7720" (UID: "218105d7-352a-40b6-8940-274d2a0a7720"). InnerVolumeSpecName "kube-api-access-2kwd4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:03:37.308160 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:03:37.308021 5378 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/218105d7-352a-40b6-8940-274d2a0a7720-maas-api-tls" pod "218105d7-352a-40b6-8940-274d2a0a7720" (UID: "218105d7-352a-40b6-8940-274d2a0a7720"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:03:37.317991 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:03:37.309492 5378 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/218105d7-352a-40b6-8940-274d2a0a7720-openshift-service-ca" pod "218105d7-352a-40b6-8940-274d2a0a7720" (UID: "218105d7-352a-40b6-8940-274d2a0a7720"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:03:37.317991 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:03:37.311314 5378 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/218105d7-352a-40b6-8940-274d2a0a7720-metrics-certs" pod "218105d7-352a-40b6-8940-274d2a0a7720" (UID: "218105d7-352a-40b6-8940-274d2a0a7720"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:03:37.424622 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:03:37.420611 5378 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/218105d7-352a-40b6-8940-274d2a0a7720-maas-api-tls\") on node \"ip-10-0-138-149.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:03:37.424622 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:03:37.420657 5378 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2kwd4\" (UniqueName: \"kubernetes.io/projected/218105d7-352a-40b6-8940-274d2a0a7720-kube-api-access-2kwd4\") on node \"ip-10-0-138-149.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:03:37.424622 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:03:37.421059 5378 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/218105d7-352a-40b6-8940-274d2a0a7720-metrics-certs\") on node \"ip-10-0-138-149.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:03:37.424622 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:03:37.421078 5378 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/218105d7-352a-40b6-8940-274d2a0a7720-openshift-service-ca\") on node \"ip-10-0-138-149.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:03:37.476939 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:03:37.476896 5378 generic.go:356] "Generic (PLEG): container finished" podID="218105d7-352a-40b6-8940-274d2a0a7720" containerID="fb4311eb069dccf57b23c5f58582cc96d98c61d689d546eea86f46f32775e1fd" exitCode=0 Oct 30 21:03:37.477125 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:03:37.476994 5378 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-83851760-854c79488f-xfj9r" event={"ID":"218105d7-352a-40b6-8940-274d2a0a7720","Type":"ContainerDied","Data":"fb4311eb069dccf57b23c5f58582cc96d98c61d689d546eea86f46f32775e1fd"} Oct 30 21:03:37.477125 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:03:37.477097 5378 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-83851760-854c79488f-xfj9r" event={"ID":"218105d7-352a-40b6-8940-274d2a0a7720","Type":"ContainerDied","Data":"a026f60d685fea530db3206f21d8c0db5fb26e8f6c17ebea6e95adde8b485ed6"} Oct 30 21:03:37.477239 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:03:37.477153 5378 scope.go:122] "RemoveContainer" containerID="fb4311eb069dccf57b23c5f58582cc96d98c61d689d546eea86f46f32775e1fd" Oct 30 21:03:37.477449 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:03:37.477340 5378 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-83851760-854c79488f-xfj9r" Oct 30 21:03:37.502773 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:03:37.502745 5378 scope.go:122] "RemoveContainer" containerID="fb4311eb069dccf57b23c5f58582cc96d98c61d689d546eea86f46f32775e1fd" Oct 30 21:03:37.503146 ip-10-0-138-149 kubenswrapper[5378]: E1030 21:03:37.503125 5378 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fb4311eb069dccf57b23c5f58582cc96d98c61d689d546eea86f46f32775e1fd\": container with ID starting with fb4311eb069dccf57b23c5f58582cc96d98c61d689d546eea86f46f32775e1fd not found: ID does not exist" containerID="fb4311eb069dccf57b23c5f58582cc96d98c61d689d546eea86f46f32775e1fd" Oct 30 21:03:37.503418 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:03:37.503398 5378 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fb4311eb069dccf57b23c5f58582cc96d98c61d689d546eea86f46f32775e1fd"} err="failed to get container status \"fb4311eb069dccf57b23c5f58582cc96d98c61d689d546eea86f46f32775e1fd\": rpc error: code = NotFound desc = could not find container \"fb4311eb069dccf57b23c5f58582cc96d98c61d689d546eea86f46f32775e1fd\": container with ID starting with fb4311eb069dccf57b23c5f58582cc96d98c61d689d546eea86f46f32775e1fd not found: ID does not exist" Oct 30 21:03:37.504364 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:03:37.504343 5378 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-bc88b34e-55b79996d-kfqbm"] Oct 30 21:03:37.537182 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:03:37.537160 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:03:37.653076 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:03:37.653043 5378 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-83851760-854c79488f-xfj9r"] Oct 30 21:03:37.663313 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:03:37.663286 5378 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-83851760-854c79488f-xfj9r"] Oct 30 21:03:37.743260 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:03:37.743230 5378 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="218105d7-352a-40b6-8940-274d2a0a7720" path="/var/lib/kubelet/pods/218105d7-352a-40b6-8940-274d2a0a7720/volumes" Oct 30 21:03:37.743764 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:03:37.743748 5378 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3c485363-d417-4168-a210-ca908192b1d6" path="/var/lib/kubelet/pods/3c485363-d417-4168-a210-ca908192b1d6/volumes" Oct 30 21:03:37.892591 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:03:37.867453 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:03:38.177588 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:03:38.177246 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:03:38.742422 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:03:38.742373 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:03:38.882317 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:03:38.882282 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:03:39.086874 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:03:39.086793 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:03:39.239269 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:03:39.239235 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:03:39.249346 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:03:39.249317 5378 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-2cf21005-openshift-default-5fdcf57455-c7jwb"] Oct 30 21:03:39.249966 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:03:39.249952 5378 cpu_manager.go:414] "containerMap: removing container" podUID="218105d7-352a-40b6-8940-274d2a0a7720" containerName="maas-api" Oct 30 21:03:39.250013 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:03:39.249968 5378 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="218105d7-352a-40b6-8940-274d2a0a7720" containerName="maas-api" Oct 30 21:03:39.250013 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:03:39.249982 5378 cpu_manager.go:414] "containerMap: removing container" podUID="3c485363-d417-4168-a210-ca908192b1d6" containerName="maas-api" Oct 30 21:03:39.250013 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:03:39.249987 5378 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3c485363-d417-4168-a210-ca908192b1d6" containerName="maas-api" Oct 30 21:03:39.250013 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:03:39.250003 5378 cpu_manager.go:414] "containerMap: removing container" podUID="bdcf0b01-1a1b-4b24-a8fb-9bf730bd0636" containerName="payload-pre-processing" Oct 30 21:03:39.250013 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:03:39.250008 5378 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bdcf0b01-1a1b-4b24-a8fb-9bf730bd0636" containerName="payload-pre-processing" Oct 30 21:03:39.250168 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:03:39.250025 5378 cpu_manager.go:414] "containerMap: removing container" podUID="e9a8452e-59e0-47f0-8a96-098c18bc6424" containerName="payload-pre-processing" Oct 30 21:03:39.250168 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:03:39.250031 5378 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e9a8452e-59e0-47f0-8a96-098c18bc6424" containerName="payload-pre-processing" Oct 30 21:03:39.250168 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:03:39.250066 5378 cpu_manager.go:414] "containerMap: removing container" podUID="65353005-fa70-4a47-a789-27024c4c0a28" containerName="payload-processing" Oct 30 21:03:39.250168 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:03:39.250071 5378 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="65353005-fa70-4a47-a789-27024c4c0a28" containerName="payload-processing" Oct 30 21:03:39.250168 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:03:39.250156 5378 memory_manager.go:365] "RemoveStaleState removing state" podUID="e9a8452e-59e0-47f0-8a96-098c18bc6424" containerName="payload-pre-processing" Oct 30 21:03:39.250309 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:03:39.250170 5378 memory_manager.go:365] "RemoveStaleState removing state" podUID="65353005-fa70-4a47-a789-27024c4c0a28" containerName="payload-processing" Oct 30 21:03:39.250309 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:03:39.250177 5378 memory_manager.go:365] "RemoveStaleState removing state" podUID="218105d7-352a-40b6-8940-274d2a0a7720" containerName="maas-api" Oct 30 21:03:39.250309 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:03:39.250184 5378 memory_manager.go:365] "RemoveStaleState removing state" podUID="3c485363-d417-4168-a210-ca908192b1d6" containerName="maas-api" Oct 30 21:03:39.250309 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:03:39.250191 5378 memory_manager.go:365] "RemoveStaleState removing state" podUID="bdcf0b01-1a1b-4b24-a8fb-9bf730bd0636" containerName="payload-pre-processing" Oct 30 21:03:39.255728 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:03:39.255711 5378 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-2cf21005-openshift-default-5fdcf57455-c7jwb" Oct 30 21:03:39.405590 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:03:39.403233 5378 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-2cf21005-openshift-default-5fdcf57455-c7jwb"] Oct 30 21:03:39.405590 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:03:39.404215 5378 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c7035375-d666-498d-b9e7-f699e1e86c99-workload-socket\") pod \"e2e-ait-2cf21005-openshift-default-5fdcf57455-c7jwb\" (UID: \"c7035375-d666-498d-b9e7-f699e1e86c99\") " pod="openshift-ingress/e2e-ait-2cf21005-openshift-default-5fdcf57455-c7jwb" Oct 30 21:03:39.405590 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:03:39.404252 5378 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c7035375-d666-498d-b9e7-f699e1e86c99-istiod-ca-cert\") pod \"e2e-ait-2cf21005-openshift-default-5fdcf57455-c7jwb\" (UID: \"c7035375-d666-498d-b9e7-f699e1e86c99\") " pod="openshift-ingress/e2e-ait-2cf21005-openshift-default-5fdcf57455-c7jwb" Oct 30 21:03:39.405590 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:03:39.404374 5378 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c7035375-d666-498d-b9e7-f699e1e86c99-istio-data\") pod \"e2e-ait-2cf21005-openshift-default-5fdcf57455-c7jwb\" (UID: \"c7035375-d666-498d-b9e7-f699e1e86c99\") " pod="openshift-ingress/e2e-ait-2cf21005-openshift-default-5fdcf57455-c7jwb" Oct 30 21:03:39.405590 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:03:39.404406 5378 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lcb5t\" (UniqueName: \"kubernetes.io/projected/c7035375-d666-498d-b9e7-f699e1e86c99-kube-api-access-lcb5t\") pod \"e2e-ait-2cf21005-openshift-default-5fdcf57455-c7jwb\" (UID: \"c7035375-d666-498d-b9e7-f699e1e86c99\") " pod="openshift-ingress/e2e-ait-2cf21005-openshift-default-5fdcf57455-c7jwb" Oct 30 21:03:39.405590 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:03:39.404446 5378 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c7035375-d666-498d-b9e7-f699e1e86c99-istio-podinfo\") pod \"e2e-ait-2cf21005-openshift-default-5fdcf57455-c7jwb\" (UID: \"c7035375-d666-498d-b9e7-f699e1e86c99\") " pod="openshift-ingress/e2e-ait-2cf21005-openshift-default-5fdcf57455-c7jwb" Oct 30 21:03:39.405590 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:03:39.404504 5378 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c7035375-d666-498d-b9e7-f699e1e86c99-workload-certs\") pod \"e2e-ait-2cf21005-openshift-default-5fdcf57455-c7jwb\" (UID: \"c7035375-d666-498d-b9e7-f699e1e86c99\") " pod="openshift-ingress/e2e-ait-2cf21005-openshift-default-5fdcf57455-c7jwb" Oct 30 21:03:39.405590 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:03:39.404557 5378 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c7035375-d666-498d-b9e7-f699e1e86c99-istio-token\") pod \"e2e-ait-2cf21005-openshift-default-5fdcf57455-c7jwb\" (UID: \"c7035375-d666-498d-b9e7-f699e1e86c99\") " pod="openshift-ingress/e2e-ait-2cf21005-openshift-default-5fdcf57455-c7jwb" Oct 30 21:03:39.405590 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:03:39.404832 5378 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c7035375-d666-498d-b9e7-f699e1e86c99-credential-socket\") pod \"e2e-ait-2cf21005-openshift-default-5fdcf57455-c7jwb\" (UID: \"c7035375-d666-498d-b9e7-f699e1e86c99\") " pod="openshift-ingress/e2e-ait-2cf21005-openshift-default-5fdcf57455-c7jwb" Oct 30 21:03:39.405590 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:03:39.404934 5378 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c7035375-d666-498d-b9e7-f699e1e86c99-istio-envoy\") pod \"e2e-ait-2cf21005-openshift-default-5fdcf57455-c7jwb\" (UID: \"c7035375-d666-498d-b9e7-f699e1e86c99\") " pod="openshift-ingress/e2e-ait-2cf21005-openshift-default-5fdcf57455-c7jwb" Oct 30 21:03:39.476000 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:03:39.460044 5378 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-2cf21005-openshift-default-dockercfg-jkdld\"" Oct 30 21:03:39.553594 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:03:39.546076 5378 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c7035375-d666-498d-b9e7-f699e1e86c99-istio-token\") pod \"e2e-ait-2cf21005-openshift-default-5fdcf57455-c7jwb\" (UID: \"c7035375-d666-498d-b9e7-f699e1e86c99\") " pod="openshift-ingress/e2e-ait-2cf21005-openshift-default-5fdcf57455-c7jwb" Oct 30 21:03:39.553594 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:03:39.546204 5378 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c7035375-d666-498d-b9e7-f699e1e86c99-credential-socket\") pod \"e2e-ait-2cf21005-openshift-default-5fdcf57455-c7jwb\" (UID: \"c7035375-d666-498d-b9e7-f699e1e86c99\") " pod="openshift-ingress/e2e-ait-2cf21005-openshift-default-5fdcf57455-c7jwb" Oct 30 21:03:39.553594 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:03:39.546300 5378 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c7035375-d666-498d-b9e7-f699e1e86c99-istio-envoy\") pod \"e2e-ait-2cf21005-openshift-default-5fdcf57455-c7jwb\" (UID: \"c7035375-d666-498d-b9e7-f699e1e86c99\") " pod="openshift-ingress/e2e-ait-2cf21005-openshift-default-5fdcf57455-c7jwb" Oct 30 21:03:39.553594 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:03:39.546364 5378 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c7035375-d666-498d-b9e7-f699e1e86c99-workload-socket\") pod \"e2e-ait-2cf21005-openshift-default-5fdcf57455-c7jwb\" (UID: \"c7035375-d666-498d-b9e7-f699e1e86c99\") " pod="openshift-ingress/e2e-ait-2cf21005-openshift-default-5fdcf57455-c7jwb" Oct 30 21:03:39.553594 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:03:39.546391 5378 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c7035375-d666-498d-b9e7-f699e1e86c99-istiod-ca-cert\") pod \"e2e-ait-2cf21005-openshift-default-5fdcf57455-c7jwb\" (UID: \"c7035375-d666-498d-b9e7-f699e1e86c99\") " pod="openshift-ingress/e2e-ait-2cf21005-openshift-default-5fdcf57455-c7jwb" Oct 30 21:03:39.553594 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:03:39.546519 5378 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c7035375-d666-498d-b9e7-f699e1e86c99-istio-data\") pod \"e2e-ait-2cf21005-openshift-default-5fdcf57455-c7jwb\" (UID: \"c7035375-d666-498d-b9e7-f699e1e86c99\") " pod="openshift-ingress/e2e-ait-2cf21005-openshift-default-5fdcf57455-c7jwb" Oct 30 21:03:39.553594 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:03:39.546551 5378 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lcb5t\" (UniqueName: \"kubernetes.io/projected/c7035375-d666-498d-b9e7-f699e1e86c99-kube-api-access-lcb5t\") pod \"e2e-ait-2cf21005-openshift-default-5fdcf57455-c7jwb\" (UID: \"c7035375-d666-498d-b9e7-f699e1e86c99\") " pod="openshift-ingress/e2e-ait-2cf21005-openshift-default-5fdcf57455-c7jwb" Oct 30 21:03:39.581168 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:03:39.580944 5378 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c7035375-d666-498d-b9e7-f699e1e86c99-istio-podinfo\") pod \"e2e-ait-2cf21005-openshift-default-5fdcf57455-c7jwb\" (UID: \"c7035375-d666-498d-b9e7-f699e1e86c99\") " pod="openshift-ingress/e2e-ait-2cf21005-openshift-default-5fdcf57455-c7jwb" Oct 30 21:03:39.581168 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:03:39.581043 5378 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c7035375-d666-498d-b9e7-f699e1e86c99-workload-certs\") pod \"e2e-ait-2cf21005-openshift-default-5fdcf57455-c7jwb\" (UID: \"c7035375-d666-498d-b9e7-f699e1e86c99\") " pod="openshift-ingress/e2e-ait-2cf21005-openshift-default-5fdcf57455-c7jwb" Oct 30 21:03:39.581821 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:03:39.581540 5378 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c7035375-d666-498d-b9e7-f699e1e86c99-workload-certs\") pod \"e2e-ait-2cf21005-openshift-default-5fdcf57455-c7jwb\" (UID: \"c7035375-d666-498d-b9e7-f699e1e86c99\") " pod="openshift-ingress/e2e-ait-2cf21005-openshift-default-5fdcf57455-c7jwb" Oct 30 21:03:39.581821 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:03:39.571405 5378 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c7035375-d666-498d-b9e7-f699e1e86c99-credential-socket\") pod \"e2e-ait-2cf21005-openshift-default-5fdcf57455-c7jwb\" (UID: \"c7035375-d666-498d-b9e7-f699e1e86c99\") " pod="openshift-ingress/e2e-ait-2cf21005-openshift-default-5fdcf57455-c7jwb" Oct 30 21:03:39.591736 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:03:39.589170 5378 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c7035375-d666-498d-b9e7-f699e1e86c99-istio-envoy\") pod \"e2e-ait-2cf21005-openshift-default-5fdcf57455-c7jwb\" (UID: \"c7035375-d666-498d-b9e7-f699e1e86c99\") " pod="openshift-ingress/e2e-ait-2cf21005-openshift-default-5fdcf57455-c7jwb" Oct 30 21:03:39.591736 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:03:39.589434 5378 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c7035375-d666-498d-b9e7-f699e1e86c99-workload-socket\") pod \"e2e-ait-2cf21005-openshift-default-5fdcf57455-c7jwb\" (UID: \"c7035375-d666-498d-b9e7-f699e1e86c99\") " pod="openshift-ingress/e2e-ait-2cf21005-openshift-default-5fdcf57455-c7jwb" Oct 30 21:03:39.591736 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:03:39.590054 5378 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c7035375-d666-498d-b9e7-f699e1e86c99-istiod-ca-cert\") pod \"e2e-ait-2cf21005-openshift-default-5fdcf57455-c7jwb\" (UID: \"c7035375-d666-498d-b9e7-f699e1e86c99\") " pod="openshift-ingress/e2e-ait-2cf21005-openshift-default-5fdcf57455-c7jwb" Oct 30 21:03:39.591736 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:03:39.590608 5378 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c7035375-d666-498d-b9e7-f699e1e86c99-istio-data\") pod \"e2e-ait-2cf21005-openshift-default-5fdcf57455-c7jwb\" (UID: \"c7035375-d666-498d-b9e7-f699e1e86c99\") " pod="openshift-ingress/e2e-ait-2cf21005-openshift-default-5fdcf57455-c7jwb" Oct 30 21:03:39.593094 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:03:39.593046 5378 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c7035375-d666-498d-b9e7-f699e1e86c99-istio-podinfo\") pod \"e2e-ait-2cf21005-openshift-default-5fdcf57455-c7jwb\" (UID: \"c7035375-d666-498d-b9e7-f699e1e86c99\") " pod="openshift-ingress/e2e-ait-2cf21005-openshift-default-5fdcf57455-c7jwb" Oct 30 21:03:39.624839 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:03:39.624002 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:03:39.725955 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:03:39.725858 5378 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c7035375-d666-498d-b9e7-f699e1e86c99-istio-token\") pod \"e2e-ait-2cf21005-openshift-default-5fdcf57455-c7jwb\" (UID: \"c7035375-d666-498d-b9e7-f699e1e86c99\") " pod="openshift-ingress/e2e-ait-2cf21005-openshift-default-5fdcf57455-c7jwb" Oct 30 21:03:39.920689 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:03:39.898066 5378 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lcb5t\" (UniqueName: \"kubernetes.io/projected/c7035375-d666-498d-b9e7-f699e1e86c99-kube-api-access-lcb5t\") pod \"e2e-ait-2cf21005-openshift-default-5fdcf57455-c7jwb\" (UID: \"c7035375-d666-498d-b9e7-f699e1e86c99\") " pod="openshift-ingress/e2e-ait-2cf21005-openshift-default-5fdcf57455-c7jwb" Oct 30 21:03:39.920689 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:03:39.898456 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:03:40.073633 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:03:40.073496 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:03:40.187586 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:03:40.179844 5378 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-2cf21005-openshift-default-5fdcf57455-c7jwb" Oct 30 21:03:40.791594 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:03:40.778465 5378 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-2cf21005-openshift-default-5fdcf57455-c7jwb"] Oct 30 21:03:40.830226 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:03:40.819128 5378 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 21:03:40.830226 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:03:40.819316 5378 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 21:03:40.830226 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:03:40.819409 5378 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 21:03:41.033032 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:03:41.032950 5378 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-derive-bf9c0727-openshift-default-7fb479dd89-jzzdk"] Oct 30 21:03:41.038634 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:03:41.038609 5378 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-bf9c0727-openshift-default-7fb479dd89-jzzdk" Oct 30 21:03:41.099682 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:03:41.099652 5378 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-derive-bf9c0727-openshift-default-7fb479dd89-jzzdk"] Oct 30 21:03:41.105813 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:03:41.105777 5378 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-derive-bf9c0727-openshift-default-dockercfg-d4rns\"" Oct 30 21:03:41.127739 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:03:41.127698 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:03:41.221586 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:03:41.220671 5378 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/eafa34dc-6d4b-4507-a2a3-989ad8cf7de2-istio-data\") pod \"e2e-derive-bf9c0727-openshift-default-7fb479dd89-jzzdk\" (UID: \"eafa34dc-6d4b-4507-a2a3-989ad8cf7de2\") " pod="openshift-ingress/e2e-derive-bf9c0727-openshift-default-7fb479dd89-jzzdk" Oct 30 21:03:41.221586 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:03:41.220754 5378 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/eafa34dc-6d4b-4507-a2a3-989ad8cf7de2-workload-socket\") pod \"e2e-derive-bf9c0727-openshift-default-7fb479dd89-jzzdk\" (UID: \"eafa34dc-6d4b-4507-a2a3-989ad8cf7de2\") " pod="openshift-ingress/e2e-derive-bf9c0727-openshift-default-7fb479dd89-jzzdk" Oct 30 21:03:41.221586 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:03:41.220891 5378 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/eafa34dc-6d4b-4507-a2a3-989ad8cf7de2-workload-certs\") pod \"e2e-derive-bf9c0727-openshift-default-7fb479dd89-jzzdk\" (UID: \"eafa34dc-6d4b-4507-a2a3-989ad8cf7de2\") " pod="openshift-ingress/e2e-derive-bf9c0727-openshift-default-7fb479dd89-jzzdk" Oct 30 21:03:41.221586 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:03:41.220963 5378 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/eafa34dc-6d4b-4507-a2a3-989ad8cf7de2-istio-token\") pod \"e2e-derive-bf9c0727-openshift-default-7fb479dd89-jzzdk\" (UID: \"eafa34dc-6d4b-4507-a2a3-989ad8cf7de2\") " pod="openshift-ingress/e2e-derive-bf9c0727-openshift-default-7fb479dd89-jzzdk" Oct 30 21:03:41.221586 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:03:41.221038 5378 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/eafa34dc-6d4b-4507-a2a3-989ad8cf7de2-istio-podinfo\") pod \"e2e-derive-bf9c0727-openshift-default-7fb479dd89-jzzdk\" (UID: \"eafa34dc-6d4b-4507-a2a3-989ad8cf7de2\") " pod="openshift-ingress/e2e-derive-bf9c0727-openshift-default-7fb479dd89-jzzdk" Oct 30 21:03:41.221586 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:03:41.221082 5378 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/eafa34dc-6d4b-4507-a2a3-989ad8cf7de2-credential-socket\") pod \"e2e-derive-bf9c0727-openshift-default-7fb479dd89-jzzdk\" (UID: \"eafa34dc-6d4b-4507-a2a3-989ad8cf7de2\") " pod="openshift-ingress/e2e-derive-bf9c0727-openshift-default-7fb479dd89-jzzdk" Oct 30 21:03:41.221586 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:03:41.221110 5378 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kqdmr\" (UniqueName: \"kubernetes.io/projected/eafa34dc-6d4b-4507-a2a3-989ad8cf7de2-kube-api-access-kqdmr\") pod \"e2e-derive-bf9c0727-openshift-default-7fb479dd89-jzzdk\" (UID: \"eafa34dc-6d4b-4507-a2a3-989ad8cf7de2\") " pod="openshift-ingress/e2e-derive-bf9c0727-openshift-default-7fb479dd89-jzzdk" Oct 30 21:03:41.221586 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:03:41.221139 5378 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/eafa34dc-6d4b-4507-a2a3-989ad8cf7de2-istio-envoy\") pod \"e2e-derive-bf9c0727-openshift-default-7fb479dd89-jzzdk\" (UID: \"eafa34dc-6d4b-4507-a2a3-989ad8cf7de2\") " pod="openshift-ingress/e2e-derive-bf9c0727-openshift-default-7fb479dd89-jzzdk" Oct 30 21:03:41.221586 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:03:41.221184 5378 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/eafa34dc-6d4b-4507-a2a3-989ad8cf7de2-istiod-ca-cert\") pod \"e2e-derive-bf9c0727-openshift-default-7fb479dd89-jzzdk\" (UID: \"eafa34dc-6d4b-4507-a2a3-989ad8cf7de2\") " pod="openshift-ingress/e2e-derive-bf9c0727-openshift-default-7fb479dd89-jzzdk" Oct 30 21:03:41.322653 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:03:41.322545 5378 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/eafa34dc-6d4b-4507-a2a3-989ad8cf7de2-istio-data\") pod \"e2e-derive-bf9c0727-openshift-default-7fb479dd89-jzzdk\" (UID: \"eafa34dc-6d4b-4507-a2a3-989ad8cf7de2\") " pod="openshift-ingress/e2e-derive-bf9c0727-openshift-default-7fb479dd89-jzzdk" Oct 30 21:03:41.322653 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:03:41.322638 5378 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/eafa34dc-6d4b-4507-a2a3-989ad8cf7de2-workload-socket\") pod \"e2e-derive-bf9c0727-openshift-default-7fb479dd89-jzzdk\" (UID: \"eafa34dc-6d4b-4507-a2a3-989ad8cf7de2\") " pod="openshift-ingress/e2e-derive-bf9c0727-openshift-default-7fb479dd89-jzzdk" Oct 30 21:03:41.322882 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:03:41.322743 5378 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/eafa34dc-6d4b-4507-a2a3-989ad8cf7de2-workload-certs\") pod \"e2e-derive-bf9c0727-openshift-default-7fb479dd89-jzzdk\" (UID: \"eafa34dc-6d4b-4507-a2a3-989ad8cf7de2\") " pod="openshift-ingress/e2e-derive-bf9c0727-openshift-default-7fb479dd89-jzzdk" Oct 30 21:03:41.322882 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:03:41.322791 5378 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/eafa34dc-6d4b-4507-a2a3-989ad8cf7de2-istio-token\") pod \"e2e-derive-bf9c0727-openshift-default-7fb479dd89-jzzdk\" (UID: \"eafa34dc-6d4b-4507-a2a3-989ad8cf7de2\") " pod="openshift-ingress/e2e-derive-bf9c0727-openshift-default-7fb479dd89-jzzdk" Oct 30 21:03:41.322882 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:03:41.322855 5378 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/eafa34dc-6d4b-4507-a2a3-989ad8cf7de2-istio-podinfo\") pod \"e2e-derive-bf9c0727-openshift-default-7fb479dd89-jzzdk\" (UID: \"eafa34dc-6d4b-4507-a2a3-989ad8cf7de2\") " pod="openshift-ingress/e2e-derive-bf9c0727-openshift-default-7fb479dd89-jzzdk" Oct 30 21:03:41.323056 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:03:41.322900 5378 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/eafa34dc-6d4b-4507-a2a3-989ad8cf7de2-credential-socket\") pod \"e2e-derive-bf9c0727-openshift-default-7fb479dd89-jzzdk\" (UID: \"eafa34dc-6d4b-4507-a2a3-989ad8cf7de2\") " pod="openshift-ingress/e2e-derive-bf9c0727-openshift-default-7fb479dd89-jzzdk" Oct 30 21:03:41.323056 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:03:41.322926 5378 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kqdmr\" (UniqueName: \"kubernetes.io/projected/eafa34dc-6d4b-4507-a2a3-989ad8cf7de2-kube-api-access-kqdmr\") pod \"e2e-derive-bf9c0727-openshift-default-7fb479dd89-jzzdk\" (UID: \"eafa34dc-6d4b-4507-a2a3-989ad8cf7de2\") " pod="openshift-ingress/e2e-derive-bf9c0727-openshift-default-7fb479dd89-jzzdk" Oct 30 21:03:41.323056 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:03:41.322958 5378 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/eafa34dc-6d4b-4507-a2a3-989ad8cf7de2-istio-envoy\") pod \"e2e-derive-bf9c0727-openshift-default-7fb479dd89-jzzdk\" (UID: \"eafa34dc-6d4b-4507-a2a3-989ad8cf7de2\") " pod="openshift-ingress/e2e-derive-bf9c0727-openshift-default-7fb479dd89-jzzdk" Oct 30 21:03:41.323056 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:03:41.322982 5378 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/eafa34dc-6d4b-4507-a2a3-989ad8cf7de2-istio-data\") pod \"e2e-derive-bf9c0727-openshift-default-7fb479dd89-jzzdk\" (UID: \"eafa34dc-6d4b-4507-a2a3-989ad8cf7de2\") " pod="openshift-ingress/e2e-derive-bf9c0727-openshift-default-7fb479dd89-jzzdk" Oct 30 21:03:41.323056 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:03:41.322992 5378 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/eafa34dc-6d4b-4507-a2a3-989ad8cf7de2-workload-socket\") pod \"e2e-derive-bf9c0727-openshift-default-7fb479dd89-jzzdk\" (UID: \"eafa34dc-6d4b-4507-a2a3-989ad8cf7de2\") " pod="openshift-ingress/e2e-derive-bf9c0727-openshift-default-7fb479dd89-jzzdk" Oct 30 21:03:41.323056 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:03:41.323005 5378 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/eafa34dc-6d4b-4507-a2a3-989ad8cf7de2-istiod-ca-cert\") pod \"e2e-derive-bf9c0727-openshift-default-7fb479dd89-jzzdk\" (UID: \"eafa34dc-6d4b-4507-a2a3-989ad8cf7de2\") " pod="openshift-ingress/e2e-derive-bf9c0727-openshift-default-7fb479dd89-jzzdk" Oct 30 21:03:41.323497 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:03:41.323472 5378 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/eafa34dc-6d4b-4507-a2a3-989ad8cf7de2-credential-socket\") pod \"e2e-derive-bf9c0727-openshift-default-7fb479dd89-jzzdk\" (UID: \"eafa34dc-6d4b-4507-a2a3-989ad8cf7de2\") " pod="openshift-ingress/e2e-derive-bf9c0727-openshift-default-7fb479dd89-jzzdk" Oct 30 21:03:41.323715 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:03:41.323695 5378 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/eafa34dc-6d4b-4507-a2a3-989ad8cf7de2-workload-certs\") pod \"e2e-derive-bf9c0727-openshift-default-7fb479dd89-jzzdk\" (UID: \"eafa34dc-6d4b-4507-a2a3-989ad8cf7de2\") " pod="openshift-ingress/e2e-derive-bf9c0727-openshift-default-7fb479dd89-jzzdk" Oct 30 21:03:41.324134 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:03:41.324106 5378 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/eafa34dc-6d4b-4507-a2a3-989ad8cf7de2-istiod-ca-cert\") pod \"e2e-derive-bf9c0727-openshift-default-7fb479dd89-jzzdk\" (UID: \"eafa34dc-6d4b-4507-a2a3-989ad8cf7de2\") " pod="openshift-ingress/e2e-derive-bf9c0727-openshift-default-7fb479dd89-jzzdk" Oct 30 21:03:41.325463 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:03:41.325435 5378 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/eafa34dc-6d4b-4507-a2a3-989ad8cf7de2-istio-envoy\") pod \"e2e-derive-bf9c0727-openshift-default-7fb479dd89-jzzdk\" (UID: \"eafa34dc-6d4b-4507-a2a3-989ad8cf7de2\") " pod="openshift-ingress/e2e-derive-bf9c0727-openshift-default-7fb479dd89-jzzdk" Oct 30 21:03:41.325967 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:03:41.325940 5378 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/eafa34dc-6d4b-4507-a2a3-989ad8cf7de2-istio-podinfo\") pod \"e2e-derive-bf9c0727-openshift-default-7fb479dd89-jzzdk\" (UID: \"eafa34dc-6d4b-4507-a2a3-989ad8cf7de2\") " pod="openshift-ingress/e2e-derive-bf9c0727-openshift-default-7fb479dd89-jzzdk" Oct 30 21:03:41.380489 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:03:41.380450 5378 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kqdmr\" (UniqueName: \"kubernetes.io/projected/eafa34dc-6d4b-4507-a2a3-989ad8cf7de2-kube-api-access-kqdmr\") pod \"e2e-derive-bf9c0727-openshift-default-7fb479dd89-jzzdk\" (UID: \"eafa34dc-6d4b-4507-a2a3-989ad8cf7de2\") " pod="openshift-ingress/e2e-derive-bf9c0727-openshift-default-7fb479dd89-jzzdk" Oct 30 21:03:41.380653 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:03:41.380616 5378 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/eafa34dc-6d4b-4507-a2a3-989ad8cf7de2-istio-token\") pod \"e2e-derive-bf9c0727-openshift-default-7fb479dd89-jzzdk\" (UID: \"eafa34dc-6d4b-4507-a2a3-989ad8cf7de2\") " pod="openshift-ingress/e2e-derive-bf9c0727-openshift-default-7fb479dd89-jzzdk" Oct 30 21:03:41.592118 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:03:41.590723 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:03:41.661780 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:03:41.661747 5378 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-bf9c0727-openshift-default-7fb479dd89-jzzdk" Oct 30 21:03:41.843489 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:03:41.843393 5378 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-2cf21005-openshift-default-5fdcf57455-c7jwb" event={"ID":"c7035375-d666-498d-b9e7-f699e1e86c99","Type":"ContainerStarted","Data":"59bfd2be3c8f8ea9773d7c0dfc4caacef2573c6729b786c89f506e047f94d370"} Oct 30 21:03:41.843489 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:03:41.843446 5378 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-2cf21005-openshift-default-5fdcf57455-c7jwb" event={"ID":"c7035375-d666-498d-b9e7-f699e1e86c99","Type":"ContainerStarted","Data":"e30b2b2df50ae6f8835d26d496c18b1b80bd047343b7f067451c91b66531e82c"} Oct 30 21:03:41.964267 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:03:41.964210 5378 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-2cf21005-openshift-default-5fdcf57455-c7jwb" podStartSLOduration=2.964196634 podStartE2EDuration="2.964196634s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:03:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:03:41.964011816 +0000 UTC m=+1512.392125735" watchObservedRunningTime="2026-10-30 21:03:41.964196634 +0000 UTC m=+1512.392310552" Oct 30 21:03:42.156540 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:03:42.155965 5378 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-derive-bf9c0727-openshift-default-7fb479dd89-jzzdk"] Oct 30 21:03:42.181580 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:03:42.180915 5378 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-2cf21005-openshift-default-5fdcf57455-c7jwb" Oct 30 21:03:42.184896 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:03:42.184320 5378 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 21:03:42.184896 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:03:42.184414 5378 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 21:03:42.184896 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:03:42.184457 5378 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 21:03:42.194073 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:03:42.194052 5378 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-2cf21005-openshift-default-5fdcf57455-c7jwb" Oct 30 21:03:42.504814 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:03:42.504699 5378 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ait-bc88b34e-55b79996d-kfqbm" podUID="727b7af5-e485-42f3-8bb5-a6b5136a3e99" containerName="payload-processing" containerID="cri-o://fade0178ccccfd995880e856a7839a0fd8e82f8252f680989f244ae4930306d6" gracePeriod=25 Oct 30 21:03:42.798466 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:03:42.798436 5378 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-bc88b34e-55b79996d-kfqbm" Oct 30 21:03:42.852920 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:03:42.852881 5378 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-bf9c0727-openshift-default-7fb479dd89-jzzdk" event={"ID":"eafa34dc-6d4b-4507-a2a3-989ad8cf7de2","Type":"ContainerStarted","Data":"c90939ef377fe76a9fbc3ed0753ebe88aaaad74d368b45d4a2658581909adb44"} Oct 30 21:03:42.853106 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:03:42.852942 5378 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-bf9c0727-openshift-default-7fb479dd89-jzzdk" event={"ID":"eafa34dc-6d4b-4507-a2a3-989ad8cf7de2","Type":"ContainerStarted","Data":"3cf3c06515afb92df7a8bd3560033ff45a29ff39fe29ee05fa78897da9b2d799"} Oct 30 21:03:42.854425 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:03:42.854399 5378 generic.go:356] "Generic (PLEG): container finished" podID="727b7af5-e485-42f3-8bb5-a6b5136a3e99" containerID="fade0178ccccfd995880e856a7839a0fd8e82f8252f680989f244ae4930306d6" exitCode=0 Oct 30 21:03:42.854547 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:03:42.854436 5378 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-bc88b34e-55b79996d-kfqbm" Oct 30 21:03:42.854547 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:03:42.854482 5378 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-bc88b34e-55b79996d-kfqbm" event={"ID":"727b7af5-e485-42f3-8bb5-a6b5136a3e99","Type":"ContainerDied","Data":"fade0178ccccfd995880e856a7839a0fd8e82f8252f680989f244ae4930306d6"} Oct 30 21:03:42.854547 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:03:42.854533 5378 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-bc88b34e-55b79996d-kfqbm" event={"ID":"727b7af5-e485-42f3-8bb5-a6b5136a3e99","Type":"ContainerDied","Data":"c2d2ef2b8e418a7d4d6581e8a1344d142c2344ae79e4d013e089e33ff529ea71"} Oct 30 21:03:42.854802 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:03:42.854550 5378 scope.go:122] "RemoveContainer" containerID="fade0178ccccfd995880e856a7839a0fd8e82f8252f680989f244ae4930306d6" Oct 30 21:03:42.854851 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:03:42.854812 5378 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-2cf21005-openshift-default-5fdcf57455-c7jwb" Oct 30 21:03:42.855933 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:03:42.855913 5378 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-2cf21005-openshift-default-5fdcf57455-c7jwb" Oct 30 21:03:42.864783 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:03:42.864766 5378 scope.go:122] "RemoveContainer" containerID="fade0178ccccfd995880e856a7839a0fd8e82f8252f680989f244ae4930306d6" Oct 30 21:03:42.865118 ip-10-0-138-149 kubenswrapper[5378]: E1030 21:03:42.865092 5378 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fade0178ccccfd995880e856a7839a0fd8e82f8252f680989f244ae4930306d6\": container with ID starting with fade0178ccccfd995880e856a7839a0fd8e82f8252f680989f244ae4930306d6 not found: ID does not exist" containerID="fade0178ccccfd995880e856a7839a0fd8e82f8252f680989f244ae4930306d6" Oct 30 21:03:42.865198 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:03:42.865128 5378 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fade0178ccccfd995880e856a7839a0fd8e82f8252f680989f244ae4930306d6"} err="failed to get container status \"fade0178ccccfd995880e856a7839a0fd8e82f8252f680989f244ae4930306d6\": rpc error: code = NotFound desc = could not find container \"fade0178ccccfd995880e856a7839a0fd8e82f8252f680989f244ae4930306d6\": container with ID starting with fade0178ccccfd995880e856a7839a0fd8e82f8252f680989f244ae4930306d6 not found: ID does not exist" Oct 30 21:03:42.875819 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:03:42.875794 5378 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/727b7af5-e485-42f3-8bb5-a6b5136a3e99-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/727b7af5-e485-42f3-8bb5-a6b5136a3e99-plugins-config-volume\") pod \"727b7af5-e485-42f3-8bb5-a6b5136a3e99\" (UID: \"727b7af5-e485-42f3-8bb5-a6b5136a3e99\") " Oct 30 21:03:42.875926 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:03:42.875849 5378 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/727b7af5-e485-42f3-8bb5-a6b5136a3e99-kube-api-access-lcjtn\" (UniqueName: \"kubernetes.io/projected/727b7af5-e485-42f3-8bb5-a6b5136a3e99-kube-api-access-lcjtn\") pod \"727b7af5-e485-42f3-8bb5-a6b5136a3e99\" (UID: \"727b7af5-e485-42f3-8bb5-a6b5136a3e99\") " Oct 30 21:03:42.876169 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:03:42.876146 5378 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/727b7af5-e485-42f3-8bb5-a6b5136a3e99-plugins-config-volume" pod "727b7af5-e485-42f3-8bb5-a6b5136a3e99" (UID: "727b7af5-e485-42f3-8bb5-a6b5136a3e99"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:03:42.876729 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:03:42.876703 5378 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/727b7af5-e485-42f3-8bb5-a6b5136a3e99-plugins-config-volume\") on node \"ip-10-0-138-149.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:03:42.878187 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:03:42.878167 5378 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/727b7af5-e485-42f3-8bb5-a6b5136a3e99-kube-api-access-lcjtn" pod "727b7af5-e485-42f3-8bb5-a6b5136a3e99" (UID: "727b7af5-e485-42f3-8bb5-a6b5136a3e99"). InnerVolumeSpecName "kube-api-access-lcjtn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:03:42.978864 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:03:42.978830 5378 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lcjtn\" (UniqueName: \"kubernetes.io/projected/727b7af5-e485-42f3-8bb5-a6b5136a3e99-kube-api-access-lcjtn\") on node \"ip-10-0-138-149.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:03:43.091806 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:03:43.091691 5378 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-derive-bf9c0727-openshift-default-7fb479dd89-jzzdk" podStartSLOduration=3.091677112 podStartE2EDuration="3.091677112s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:03:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:03:43.089450183 +0000 UTC m=+1513.517564125" watchObservedRunningTime="2026-10-30 21:03:43.091677112 +0000 UTC m=+1513.519791030" Oct 30 21:03:43.394846 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:03:43.391068 5378 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-bc88b34e-55b79996d-kfqbm"] Oct 30 21:03:43.397743 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:03:43.397713 5378 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-bc88b34e-55b79996d-kfqbm"] Oct 30 21:03:43.662777 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:03:43.662686 5378 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-derive-bf9c0727-openshift-default-7fb479dd89-jzzdk" Oct 30 21:03:43.668128 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:03:43.668104 5378 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-derive-bf9c0727-openshift-default-7fb479dd89-jzzdk" Oct 30 21:03:43.742671 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:03:43.742638 5378 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="727b7af5-e485-42f3-8bb5-a6b5136a3e99" path="/var/lib/kubelet/pods/727b7af5-e485-42f3-8bb5-a6b5136a3e99/volumes" Oct 30 21:03:43.863349 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:03:43.863318 5378 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-derive-bf9c0727-openshift-default-7fb479dd89-jzzdk" Oct 30 21:03:43.864351 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:03:43.864329 5378 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-derive-bf9c0727-openshift-default-7fb479dd89-jzzdk" Oct 30 21:03:44.150601 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:03:44.150505 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:03:44.390866 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:03:44.390826 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:03:44.545854 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:03:44.545742 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:03:44.706493 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:03:44.706457 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:03:44.857427 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:03:44.857394 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:03:45.157786 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:03:45.157686 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:03:45.327675 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:03:45.327637 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:03:45.526422 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:03:45.526324 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:03:47.551288 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:03:47.551239 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:03:47.734587 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:03:47.733751 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:03:48.839442 ip-10-0-138-149 kubenswrapper[5378]: E1030 21:03:48.839406 5378 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q6mx7" containerName="extractor" Oct 30 21:03:49.187682 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:03:49.187651 5378 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-r9g59"] Oct 30 21:03:50.491991 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:03:50.491959 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:03:52.644920 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:03:52.644887 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:03:52.992324 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:03:52.992122 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:03:53.084768 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:03:53.084718 5378 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-bf9c0727-55f74857d7-m98lb"] Oct 30 21:03:53.086223 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:03:53.086198 5378 cpu_manager.go:414] "containerMap: removing container" podUID="727b7af5-e485-42f3-8bb5-a6b5136a3e99" containerName="payload-processing" Oct 30 21:03:53.086376 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:03:53.086364 5378 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="727b7af5-e485-42f3-8bb5-a6b5136a3e99" containerName="payload-processing" Oct 30 21:03:53.086907 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:03:53.086874 5378 memory_manager.go:365] "RemoveStaleState removing state" podUID="727b7af5-e485-42f3-8bb5-a6b5136a3e99" containerName="payload-processing" Oct 30 21:03:53.091670 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:03:53.091649 5378 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-bf9c0727-55f74857d7-m98lb" Oct 30 21:03:53.202589 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:03:53.199400 5378 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-derive-bf9c0727\"" Oct 30 21:03:53.202589 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:03:53.199479 5378 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-derive-bf9c0727-dockercfg-wz5r9\"" Oct 30 21:03:53.207687 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:03:53.205470 5378 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-bf9c0727-55f74857d7-m98lb"] Oct 30 21:03:53.224489 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:03:53.223850 5378 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jvvnf\" (UniqueName: \"kubernetes.io/projected/9723840f-d122-4d53-8600-346cd5f2db0d-kube-api-access-jvvnf\") pod \"payload-pre-processing-e2e-derive-bf9c0727-55f74857d7-m98lb\" (UID: \"9723840f-d122-4d53-8600-346cd5f2db0d\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-bf9c0727-55f74857d7-m98lb" Oct 30 21:03:53.224489 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:03:53.224292 5378 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9723840f-d122-4d53-8600-346cd5f2db0d-plugins-config-volume\") pod \"payload-pre-processing-e2e-derive-bf9c0727-55f74857d7-m98lb\" (UID: \"9723840f-d122-4d53-8600-346cd5f2db0d\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-bf9c0727-55f74857d7-m98lb" Oct 30 21:03:53.466553 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:03:53.466510 5378 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jvvnf\" (UniqueName: \"kubernetes.io/projected/9723840f-d122-4d53-8600-346cd5f2db0d-kube-api-access-jvvnf\") pod \"payload-pre-processing-e2e-derive-bf9c0727-55f74857d7-m98lb\" (UID: \"9723840f-d122-4d53-8600-346cd5f2db0d\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-bf9c0727-55f74857d7-m98lb" Oct 30 21:03:53.483766 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:03:53.483725 5378 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9723840f-d122-4d53-8600-346cd5f2db0d-plugins-config-volume\") pod \"payload-pre-processing-e2e-derive-bf9c0727-55f74857d7-m98lb\" (UID: \"9723840f-d122-4d53-8600-346cd5f2db0d\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-bf9c0727-55f74857d7-m98lb" Oct 30 21:03:53.507621 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:03:53.485376 5378 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9723840f-d122-4d53-8600-346cd5f2db0d-plugins-config-volume\") pod \"payload-pre-processing-e2e-derive-bf9c0727-55f74857d7-m98lb\" (UID: \"9723840f-d122-4d53-8600-346cd5f2db0d\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-bf9c0727-55f74857d7-m98lb" Oct 30 21:03:53.617425 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:03:53.617392 5378 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jvvnf\" (UniqueName: \"kubernetes.io/projected/9723840f-d122-4d53-8600-346cd5f2db0d-kube-api-access-jvvnf\") pod \"payload-pre-processing-e2e-derive-bf9c0727-55f74857d7-m98lb\" (UID: \"9723840f-d122-4d53-8600-346cd5f2db0d\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-bf9c0727-55f74857d7-m98lb" Oct 30 21:03:53.761293 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:03:53.761209 5378 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-bf9c0727-55f74857d7-m98lb" Oct 30 21:03:54.142463 ip-10-0-138-149 kubenswrapper[5378]: W1030 21:03:54.142433 5378 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9723840f_d122_4d53_8600_346cd5f2db0d.slice/crio-456af6ca162cff6c24cd8b7a0485325802b5a9739e04414b0080c65fe15ab06f WatchSource:0}: Error finding container 456af6ca162cff6c24cd8b7a0485325802b5a9739e04414b0080c65fe15ab06f: Status 404 returned error can't find the container with id 456af6ca162cff6c24cd8b7a0485325802b5a9739e04414b0080c65fe15ab06f Oct 30 21:03:54.142626 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:03:54.142603 5378 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-bf9c0727-55f74857d7-m98lb"] Oct 30 21:03:55.077690 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:03:55.077644 5378 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-bf9c0727-55f74857d7-m98lb" event={"ID":"9723840f-d122-4d53-8600-346cd5f2db0d","Type":"ContainerStarted","Data":"37a55991a0ab07211368ac400f64601b9d244b9b8b202b41e2c37d3ee08d72e8"} Oct 30 21:03:55.078133 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:03:55.077722 5378 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-bf9c0727-55f74857d7-m98lb" event={"ID":"9723840f-d122-4d53-8600-346cd5f2db0d","Type":"ContainerStarted","Data":"456af6ca162cff6c24cd8b7a0485325802b5a9739e04414b0080c65fe15ab06f"} Oct 30 21:03:55.078133 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:03:55.077764 5378 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-derive-bf9c0727-55f74857d7-m98lb" Oct 30 21:03:55.202799 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:03:55.202744 5378 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-derive-bf9c0727-55f74857d7-m98lb" podStartSLOduration=2.202730368 podStartE2EDuration="2.202730368s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:03:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:03:55.19933348 +0000 UTC m=+1525.627447399" watchObservedRunningTime="2026-10-30 21:03:55.202730368 +0000 UTC m=+1525.630844288" Oct 30 21:03:55.524054 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:03:55.524023 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:04:00.376362 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:04:00.376327 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:04:04.467009 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:04:04.466977 5378 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-bc88b34e-67f64959c8-5kndg"] Oct 30 21:04:04.467426 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:04:04.467195 5378 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-bc88b34e-67f64959c8-5kndg" podUID="2fcd8f8c-42a7-4a72-9004-97c7f84fffac" containerName="maas-api" containerID="cri-o://b3a953496eb8de8d9cd004730ff7563cc90d7e8f55b8b6c0addd459f12e184e0" gracePeriod=30 Oct 30 21:04:04.552476 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:04:04.552447 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:04:04.556469 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:04:04.556435 5378 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-ait-bc88b34e-67f64959c8-5kndg" podUID="2fcd8f8c-42a7-4a72-9004-97c7f84fffac" containerName="maas-api" probeResult="failure" output="Get \"https://10.133.0.104:8443/health\": dial tcp 10.133.0.104:8443: connect: connection refused" Oct 30 21:04:04.985755 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:04:04.985392 5378 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-bc88b34e-67f64959c8-5kndg" Oct 30 21:04:05.151071 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:04:05.151032 5378 generic.go:356] "Generic (PLEG): container finished" podID="2fcd8f8c-42a7-4a72-9004-97c7f84fffac" containerID="b3a953496eb8de8d9cd004730ff7563cc90d7e8f55b8b6c0addd459f12e184e0" exitCode=0 Oct 30 21:04:05.151389 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:04:05.151098 5378 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-bc88b34e-67f64959c8-5kndg" Oct 30 21:04:05.152298 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:04:05.151118 5378 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-bc88b34e-67f64959c8-5kndg" event={"ID":"2fcd8f8c-42a7-4a72-9004-97c7f84fffac","Type":"ContainerDied","Data":"b3a953496eb8de8d9cd004730ff7563cc90d7e8f55b8b6c0addd459f12e184e0"} Oct 30 21:04:05.152298 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:04:05.151795 5378 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-bc88b34e-67f64959c8-5kndg" event={"ID":"2fcd8f8c-42a7-4a72-9004-97c7f84fffac","Type":"ContainerDied","Data":"e9fecc9a48e03dedefaeb67b2578566e5335f7295f4eea65d3d3443941acca5a"} Oct 30 21:04:05.152298 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:04:05.151867 5378 scope.go:122] "RemoveContainer" containerID="b3a953496eb8de8d9cd004730ff7563cc90d7e8f55b8b6c0addd459f12e184e0" Oct 30 21:04:05.165916 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:04:05.165888 5378 scope.go:122] "RemoveContainer" containerID="b3a953496eb8de8d9cd004730ff7563cc90d7e8f55b8b6c0addd459f12e184e0" Oct 30 21:04:05.166285 ip-10-0-138-149 kubenswrapper[5378]: E1030 21:04:05.166257 5378 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b3a953496eb8de8d9cd004730ff7563cc90d7e8f55b8b6c0addd459f12e184e0\": container with ID starting with b3a953496eb8de8d9cd004730ff7563cc90d7e8f55b8b6c0addd459f12e184e0 not found: ID does not exist" containerID="b3a953496eb8de8d9cd004730ff7563cc90d7e8f55b8b6c0addd459f12e184e0" Oct 30 21:04:05.166390 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:04:05.166296 5378 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b3a953496eb8de8d9cd004730ff7563cc90d7e8f55b8b6c0addd459f12e184e0"} err="failed to get container status \"b3a953496eb8de8d9cd004730ff7563cc90d7e8f55b8b6c0addd459f12e184e0\": rpc error: code = NotFound desc = could not find container \"b3a953496eb8de8d9cd004730ff7563cc90d7e8f55b8b6c0addd459f12e184e0\": container with ID starting with b3a953496eb8de8d9cd004730ff7563cc90d7e8f55b8b6c0addd459f12e184e0 not found: ID does not exist" Oct 30 21:04:05.174889 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:04:05.174861 5378 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2fcd8f8c-42a7-4a72-9004-97c7f84fffac-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2fcd8f8c-42a7-4a72-9004-97c7f84fffac-maas-api-tls\") pod \"2fcd8f8c-42a7-4a72-9004-97c7f84fffac\" (UID: \"2fcd8f8c-42a7-4a72-9004-97c7f84fffac\") " Oct 30 21:04:05.175014 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:04:05.174957 5378 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2fcd8f8c-42a7-4a72-9004-97c7f84fffac-metrics-certs\" (UniqueName: \"kubernetes.io/secret/2fcd8f8c-42a7-4a72-9004-97c7f84fffac-metrics-certs\") pod \"2fcd8f8c-42a7-4a72-9004-97c7f84fffac\" (UID: \"2fcd8f8c-42a7-4a72-9004-97c7f84fffac\") " Oct 30 21:04:05.175082 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:04:05.175066 5378 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2fcd8f8c-42a7-4a72-9004-97c7f84fffac-kube-api-access-ghglj\" (UniqueName: \"kubernetes.io/projected/2fcd8f8c-42a7-4a72-9004-97c7f84fffac-kube-api-access-ghglj\") pod \"2fcd8f8c-42a7-4a72-9004-97c7f84fffac\" (UID: \"2fcd8f8c-42a7-4a72-9004-97c7f84fffac\") " Oct 30 21:04:05.175153 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:04:05.175136 5378 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2fcd8f8c-42a7-4a72-9004-97c7f84fffac-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2fcd8f8c-42a7-4a72-9004-97c7f84fffac-openshift-service-ca\") pod \"2fcd8f8c-42a7-4a72-9004-97c7f84fffac\" (UID: \"2fcd8f8c-42a7-4a72-9004-97c7f84fffac\") " Oct 30 21:04:05.175557 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:04:05.175532 5378 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2fcd8f8c-42a7-4a72-9004-97c7f84fffac-openshift-service-ca" pod "2fcd8f8c-42a7-4a72-9004-97c7f84fffac" (UID: "2fcd8f8c-42a7-4a72-9004-97c7f84fffac"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:04:05.175944 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:04:05.175913 5378 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2fcd8f8c-42a7-4a72-9004-97c7f84fffac-openshift-service-ca\") on node \"ip-10-0-138-149.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:04:05.177075 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:04:05.177052 5378 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2fcd8f8c-42a7-4a72-9004-97c7f84fffac-maas-api-tls" pod "2fcd8f8c-42a7-4a72-9004-97c7f84fffac" (UID: "2fcd8f8c-42a7-4a72-9004-97c7f84fffac"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:04:05.177321 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:04:05.177299 5378 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2fcd8f8c-42a7-4a72-9004-97c7f84fffac-metrics-certs" pod "2fcd8f8c-42a7-4a72-9004-97c7f84fffac" (UID: "2fcd8f8c-42a7-4a72-9004-97c7f84fffac"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:04:05.177706 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:04:05.177686 5378 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2fcd8f8c-42a7-4a72-9004-97c7f84fffac-kube-api-access-ghglj" pod "2fcd8f8c-42a7-4a72-9004-97c7f84fffac" (UID: "2fcd8f8c-42a7-4a72-9004-97c7f84fffac"). InnerVolumeSpecName "kube-api-access-ghglj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:04:05.277143 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:04:05.277107 5378 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2fcd8f8c-42a7-4a72-9004-97c7f84fffac-maas-api-tls\") on node \"ip-10-0-138-149.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:04:05.277143 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:04:05.277138 5378 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2fcd8f8c-42a7-4a72-9004-97c7f84fffac-metrics-certs\") on node \"ip-10-0-138-149.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:04:05.277143 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:04:05.277150 5378 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ghglj\" (UniqueName: \"kubernetes.io/projected/2fcd8f8c-42a7-4a72-9004-97c7f84fffac-kube-api-access-ghglj\") on node \"ip-10-0-138-149.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:04:05.732627 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:04:05.725815 5378 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-bc88b34e-67f64959c8-5kndg"] Oct 30 21:04:05.753175 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:04:05.752990 5378 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-bc88b34e-67f64959c8-5kndg"] Oct 30 21:04:06.088354 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:04:06.088325 5378 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-derive-bf9c0727-55f74857d7-m98lb" Oct 30 21:04:06.188188 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:04:06.188159 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:04:06.453250 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:04:06.453219 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:04:06.584272 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:04:06.584232 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:04:07.744059 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:04:07.744019 5378 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2fcd8f8c-42a7-4a72-9004-97c7f84fffac" path="/var/lib/kubelet/pods/2fcd8f8c-42a7-4a72-9004-97c7f84fffac/volumes" Oct 30 21:04:08.257313 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:04:08.257267 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:04:09.077298 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:04:09.077269 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:04:09.643601 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:04:09.643517 5378 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w4-5c45e5-openshift-default-9dcdd54d9-s697g"] Oct 30 21:04:09.647000 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:04:09.645325 5378 cpu_manager.go:414] "containerMap: removing container" podUID="2fcd8f8c-42a7-4a72-9004-97c7f84fffac" containerName="maas-api" Oct 30 21:04:09.647000 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:04:09.645353 5378 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2fcd8f8c-42a7-4a72-9004-97c7f84fffac" containerName="maas-api" Oct 30 21:04:09.647000 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:04:09.645859 5378 memory_manager.go:365] "RemoveStaleState removing state" podUID="2fcd8f8c-42a7-4a72-9004-97c7f84fffac" containerName="maas-api" Oct 30 21:04:09.708586 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:04:09.695896 5378 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-5c45e5-openshift-default-9dcdd54d9-s697g" Oct 30 21:04:09.731588 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:04:09.722271 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:04:09.840587 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:04:09.804646 5378 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/23f08320-e0a1-42bc-8fd1-de3492b59bca-istio-envoy\") pod \"e2e-shared-a-w4-5c45e5-openshift-default-9dcdd54d9-s697g\" (UID: \"23f08320-e0a1-42bc-8fd1-de3492b59bca\") " pod="openshift-ingress/e2e-shared-a-w4-5c45e5-openshift-default-9dcdd54d9-s697g" Oct 30 21:04:09.840587 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:04:09.804779 5378 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/23f08320-e0a1-42bc-8fd1-de3492b59bca-istiod-ca-cert\") pod \"e2e-shared-a-w4-5c45e5-openshift-default-9dcdd54d9-s697g\" (UID: \"23f08320-e0a1-42bc-8fd1-de3492b59bca\") " pod="openshift-ingress/e2e-shared-a-w4-5c45e5-openshift-default-9dcdd54d9-s697g" Oct 30 21:04:09.840587 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:04:09.804817 5378 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/23f08320-e0a1-42bc-8fd1-de3492b59bca-istio-token\") pod \"e2e-shared-a-w4-5c45e5-openshift-default-9dcdd54d9-s697g\" (UID: \"23f08320-e0a1-42bc-8fd1-de3492b59bca\") " pod="openshift-ingress/e2e-shared-a-w4-5c45e5-openshift-default-9dcdd54d9-s697g" Oct 30 21:04:09.840587 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:04:09.804879 5378 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/23f08320-e0a1-42bc-8fd1-de3492b59bca-workload-socket\") pod \"e2e-shared-a-w4-5c45e5-openshift-default-9dcdd54d9-s697g\" (UID: \"23f08320-e0a1-42bc-8fd1-de3492b59bca\") " pod="openshift-ingress/e2e-shared-a-w4-5c45e5-openshift-default-9dcdd54d9-s697g" Oct 30 21:04:09.840587 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:04:09.804930 5378 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/23f08320-e0a1-42bc-8fd1-de3492b59bca-istio-podinfo\") pod \"e2e-shared-a-w4-5c45e5-openshift-default-9dcdd54d9-s697g\" (UID: \"23f08320-e0a1-42bc-8fd1-de3492b59bca\") " pod="openshift-ingress/e2e-shared-a-w4-5c45e5-openshift-default-9dcdd54d9-s697g" Oct 30 21:04:09.840587 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:04:09.804969 5378 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/23f08320-e0a1-42bc-8fd1-de3492b59bca-credential-socket\") pod \"e2e-shared-a-w4-5c45e5-openshift-default-9dcdd54d9-s697g\" (UID: \"23f08320-e0a1-42bc-8fd1-de3492b59bca\") " pod="openshift-ingress/e2e-shared-a-w4-5c45e5-openshift-default-9dcdd54d9-s697g" Oct 30 21:04:09.840587 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:04:09.805104 5378 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/23f08320-e0a1-42bc-8fd1-de3492b59bca-workload-certs\") pod \"e2e-shared-a-w4-5c45e5-openshift-default-9dcdd54d9-s697g\" (UID: \"23f08320-e0a1-42bc-8fd1-de3492b59bca\") " pod="openshift-ingress/e2e-shared-a-w4-5c45e5-openshift-default-9dcdd54d9-s697g" Oct 30 21:04:09.840587 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:04:09.805128 5378 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/23f08320-e0a1-42bc-8fd1-de3492b59bca-istio-data\") pod \"e2e-shared-a-w4-5c45e5-openshift-default-9dcdd54d9-s697g\" (UID: \"23f08320-e0a1-42bc-8fd1-de3492b59bca\") " pod="openshift-ingress/e2e-shared-a-w4-5c45e5-openshift-default-9dcdd54d9-s697g" Oct 30 21:04:09.840587 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:04:09.805230 5378 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-phqsd\" (UniqueName: \"kubernetes.io/projected/23f08320-e0a1-42bc-8fd1-de3492b59bca-kube-api-access-phqsd\") pod \"e2e-shared-a-w4-5c45e5-openshift-default-9dcdd54d9-s697g\" (UID: \"23f08320-e0a1-42bc-8fd1-de3492b59bca\") " pod="openshift-ingress/e2e-shared-a-w4-5c45e5-openshift-default-9dcdd54d9-s697g" Oct 30 21:04:10.021833 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:04:09.893153 5378 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w4-5c45e5-openshift-default-dockercfg-6qpv6\"" Oct 30 21:04:10.044676 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:04:09.912624 5378 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/23f08320-e0a1-42bc-8fd1-de3492b59bca-istio-envoy\") pod \"e2e-shared-a-w4-5c45e5-openshift-default-9dcdd54d9-s697g\" (UID: \"23f08320-e0a1-42bc-8fd1-de3492b59bca\") " pod="openshift-ingress/e2e-shared-a-w4-5c45e5-openshift-default-9dcdd54d9-s697g" Oct 30 21:04:10.049859 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:04:10.049761 5378 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/23f08320-e0a1-42bc-8fd1-de3492b59bca-istiod-ca-cert\") pod \"e2e-shared-a-w4-5c45e5-openshift-default-9dcdd54d9-s697g\" (UID: \"23f08320-e0a1-42bc-8fd1-de3492b59bca\") " pod="openshift-ingress/e2e-shared-a-w4-5c45e5-openshift-default-9dcdd54d9-s697g" Oct 30 21:04:10.061905 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:04:10.061868 5378 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/23f08320-e0a1-42bc-8fd1-de3492b59bca-istio-token\") pod \"e2e-shared-a-w4-5c45e5-openshift-default-9dcdd54d9-s697g\" (UID: \"23f08320-e0a1-42bc-8fd1-de3492b59bca\") " pod="openshift-ingress/e2e-shared-a-w4-5c45e5-openshift-default-9dcdd54d9-s697g" Oct 30 21:04:10.062182 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:04:10.062164 5378 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/23f08320-e0a1-42bc-8fd1-de3492b59bca-workload-socket\") pod \"e2e-shared-a-w4-5c45e5-openshift-default-9dcdd54d9-s697g\" (UID: \"23f08320-e0a1-42bc-8fd1-de3492b59bca\") " pod="openshift-ingress/e2e-shared-a-w4-5c45e5-openshift-default-9dcdd54d9-s697g" Oct 30 21:04:10.069801 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:04:10.062296 5378 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/23f08320-e0a1-42bc-8fd1-de3492b59bca-istio-podinfo\") pod \"e2e-shared-a-w4-5c45e5-openshift-default-9dcdd54d9-s697g\" (UID: \"23f08320-e0a1-42bc-8fd1-de3492b59bca\") " pod="openshift-ingress/e2e-shared-a-w4-5c45e5-openshift-default-9dcdd54d9-s697g" Oct 30 21:04:10.070048 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:04:10.070026 5378 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/23f08320-e0a1-42bc-8fd1-de3492b59bca-credential-socket\") pod \"e2e-shared-a-w4-5c45e5-openshift-default-9dcdd54d9-s697g\" (UID: \"23f08320-e0a1-42bc-8fd1-de3492b59bca\") " pod="openshift-ingress/e2e-shared-a-w4-5c45e5-openshift-default-9dcdd54d9-s697g" Oct 30 21:04:10.099441 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:04:10.074710 5378 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/23f08320-e0a1-42bc-8fd1-de3492b59bca-workload-certs\") pod \"e2e-shared-a-w4-5c45e5-openshift-default-9dcdd54d9-s697g\" (UID: \"23f08320-e0a1-42bc-8fd1-de3492b59bca\") " pod="openshift-ingress/e2e-shared-a-w4-5c45e5-openshift-default-9dcdd54d9-s697g" Oct 30 21:04:10.100065 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:04:10.100030 5378 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/23f08320-e0a1-42bc-8fd1-de3492b59bca-istio-data\") pod \"e2e-shared-a-w4-5c45e5-openshift-default-9dcdd54d9-s697g\" (UID: \"23f08320-e0a1-42bc-8fd1-de3492b59bca\") " pod="openshift-ingress/e2e-shared-a-w4-5c45e5-openshift-default-9dcdd54d9-s697g" Oct 30 21:04:10.105267 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:04:10.102779 5378 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-phqsd\" (UniqueName: \"kubernetes.io/projected/23f08320-e0a1-42bc-8fd1-de3492b59bca-kube-api-access-phqsd\") pod \"e2e-shared-a-w4-5c45e5-openshift-default-9dcdd54d9-s697g\" (UID: \"23f08320-e0a1-42bc-8fd1-de3492b59bca\") " pod="openshift-ingress/e2e-shared-a-w4-5c45e5-openshift-default-9dcdd54d9-s697g" Oct 30 21:04:10.138590 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:04:10.066192 5378 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/23f08320-e0a1-42bc-8fd1-de3492b59bca-istiod-ca-cert\") pod \"e2e-shared-a-w4-5c45e5-openshift-default-9dcdd54d9-s697g\" (UID: \"23f08320-e0a1-42bc-8fd1-de3492b59bca\") " pod="openshift-ingress/e2e-shared-a-w4-5c45e5-openshift-default-9dcdd54d9-s697g" Oct 30 21:04:10.138838 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:04:09.960262 5378 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/23f08320-e0a1-42bc-8fd1-de3492b59bca-istio-envoy\") pod \"e2e-shared-a-w4-5c45e5-openshift-default-9dcdd54d9-s697g\" (UID: \"23f08320-e0a1-42bc-8fd1-de3492b59bca\") " pod="openshift-ingress/e2e-shared-a-w4-5c45e5-openshift-default-9dcdd54d9-s697g" Oct 30 21:04:10.138953 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:04:10.075976 5378 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/23f08320-e0a1-42bc-8fd1-de3492b59bca-workload-certs\") pod \"e2e-shared-a-w4-5c45e5-openshift-default-9dcdd54d9-s697g\" (UID: \"23f08320-e0a1-42bc-8fd1-de3492b59bca\") " pod="openshift-ingress/e2e-shared-a-w4-5c45e5-openshift-default-9dcdd54d9-s697g" Oct 30 21:04:10.139049 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:04:10.104348 5378 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/23f08320-e0a1-42bc-8fd1-de3492b59bca-istio-data\") pod \"e2e-shared-a-w4-5c45e5-openshift-default-9dcdd54d9-s697g\" (UID: \"23f08320-e0a1-42bc-8fd1-de3492b59bca\") " pod="openshift-ingress/e2e-shared-a-w4-5c45e5-openshift-default-9dcdd54d9-s697g" Oct 30 21:04:10.139132 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:04:10.066741 5378 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/23f08320-e0a1-42bc-8fd1-de3492b59bca-workload-socket\") pod \"e2e-shared-a-w4-5c45e5-openshift-default-9dcdd54d9-s697g\" (UID: \"23f08320-e0a1-42bc-8fd1-de3492b59bca\") " pod="openshift-ingress/e2e-shared-a-w4-5c45e5-openshift-default-9dcdd54d9-s697g" Oct 30 21:04:10.139218 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:04:10.071059 5378 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/23f08320-e0a1-42bc-8fd1-de3492b59bca-credential-socket\") pod \"e2e-shared-a-w4-5c45e5-openshift-default-9dcdd54d9-s697g\" (UID: \"23f08320-e0a1-42bc-8fd1-de3492b59bca\") " pod="openshift-ingress/e2e-shared-a-w4-5c45e5-openshift-default-9dcdd54d9-s697g" Oct 30 21:04:10.206590 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:04:10.133880 5378 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/23f08320-e0a1-42bc-8fd1-de3492b59bca-istio-podinfo\") pod \"e2e-shared-a-w4-5c45e5-openshift-default-9dcdd54d9-s697g\" (UID: \"23f08320-e0a1-42bc-8fd1-de3492b59bca\") " pod="openshift-ingress/e2e-shared-a-w4-5c45e5-openshift-default-9dcdd54d9-s697g" Oct 30 21:04:10.206590 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:04:10.147259 5378 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/23f08320-e0a1-42bc-8fd1-de3492b59bca-istio-token\") pod \"e2e-shared-a-w4-5c45e5-openshift-default-9dcdd54d9-s697g\" (UID: \"23f08320-e0a1-42bc-8fd1-de3492b59bca\") " pod="openshift-ingress/e2e-shared-a-w4-5c45e5-openshift-default-9dcdd54d9-s697g" Oct 30 21:04:10.206590 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:04:10.194896 5378 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-phqsd\" (UniqueName: \"kubernetes.io/projected/23f08320-e0a1-42bc-8fd1-de3492b59bca-kube-api-access-phqsd\") pod \"e2e-shared-a-w4-5c45e5-openshift-default-9dcdd54d9-s697g\" (UID: \"23f08320-e0a1-42bc-8fd1-de3492b59bca\") " pod="openshift-ingress/e2e-shared-a-w4-5c45e5-openshift-default-9dcdd54d9-s697g" Oct 30 21:04:10.223281 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:04:10.223233 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:04:10.239807 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:04:10.239772 5378 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-5c45e5-openshift-default-9dcdd54d9-s697g"] Oct 30 21:04:10.436120 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:04:10.436085 5378 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-5c45e5-openshift-default-9dcdd54d9-s697g" Oct 30 21:04:10.868064 ip-10-0-138-149 kubenswrapper[5378]: W1030 21:04:10.868035 5378 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod23f08320_e0a1_42bc_8fd1_de3492b59bca.slice/crio-d502d498f085e301c28891c26418385a3ab54683c5bf0ab01b57a2a67ffbc272 WatchSource:0}: Error finding container d502d498f085e301c28891c26418385a3ab54683c5bf0ab01b57a2a67ffbc272: Status 404 returned error can't find the container with id d502d498f085e301c28891c26418385a3ab54683c5bf0ab01b57a2a67ffbc272 Oct 30 21:04:10.868373 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:04:10.868351 5378 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-5c45e5-openshift-default-9dcdd54d9-s697g"] Oct 30 21:04:10.870921 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:04:10.870891 5378 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 21:04:10.870997 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:04:10.870966 5378 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 21:04:10.871036 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:04:10.870997 5378 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 21:04:11.248241 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:04:11.248143 5378 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-5c45e5-openshift-default-9dcdd54d9-s697g" event={"ID":"23f08320-e0a1-42bc-8fd1-de3492b59bca","Type":"ContainerStarted","Data":"565200d98f097d3eda2ed71305f48ca3f2478d53f968badf2cb34cdaa05da40a"} Oct 30 21:04:11.248241 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:04:11.248199 5378 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-5c45e5-openshift-default-9dcdd54d9-s697g" event={"ID":"23f08320-e0a1-42bc-8fd1-de3492b59bca","Type":"ContainerStarted","Data":"d502d498f085e301c28891c26418385a3ab54683c5bf0ab01b57a2a67ffbc272"} Oct 30 21:04:11.369326 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:04:11.369270 5378 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w4-5c45e5-openshift-default-9dcdd54d9-s697g" podStartSLOduration=2.369253811 podStartE2EDuration="2.369253811s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:04:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:04:11.367248822 +0000 UTC m=+1541.795362740" watchObservedRunningTime="2026-10-30 21:04:11.369253811 +0000 UTC m=+1541.797367729" Oct 30 21:04:11.436714 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:04:11.436684 5378 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w4-5c45e5-openshift-default-9dcdd54d9-s697g" Oct 30 21:04:11.442306 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:04:11.442279 5378 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w4-5c45e5-openshift-default-9dcdd54d9-s697g" Oct 30 21:04:11.578105 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:04:11.577483 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:04:11.792833 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:04:11.792792 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:04:12.255056 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:04:12.255023 5378 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w4-5c45e5-openshift-default-9dcdd54d9-s697g" Oct 30 21:04:12.256078 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:04:12.256058 5378 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w4-5c45e5-openshift-default-9dcdd54d9-s697g" Oct 30 21:04:14.289788 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:04:14.289753 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:04:14.502293 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:04:14.502251 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:04:14.663196 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:04:14.663159 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:04:17.095559 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:04:17.095525 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:04:17.548872 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:04:17.548785 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:04:17.841276 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:04:17.804066 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:04:18.000596 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:04:17.995480 5378 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w4-5c45e5-76458b97d7-7vrgq"] Oct 30 21:04:18.031600 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:04:18.013878 5378 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-5c45e5-76458b97d7-7vrgq" Oct 30 21:04:18.065303 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:04:18.065264 5378 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w4-5c45e5-76458b97d7-7vrgq"] Oct 30 21:04:18.108613 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:04:18.106600 5378 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w4-5c45e5\"" Oct 30 21:04:18.108613 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:04:18.106839 5378 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w4-5c45e5-dockercfg-bhs65\"" Oct 30 21:04:18.132589 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:04:18.132093 5378 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gzqg7\" (UniqueName: \"kubernetes.io/projected/52b8c06c-c948-452c-b317-523f09618599-kube-api-access-gzqg7\") pod \"payload-pre-processing-e2e-shared-a-w4-5c45e5-76458b97d7-7vrgq\" (UID: \"52b8c06c-c948-452c-b317-523f09618599\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-5c45e5-76458b97d7-7vrgq" Oct 30 21:04:18.132589 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:04:18.132216 5378 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/52b8c06c-c948-452c-b317-523f09618599-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w4-5c45e5-76458b97d7-7vrgq\" (UID: \"52b8c06c-c948-452c-b317-523f09618599\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-5c45e5-76458b97d7-7vrgq" Oct 30 21:04:18.237706 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:04:18.237667 5378 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w4-5c45e5-7f47b69b66-6ks8n"] Oct 30 21:04:18.374392 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:04:18.321427 5378 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gzqg7\" (UniqueName: \"kubernetes.io/projected/52b8c06c-c948-452c-b317-523f09618599-kube-api-access-gzqg7\") pod \"payload-pre-processing-e2e-shared-a-w4-5c45e5-76458b97d7-7vrgq\" (UID: \"52b8c06c-c948-452c-b317-523f09618599\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-5c45e5-76458b97d7-7vrgq" Oct 30 21:04:18.374392 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:04:18.365702 5378 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/52b8c06c-c948-452c-b317-523f09618599-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w4-5c45e5-76458b97d7-7vrgq\" (UID: \"52b8c06c-c948-452c-b317-523f09618599\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-5c45e5-76458b97d7-7vrgq" Oct 30 21:04:18.374392 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:04:18.371847 5378 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/52b8c06c-c948-452c-b317-523f09618599-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w4-5c45e5-76458b97d7-7vrgq\" (UID: \"52b8c06c-c948-452c-b317-523f09618599\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-5c45e5-76458b97d7-7vrgq" Oct 30 21:04:18.399714 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:04:18.398678 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:04:18.447591 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:04:18.440344 5378 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gzqg7\" (UniqueName: \"kubernetes.io/projected/52b8c06c-c948-452c-b317-523f09618599-kube-api-access-gzqg7\") pod \"payload-pre-processing-e2e-shared-a-w4-5c45e5-76458b97d7-7vrgq\" (UID: \"52b8c06c-c948-452c-b317-523f09618599\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-5c45e5-76458b97d7-7vrgq" Oct 30 21:04:18.499070 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:04:18.481858 5378 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w4-5c45e5-7f47b69b66-6ks8n"] Oct 30 21:04:18.499070 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:04:18.482033 5378 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-5c45e5-7f47b69b66-6ks8n" Oct 30 21:04:18.587435 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:04:18.587400 5378 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a6c2ebb4-9a45-4038-87a1-a40a242ec833-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w4-5c45e5-7f47b69b66-6ks8n\" (UID: \"a6c2ebb4-9a45-4038-87a1-a40a242ec833\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w4-5c45e5-7f47b69b66-6ks8n" Oct 30 21:04:18.587614 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:04:18.587506 5378 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fxs2k\" (UniqueName: \"kubernetes.io/projected/a6c2ebb4-9a45-4038-87a1-a40a242ec833-kube-api-access-fxs2k\") pod \"payload-processing-e2e-shared-a-w4-5c45e5-7f47b69b66-6ks8n\" (UID: \"a6c2ebb4-9a45-4038-87a1-a40a242ec833\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w4-5c45e5-7f47b69b66-6ks8n" Oct 30 21:04:18.680492 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:04:18.680411 5378 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-5c45e5-76458b97d7-7vrgq" Oct 30 21:04:18.688491 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:04:18.688463 5378 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a6c2ebb4-9a45-4038-87a1-a40a242ec833-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w4-5c45e5-7f47b69b66-6ks8n\" (UID: \"a6c2ebb4-9a45-4038-87a1-a40a242ec833\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w4-5c45e5-7f47b69b66-6ks8n" Oct 30 21:04:18.688644 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:04:18.688522 5378 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fxs2k\" (UniqueName: \"kubernetes.io/projected/a6c2ebb4-9a45-4038-87a1-a40a242ec833-kube-api-access-fxs2k\") pod \"payload-processing-e2e-shared-a-w4-5c45e5-7f47b69b66-6ks8n\" (UID: \"a6c2ebb4-9a45-4038-87a1-a40a242ec833\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w4-5c45e5-7f47b69b66-6ks8n" Oct 30 21:04:18.689187 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:04:18.689164 5378 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a6c2ebb4-9a45-4038-87a1-a40a242ec833-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w4-5c45e5-7f47b69b66-6ks8n\" (UID: \"a6c2ebb4-9a45-4038-87a1-a40a242ec833\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w4-5c45e5-7f47b69b66-6ks8n" Oct 30 21:04:18.747945 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:04:18.747881 5378 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fxs2k\" (UniqueName: \"kubernetes.io/projected/a6c2ebb4-9a45-4038-87a1-a40a242ec833-kube-api-access-fxs2k\") pod \"payload-processing-e2e-shared-a-w4-5c45e5-7f47b69b66-6ks8n\" (UID: \"a6c2ebb4-9a45-4038-87a1-a40a242ec833\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w4-5c45e5-7f47b69b66-6ks8n" Oct 30 21:04:18.840910 ip-10-0-138-149 kubenswrapper[5378]: E1030 21:04:18.840871 5378 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q6mx7" containerName="extractor" Oct 30 21:04:18.842240 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:04:18.842221 5378 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-5c45e5-7f47b69b66-6ks8n" Oct 30 21:04:19.066837 ip-10-0-138-149 kubenswrapper[5378]: W1030 21:04:19.066806 5378 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod52b8c06c_c948_452c_b317_523f09618599.slice/crio-19070c021d669e4e82893b06f13f9921cadfdefab97e446ca232e57e1d948602 WatchSource:0}: Error finding container 19070c021d669e4e82893b06f13f9921cadfdefab97e446ca232e57e1d948602: Status 404 returned error can't find the container with id 19070c021d669e4e82893b06f13f9921cadfdefab97e446ca232e57e1d948602 Oct 30 21:04:19.067725 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:04:19.067640 5378 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w4-5c45e5-76458b97d7-7vrgq"] Oct 30 21:04:19.219143 ip-10-0-138-149 kubenswrapper[5378]: W1030 21:04:19.219109 5378 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda6c2ebb4_9a45_4038_87a1_a40a242ec833.slice/crio-48068a32578b21d9960579b977a170f4486d82e66a3f1c257e11207d66d9a69a WatchSource:0}: Error finding container 48068a32578b21d9960579b977a170f4486d82e66a3f1c257e11207d66d9a69a: Status 404 returned error can't find the container with id 48068a32578b21d9960579b977a170f4486d82e66a3f1c257e11207d66d9a69a Oct 30 21:04:19.219628 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:04:19.219512 5378 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w4-5c45e5-7f47b69b66-6ks8n"] Oct 30 21:04:19.410121 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:04:19.410080 5378 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-5c45e5-7f47b69b66-6ks8n" event={"ID":"a6c2ebb4-9a45-4038-87a1-a40a242ec833","Type":"ContainerStarted","Data":"8a92fb2d43ce992c25c5b749b25eb9203767ecfb98b1e4a890a653a405f5963e"} Oct 30 21:04:19.410325 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:04:19.410147 5378 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-5c45e5-7f47b69b66-6ks8n" event={"ID":"a6c2ebb4-9a45-4038-87a1-a40a242ec833","Type":"ContainerStarted","Data":"48068a32578b21d9960579b977a170f4486d82e66a3f1c257e11207d66d9a69a"} Oct 30 21:04:19.410325 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:04:19.410177 5378 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-5c45e5-7f47b69b66-6ks8n" Oct 30 21:04:19.412091 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:04:19.412063 5378 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-5c45e5-76458b97d7-7vrgq" event={"ID":"52b8c06c-c948-452c-b317-523f09618599","Type":"ContainerStarted","Data":"2cab747c206431d8e8afdc24b84ad6a98dcdba1b4de6597508167580089d634a"} Oct 30 21:04:19.412221 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:04:19.412109 5378 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-5c45e5-76458b97d7-7vrgq" event={"ID":"52b8c06c-c948-452c-b317-523f09618599","Type":"ContainerStarted","Data":"19070c021d669e4e82893b06f13f9921cadfdefab97e446ca232e57e1d948602"} Oct 30 21:04:19.412221 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:04:19.412134 5378 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-5c45e5-76458b97d7-7vrgq" Oct 30 21:04:19.531326 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:04:19.531270 5378 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-5c45e5-7f47b69b66-6ks8n" podStartSLOduration=1.531255438 podStartE2EDuration="1.531255438s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:04:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:04:19.527915048 +0000 UTC m=+1549.956028964" watchObservedRunningTime="2026-10-30 21:04:19.531255438 +0000 UTC m=+1549.959369356" Oct 30 21:04:19.657301 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:04:19.657236 5378 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-5c45e5-76458b97d7-7vrgq" podStartSLOduration=2.657216577 podStartE2EDuration="2.657216577s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:04:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:04:19.652510615 +0000 UTC m=+1550.080624535" watchObservedRunningTime="2026-10-30 21:04:19.657216577 +0000 UTC m=+1550.085330495" Oct 30 21:04:22.276706 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:04:22.276666 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:04:26.824232 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:04:26.824199 5378 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-2cf21005-7d898b7f7b-l7vts"] Oct 30 21:04:26.829088 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:04:26.829067 5378 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-2cf21005-7d898b7f7b-l7vts" Oct 30 21:04:26.885942 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:04:26.885910 5378 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-2cf21005-7d898b7f7b-l7vts"] Oct 30 21:04:26.923711 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:04:26.923676 5378 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e132414d-c8ea-4b13-a8d6-639bfe64effe-maas-api-tls\") pod \"maas-api-e2e-ait-2cf21005-7d898b7f7b-l7vts\" (UID: \"e132414d-c8ea-4b13-a8d6-639bfe64effe\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-2cf21005-7d898b7f7b-l7vts" Oct 30 21:04:26.923871 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:04:26.923754 5378 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e132414d-c8ea-4b13-a8d6-639bfe64effe-openshift-service-ca\") pod \"maas-api-e2e-ait-2cf21005-7d898b7f7b-l7vts\" (UID: \"e132414d-c8ea-4b13-a8d6-639bfe64effe\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-2cf21005-7d898b7f7b-l7vts" Oct 30 21:04:26.923921 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:04:26.923890 5378 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w6w9w\" (UniqueName: \"kubernetes.io/projected/e132414d-c8ea-4b13-a8d6-639bfe64effe-kube-api-access-w6w9w\") pod \"maas-api-e2e-ait-2cf21005-7d898b7f7b-l7vts\" (UID: \"e132414d-c8ea-4b13-a8d6-639bfe64effe\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-2cf21005-7d898b7f7b-l7vts" Oct 30 21:04:26.923976 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:04:26.923961 5378 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e132414d-c8ea-4b13-a8d6-639bfe64effe-metrics-certs\") pod \"maas-api-e2e-ait-2cf21005-7d898b7f7b-l7vts\" (UID: \"e132414d-c8ea-4b13-a8d6-639bfe64effe\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-2cf21005-7d898b7f7b-l7vts" Oct 30 21:04:26.931474 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:04:26.931447 5378 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-2cf21005-serving-cert\"" Oct 30 21:04:27.007338 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:04:27.007307 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:04:27.024878 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:04:27.024854 5378 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e132414d-c8ea-4b13-a8d6-639bfe64effe-maas-api-tls\") pod \"maas-api-e2e-ait-2cf21005-7d898b7f7b-l7vts\" (UID: \"e132414d-c8ea-4b13-a8d6-639bfe64effe\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-2cf21005-7d898b7f7b-l7vts" Oct 30 21:04:27.025008 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:04:27.024911 5378 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e132414d-c8ea-4b13-a8d6-639bfe64effe-openshift-service-ca\") pod \"maas-api-e2e-ait-2cf21005-7d898b7f7b-l7vts\" (UID: \"e132414d-c8ea-4b13-a8d6-639bfe64effe\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-2cf21005-7d898b7f7b-l7vts" Oct 30 21:04:27.025008 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:04:27.024963 5378 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w6w9w\" (UniqueName: \"kubernetes.io/projected/e132414d-c8ea-4b13-a8d6-639bfe64effe-kube-api-access-w6w9w\") pod \"maas-api-e2e-ait-2cf21005-7d898b7f7b-l7vts\" (UID: \"e132414d-c8ea-4b13-a8d6-639bfe64effe\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-2cf21005-7d898b7f7b-l7vts" Oct 30 21:04:27.025008 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:04:27.024997 5378 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e132414d-c8ea-4b13-a8d6-639bfe64effe-metrics-certs\") pod \"maas-api-e2e-ait-2cf21005-7d898b7f7b-l7vts\" (UID: \"e132414d-c8ea-4b13-a8d6-639bfe64effe\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-2cf21005-7d898b7f7b-l7vts" Oct 30 21:04:27.025121 ip-10-0-138-149 kubenswrapper[5378]: E1030 21:04:27.024997 5378 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ait-2cf21005-serving-cert: secret "maas-api-e2e-ait-2cf21005-serving-cert" not found Oct 30 21:04:27.025175 ip-10-0-138-149 kubenswrapper[5378]: E1030 21:04:27.025162 5378 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e132414d-c8ea-4b13-a8d6-639bfe64effe-maas-api-tls podName:e132414d-c8ea-4b13-a8d6-639bfe64effe nodeName:}" failed. No retries permitted until 2026-10-30 21:04:27.525140721 +0000 UTC m=+1557.953254636 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/e132414d-c8ea-4b13-a8d6-639bfe64effe-maas-api-tls") pod "maas-api-e2e-ait-2cf21005-7d898b7f7b-l7vts" (UID: "e132414d-c8ea-4b13-a8d6-639bfe64effe") : secret "maas-api-e2e-ait-2cf21005-serving-cert" not found Oct 30 21:04:27.025658 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:04:27.025631 5378 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e132414d-c8ea-4b13-a8d6-639bfe64effe-openshift-service-ca\") pod \"maas-api-e2e-ait-2cf21005-7d898b7f7b-l7vts\" (UID: \"e132414d-c8ea-4b13-a8d6-639bfe64effe\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-2cf21005-7d898b7f7b-l7vts" Oct 30 21:04:27.027400 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:04:27.027374 5378 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e132414d-c8ea-4b13-a8d6-639bfe64effe-metrics-certs\") pod \"maas-api-e2e-ait-2cf21005-7d898b7f7b-l7vts\" (UID: \"e132414d-c8ea-4b13-a8d6-639bfe64effe\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-2cf21005-7d898b7f7b-l7vts" Oct 30 21:04:27.129266 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:04:27.129232 5378 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w6w9w\" (UniqueName: \"kubernetes.io/projected/e132414d-c8ea-4b13-a8d6-639bfe64effe-kube-api-access-w6w9w\") pod \"maas-api-e2e-ait-2cf21005-7d898b7f7b-l7vts\" (UID: \"e132414d-c8ea-4b13-a8d6-639bfe64effe\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-2cf21005-7d898b7f7b-l7vts" Oct 30 21:04:27.173828 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:04:27.173786 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:04:27.330020 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:04:27.329976 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:04:27.532434 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:04:27.532335 5378 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e132414d-c8ea-4b13-a8d6-639bfe64effe-maas-api-tls\") pod \"maas-api-e2e-ait-2cf21005-7d898b7f7b-l7vts\" (UID: \"e132414d-c8ea-4b13-a8d6-639bfe64effe\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-2cf21005-7d898b7f7b-l7vts" Oct 30 21:04:27.534976 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:04:27.534948 5378 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e132414d-c8ea-4b13-a8d6-639bfe64effe-maas-api-tls\") pod \"maas-api-e2e-ait-2cf21005-7d898b7f7b-l7vts\" (UID: \"e132414d-c8ea-4b13-a8d6-639bfe64effe\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-2cf21005-7d898b7f7b-l7vts" Oct 30 21:04:27.745849 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:04:27.745817 5378 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-2cf21005-7d898b7f7b-l7vts" Oct 30 21:04:28.127731 ip-10-0-138-149 kubenswrapper[5378]: W1030 21:04:28.127700 5378 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode132414d_c8ea_4b13_a8d6_639bfe64effe.slice/crio-34478f3ae8c93947cd353445469cea698d0a9925576c1e23c41fb1596a72880f WatchSource:0}: Error finding container 34478f3ae8c93947cd353445469cea698d0a9925576c1e23c41fb1596a72880f: Status 404 returned error can't find the container with id 34478f3ae8c93947cd353445469cea698d0a9925576c1e23c41fb1596a72880f Oct 30 21:04:28.128217 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:04:28.128169 5378 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-2cf21005-7d898b7f7b-l7vts"] Oct 30 21:04:28.499425 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:04:28.499332 5378 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-2cf21005-7d898b7f7b-l7vts" event={"ID":"e132414d-c8ea-4b13-a8d6-639bfe64effe","Type":"ContainerStarted","Data":"34478f3ae8c93947cd353445469cea698d0a9925576c1e23c41fb1596a72880f"} Oct 30 21:04:29.469332 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:04:29.469299 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:04:29.508639 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:04:29.508604 5378 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-2cf21005-7d898b7f7b-l7vts" event={"ID":"e132414d-c8ea-4b13-a8d6-639bfe64effe","Type":"ContainerStarted","Data":"f67c442517d5b30da76988ff53f6b38fece6aea343fdf703ea62e9787f31e1ad"} Oct 30 21:04:29.508830 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:04:29.508735 5378 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-2cf21005-7d898b7f7b-l7vts" Oct 30 21:04:29.629986 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:04:29.629939 5378 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-2cf21005-7d898b7f7b-l7vts" podStartSLOduration=2.7023364020000002 podStartE2EDuration="3.629923768s" totalImagesPullingTime="927.587366ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:04:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:04:29.628548019 +0000 UTC m=+1560.056661940" watchObservedRunningTime="2026-10-30 21:04:29.629923768 +0000 UTC m=+1560.058037687" Oct 30 21:04:30.420654 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:04:30.417239 5378 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w2-de53c2-openshift-default-58f78dcbb7-lxxs8"] Oct 30 21:04:30.479637 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:04:30.474389 5378 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-5c45e5-7f47b69b66-6ks8n" Oct 30 21:04:30.479637 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:04:30.474460 5378 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-5c45e5-76458b97d7-7vrgq" Oct 30 21:04:30.479637 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:04:30.475511 5378 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-de53c2-openshift-default-58f78dcbb7-lxxs8" Oct 30 21:04:30.487591 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:04:30.487538 5378 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-de53c2-openshift-default-58f78dcbb7-lxxs8"] Oct 30 21:04:30.509297 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:04:30.509259 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:04:30.529675 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:04:30.529646 5378 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w2-de53c2-openshift-default-dockercfg-hmt7b\"" Oct 30 21:04:30.617973 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:04:30.617928 5378 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a5fd3bfb-32af-4284-9bed-e0e55a9dacd3-workload-certs\") pod \"e2e-shared-a-w2-de53c2-openshift-default-58f78dcbb7-lxxs8\" (UID: \"a5fd3bfb-32af-4284-9bed-e0e55a9dacd3\") " pod="openshift-ingress/e2e-shared-a-w2-de53c2-openshift-default-58f78dcbb7-lxxs8" Oct 30 21:04:30.618177 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:04:30.618048 5378 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a5fd3bfb-32af-4284-9bed-e0e55a9dacd3-workload-socket\") pod \"e2e-shared-a-w2-de53c2-openshift-default-58f78dcbb7-lxxs8\" (UID: \"a5fd3bfb-32af-4284-9bed-e0e55a9dacd3\") " pod="openshift-ingress/e2e-shared-a-w2-de53c2-openshift-default-58f78dcbb7-lxxs8" Oct 30 21:04:30.618177 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:04:30.618089 5378 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a5fd3bfb-32af-4284-9bed-e0e55a9dacd3-istiod-ca-cert\") pod \"e2e-shared-a-w2-de53c2-openshift-default-58f78dcbb7-lxxs8\" (UID: \"a5fd3bfb-32af-4284-9bed-e0e55a9dacd3\") " pod="openshift-ingress/e2e-shared-a-w2-de53c2-openshift-default-58f78dcbb7-lxxs8" Oct 30 21:04:30.618177 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:04:30.618129 5378 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a5fd3bfb-32af-4284-9bed-e0e55a9dacd3-istio-token\") pod \"e2e-shared-a-w2-de53c2-openshift-default-58f78dcbb7-lxxs8\" (UID: \"a5fd3bfb-32af-4284-9bed-e0e55a9dacd3\") " pod="openshift-ingress/e2e-shared-a-w2-de53c2-openshift-default-58f78dcbb7-lxxs8" Oct 30 21:04:30.618329 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:04:30.618220 5378 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a5fd3bfb-32af-4284-9bed-e0e55a9dacd3-credential-socket\") pod \"e2e-shared-a-w2-de53c2-openshift-default-58f78dcbb7-lxxs8\" (UID: \"a5fd3bfb-32af-4284-9bed-e0e55a9dacd3\") " pod="openshift-ingress/e2e-shared-a-w2-de53c2-openshift-default-58f78dcbb7-lxxs8" Oct 30 21:04:30.618469 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:04:30.618445 5378 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r5mfq\" (UniqueName: \"kubernetes.io/projected/a5fd3bfb-32af-4284-9bed-e0e55a9dacd3-kube-api-access-r5mfq\") pod \"e2e-shared-a-w2-de53c2-openshift-default-58f78dcbb7-lxxs8\" (UID: \"a5fd3bfb-32af-4284-9bed-e0e55a9dacd3\") " pod="openshift-ingress/e2e-shared-a-w2-de53c2-openshift-default-58f78dcbb7-lxxs8" Oct 30 21:04:30.618600 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:04:30.618585 5378 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a5fd3bfb-32af-4284-9bed-e0e55a9dacd3-istio-data\") pod \"e2e-shared-a-w2-de53c2-openshift-default-58f78dcbb7-lxxs8\" (UID: \"a5fd3bfb-32af-4284-9bed-e0e55a9dacd3\") " pod="openshift-ingress/e2e-shared-a-w2-de53c2-openshift-default-58f78dcbb7-lxxs8" Oct 30 21:04:30.618695 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:04:30.618672 5378 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a5fd3bfb-32af-4284-9bed-e0e55a9dacd3-istio-podinfo\") pod \"e2e-shared-a-w2-de53c2-openshift-default-58f78dcbb7-lxxs8\" (UID: \"a5fd3bfb-32af-4284-9bed-e0e55a9dacd3\") " pod="openshift-ingress/e2e-shared-a-w2-de53c2-openshift-default-58f78dcbb7-lxxs8" Oct 30 21:04:30.618868 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:04:30.618847 5378 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a5fd3bfb-32af-4284-9bed-e0e55a9dacd3-istio-envoy\") pod \"e2e-shared-a-w2-de53c2-openshift-default-58f78dcbb7-lxxs8\" (UID: \"a5fd3bfb-32af-4284-9bed-e0e55a9dacd3\") " pod="openshift-ingress/e2e-shared-a-w2-de53c2-openshift-default-58f78dcbb7-lxxs8" Oct 30 21:04:30.709237 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:04:30.709150 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:04:30.730096 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:04:30.730047 5378 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a5fd3bfb-32af-4284-9bed-e0e55a9dacd3-istio-podinfo\") pod \"e2e-shared-a-w2-de53c2-openshift-default-58f78dcbb7-lxxs8\" (UID: \"a5fd3bfb-32af-4284-9bed-e0e55a9dacd3\") " pod="openshift-ingress/e2e-shared-a-w2-de53c2-openshift-default-58f78dcbb7-lxxs8" Oct 30 21:04:30.730335 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:04:30.730209 5378 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a5fd3bfb-32af-4284-9bed-e0e55a9dacd3-istio-envoy\") pod \"e2e-shared-a-w2-de53c2-openshift-default-58f78dcbb7-lxxs8\" (UID: \"a5fd3bfb-32af-4284-9bed-e0e55a9dacd3\") " pod="openshift-ingress/e2e-shared-a-w2-de53c2-openshift-default-58f78dcbb7-lxxs8" Oct 30 21:04:30.730404 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:04:30.730376 5378 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a5fd3bfb-32af-4284-9bed-e0e55a9dacd3-workload-certs\") pod \"e2e-shared-a-w2-de53c2-openshift-default-58f78dcbb7-lxxs8\" (UID: \"a5fd3bfb-32af-4284-9bed-e0e55a9dacd3\") " pod="openshift-ingress/e2e-shared-a-w2-de53c2-openshift-default-58f78dcbb7-lxxs8" Oct 30 21:04:30.730471 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:04:30.730451 5378 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a5fd3bfb-32af-4284-9bed-e0e55a9dacd3-workload-socket\") pod \"e2e-shared-a-w2-de53c2-openshift-default-58f78dcbb7-lxxs8\" (UID: \"a5fd3bfb-32af-4284-9bed-e0e55a9dacd3\") " pod="openshift-ingress/e2e-shared-a-w2-de53c2-openshift-default-58f78dcbb7-lxxs8" Oct 30 21:04:30.730540 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:04:30.730489 5378 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a5fd3bfb-32af-4284-9bed-e0e55a9dacd3-istiod-ca-cert\") pod \"e2e-shared-a-w2-de53c2-openshift-default-58f78dcbb7-lxxs8\" (UID: \"a5fd3bfb-32af-4284-9bed-e0e55a9dacd3\") " pod="openshift-ingress/e2e-shared-a-w2-de53c2-openshift-default-58f78dcbb7-lxxs8" Oct 30 21:04:30.730540 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:04:30.730531 5378 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a5fd3bfb-32af-4284-9bed-e0e55a9dacd3-istio-token\") pod \"e2e-shared-a-w2-de53c2-openshift-default-58f78dcbb7-lxxs8\" (UID: \"a5fd3bfb-32af-4284-9bed-e0e55a9dacd3\") " pod="openshift-ingress/e2e-shared-a-w2-de53c2-openshift-default-58f78dcbb7-lxxs8" Oct 30 21:04:30.730703 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:04:30.730641 5378 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a5fd3bfb-32af-4284-9bed-e0e55a9dacd3-credential-socket\") pod \"e2e-shared-a-w2-de53c2-openshift-default-58f78dcbb7-lxxs8\" (UID: \"a5fd3bfb-32af-4284-9bed-e0e55a9dacd3\") " pod="openshift-ingress/e2e-shared-a-w2-de53c2-openshift-default-58f78dcbb7-lxxs8" Oct 30 21:04:30.730703 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:04:30.730686 5378 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r5mfq\" (UniqueName: \"kubernetes.io/projected/a5fd3bfb-32af-4284-9bed-e0e55a9dacd3-kube-api-access-r5mfq\") pod \"e2e-shared-a-w2-de53c2-openshift-default-58f78dcbb7-lxxs8\" (UID: \"a5fd3bfb-32af-4284-9bed-e0e55a9dacd3\") " pod="openshift-ingress/e2e-shared-a-w2-de53c2-openshift-default-58f78dcbb7-lxxs8" Oct 30 21:04:30.730813 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:04:30.730768 5378 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a5fd3bfb-32af-4284-9bed-e0e55a9dacd3-istio-data\") pod \"e2e-shared-a-w2-de53c2-openshift-default-58f78dcbb7-lxxs8\" (UID: \"a5fd3bfb-32af-4284-9bed-e0e55a9dacd3\") " pod="openshift-ingress/e2e-shared-a-w2-de53c2-openshift-default-58f78dcbb7-lxxs8" Oct 30 21:04:30.769114 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:04:30.766952 5378 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a5fd3bfb-32af-4284-9bed-e0e55a9dacd3-credential-socket\") pod \"e2e-shared-a-w2-de53c2-openshift-default-58f78dcbb7-lxxs8\" (UID: \"a5fd3bfb-32af-4284-9bed-e0e55a9dacd3\") " pod="openshift-ingress/e2e-shared-a-w2-de53c2-openshift-default-58f78dcbb7-lxxs8" Oct 30 21:04:30.769869 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:04:30.769750 5378 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a5fd3bfb-32af-4284-9bed-e0e55a9dacd3-workload-certs\") pod \"e2e-shared-a-w2-de53c2-openshift-default-58f78dcbb7-lxxs8\" (UID: \"a5fd3bfb-32af-4284-9bed-e0e55a9dacd3\") " pod="openshift-ingress/e2e-shared-a-w2-de53c2-openshift-default-58f78dcbb7-lxxs8" Oct 30 21:04:30.770817 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:04:30.770790 5378 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a5fd3bfb-32af-4284-9bed-e0e55a9dacd3-istiod-ca-cert\") pod \"e2e-shared-a-w2-de53c2-openshift-default-58f78dcbb7-lxxs8\" (UID: \"a5fd3bfb-32af-4284-9bed-e0e55a9dacd3\") " pod="openshift-ingress/e2e-shared-a-w2-de53c2-openshift-default-58f78dcbb7-lxxs8" Oct 30 21:04:30.774164 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:04:30.771994 5378 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a5fd3bfb-32af-4284-9bed-e0e55a9dacd3-istio-data\") pod \"e2e-shared-a-w2-de53c2-openshift-default-58f78dcbb7-lxxs8\" (UID: \"a5fd3bfb-32af-4284-9bed-e0e55a9dacd3\") " pod="openshift-ingress/e2e-shared-a-w2-de53c2-openshift-default-58f78dcbb7-lxxs8" Oct 30 21:04:30.774164 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:04:30.772355 5378 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a5fd3bfb-32af-4284-9bed-e0e55a9dacd3-workload-socket\") pod \"e2e-shared-a-w2-de53c2-openshift-default-58f78dcbb7-lxxs8\" (UID: \"a5fd3bfb-32af-4284-9bed-e0e55a9dacd3\") " pod="openshift-ingress/e2e-shared-a-w2-de53c2-openshift-default-58f78dcbb7-lxxs8" Oct 30 21:04:30.839806 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:04:30.836975 5378 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a5fd3bfb-32af-4284-9bed-e0e55a9dacd3-istio-token\") pod \"e2e-shared-a-w2-de53c2-openshift-default-58f78dcbb7-lxxs8\" (UID: \"a5fd3bfb-32af-4284-9bed-e0e55a9dacd3\") " pod="openshift-ingress/e2e-shared-a-w2-de53c2-openshift-default-58f78dcbb7-lxxs8" Oct 30 21:04:30.843010 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:04:30.842979 5378 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a5fd3bfb-32af-4284-9bed-e0e55a9dacd3-istio-podinfo\") pod \"e2e-shared-a-w2-de53c2-openshift-default-58f78dcbb7-lxxs8\" (UID: \"a5fd3bfb-32af-4284-9bed-e0e55a9dacd3\") " pod="openshift-ingress/e2e-shared-a-w2-de53c2-openshift-default-58f78dcbb7-lxxs8" Oct 30 21:04:30.844672 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:04:30.844638 5378 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a5fd3bfb-32af-4284-9bed-e0e55a9dacd3-istio-envoy\") pod \"e2e-shared-a-w2-de53c2-openshift-default-58f78dcbb7-lxxs8\" (UID: \"a5fd3bfb-32af-4284-9bed-e0e55a9dacd3\") " pod="openshift-ingress/e2e-shared-a-w2-de53c2-openshift-default-58f78dcbb7-lxxs8" Oct 30 21:04:30.883752 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:04:30.883686 5378 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r5mfq\" (UniqueName: \"kubernetes.io/projected/a5fd3bfb-32af-4284-9bed-e0e55a9dacd3-kube-api-access-r5mfq\") pod \"e2e-shared-a-w2-de53c2-openshift-default-58f78dcbb7-lxxs8\" (UID: \"a5fd3bfb-32af-4284-9bed-e0e55a9dacd3\") " pod="openshift-ingress/e2e-shared-a-w2-de53c2-openshift-default-58f78dcbb7-lxxs8" Oct 30 21:04:31.121974 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:04:31.114549 5378 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-de53c2-openshift-default-58f78dcbb7-lxxs8" Oct 30 21:04:31.570700 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:04:31.570670 5378 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-de53c2-openshift-default-58f78dcbb7-lxxs8"] Oct 30 21:04:31.571312 ip-10-0-138-149 kubenswrapper[5378]: W1030 21:04:31.571248 5378 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda5fd3bfb_32af_4284_9bed_e0e55a9dacd3.slice/crio-edc1a3f617cdfdf7d015bbf35f946a161f2e29a7ba9a97f9b321a0fee14d142b WatchSource:0}: Error finding container edc1a3f617cdfdf7d015bbf35f946a161f2e29a7ba9a97f9b321a0fee14d142b: Status 404 returned error can't find the container with id edc1a3f617cdfdf7d015bbf35f946a161f2e29a7ba9a97f9b321a0fee14d142b Oct 30 21:04:31.574512 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:04:31.574471 5378 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 21:04:31.574627 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:04:31.574559 5378 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 21:04:31.574675 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:04:31.574624 5378 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 21:04:32.104480 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:04:32.104441 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:04:32.359634 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:04:32.359312 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:04:32.533433 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:04:32.533394 5378 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-de53c2-openshift-default-58f78dcbb7-lxxs8" event={"ID":"a5fd3bfb-32af-4284-9bed-e0e55a9dacd3","Type":"ContainerStarted","Data":"36d9a82525d731c9a76b2b3ef95df6120d9c572ec2ea1468eb7a27de25648b71"} Oct 30 21:04:32.533670 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:04:32.533457 5378 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-de53c2-openshift-default-58f78dcbb7-lxxs8" event={"ID":"a5fd3bfb-32af-4284-9bed-e0e55a9dacd3","Type":"ContainerStarted","Data":"edc1a3f617cdfdf7d015bbf35f946a161f2e29a7ba9a97f9b321a0fee14d142b"} Oct 30 21:04:32.618901 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:04:32.618800 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:04:32.666729 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:04:32.666674 5378 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w2-de53c2-openshift-default-58f78dcbb7-lxxs8" podStartSLOduration=2.666659797 podStartE2EDuration="2.666659797s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:04:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:04:32.66346323 +0000 UTC m=+1563.091577152" watchObservedRunningTime="2026-10-30 21:04:32.666659797 +0000 UTC m=+1563.094773716" Oct 30 21:04:32.822602 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:04:32.822546 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:04:33.115888 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:04:33.115858 5378 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w2-de53c2-openshift-default-58f78dcbb7-lxxs8" Oct 30 21:04:33.122048 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:04:33.122023 5378 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w2-de53c2-openshift-default-58f78dcbb7-lxxs8" Oct 30 21:04:33.470181 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:04:33.470100 5378 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-19db3d5f-openshift-default-7f4c69b8fc-vjtzt"] Oct 30 21:04:33.475626 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:04:33.475599 5378 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-19db3d5f-openshift-default-7f4c69b8fc-vjtzt" Oct 30 21:04:33.521712 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:04:33.520649 5378 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0406854a-c204-4495-bdc8-23afaad6457f-istio-podinfo\") pod \"e2e-mt-19db3d5f-openshift-default-7f4c69b8fc-vjtzt\" (UID: \"0406854a-c204-4495-bdc8-23afaad6457f\") " pod="openshift-ingress/e2e-mt-19db3d5f-openshift-default-7f4c69b8fc-vjtzt" Oct 30 21:04:33.521712 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:04:33.520697 5378 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0406854a-c204-4495-bdc8-23afaad6457f-istiod-ca-cert\") pod \"e2e-mt-19db3d5f-openshift-default-7f4c69b8fc-vjtzt\" (UID: \"0406854a-c204-4495-bdc8-23afaad6457f\") " pod="openshift-ingress/e2e-mt-19db3d5f-openshift-default-7f4c69b8fc-vjtzt" Oct 30 21:04:33.521712 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:04:33.520724 5378 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0406854a-c204-4495-bdc8-23afaad6457f-istio-envoy\") pod \"e2e-mt-19db3d5f-openshift-default-7f4c69b8fc-vjtzt\" (UID: \"0406854a-c204-4495-bdc8-23afaad6457f\") " pod="openshift-ingress/e2e-mt-19db3d5f-openshift-default-7f4c69b8fc-vjtzt" Oct 30 21:04:33.521712 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:04:33.520745 5378 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0406854a-c204-4495-bdc8-23afaad6457f-istio-token\") pod \"e2e-mt-19db3d5f-openshift-default-7f4c69b8fc-vjtzt\" (UID: \"0406854a-c204-4495-bdc8-23afaad6457f\") " pod="openshift-ingress/e2e-mt-19db3d5f-openshift-default-7f4c69b8fc-vjtzt" Oct 30 21:04:33.521712 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:04:33.520796 5378 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f6szh\" (UniqueName: \"kubernetes.io/projected/0406854a-c204-4495-bdc8-23afaad6457f-kube-api-access-f6szh\") pod \"e2e-mt-19db3d5f-openshift-default-7f4c69b8fc-vjtzt\" (UID: \"0406854a-c204-4495-bdc8-23afaad6457f\") " pod="openshift-ingress/e2e-mt-19db3d5f-openshift-default-7f4c69b8fc-vjtzt" Oct 30 21:04:33.521712 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:04:33.520837 5378 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0406854a-c204-4495-bdc8-23afaad6457f-workload-certs\") pod \"e2e-mt-19db3d5f-openshift-default-7f4c69b8fc-vjtzt\" (UID: \"0406854a-c204-4495-bdc8-23afaad6457f\") " pod="openshift-ingress/e2e-mt-19db3d5f-openshift-default-7f4c69b8fc-vjtzt" Oct 30 21:04:33.521712 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:04:33.520891 5378 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0406854a-c204-4495-bdc8-23afaad6457f-istio-data\") pod \"e2e-mt-19db3d5f-openshift-default-7f4c69b8fc-vjtzt\" (UID: \"0406854a-c204-4495-bdc8-23afaad6457f\") " pod="openshift-ingress/e2e-mt-19db3d5f-openshift-default-7f4c69b8fc-vjtzt" Oct 30 21:04:33.521712 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:04:33.521098 5378 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0406854a-c204-4495-bdc8-23afaad6457f-workload-socket\") pod \"e2e-mt-19db3d5f-openshift-default-7f4c69b8fc-vjtzt\" (UID: \"0406854a-c204-4495-bdc8-23afaad6457f\") " pod="openshift-ingress/e2e-mt-19db3d5f-openshift-default-7f4c69b8fc-vjtzt" Oct 30 21:04:33.521712 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:04:33.521158 5378 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0406854a-c204-4495-bdc8-23afaad6457f-credential-socket\") pod \"e2e-mt-19db3d5f-openshift-default-7f4c69b8fc-vjtzt\" (UID: \"0406854a-c204-4495-bdc8-23afaad6457f\") " pod="openshift-ingress/e2e-mt-19db3d5f-openshift-default-7f4c69b8fc-vjtzt" Oct 30 21:04:33.549601 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:04:33.546883 5378 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-19db3d5f-openshift-default-dockercfg-xfw9p\"" Oct 30 21:04:33.569588 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:04:33.568556 5378 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-19db3d5f-openshift-default-7f4c69b8fc-vjtzt"] Oct 30 21:04:33.632591 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:04:33.632498 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:04:33.657014 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:04:33.646604 5378 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0406854a-c204-4495-bdc8-23afaad6457f-credential-socket\") pod \"e2e-mt-19db3d5f-openshift-default-7f4c69b8fc-vjtzt\" (UID: \"0406854a-c204-4495-bdc8-23afaad6457f\") " pod="openshift-ingress/e2e-mt-19db3d5f-openshift-default-7f4c69b8fc-vjtzt" Oct 30 21:04:33.677618 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:04:33.663888 5378 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0406854a-c204-4495-bdc8-23afaad6457f-istio-podinfo\") pod \"e2e-mt-19db3d5f-openshift-default-7f4c69b8fc-vjtzt\" (UID: \"0406854a-c204-4495-bdc8-23afaad6457f\") " pod="openshift-ingress/e2e-mt-19db3d5f-openshift-default-7f4c69b8fc-vjtzt" Oct 30 21:04:33.677895 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:04:33.677867 5378 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0406854a-c204-4495-bdc8-23afaad6457f-istiod-ca-cert\") pod \"e2e-mt-19db3d5f-openshift-default-7f4c69b8fc-vjtzt\" (UID: \"0406854a-c204-4495-bdc8-23afaad6457f\") " pod="openshift-ingress/e2e-mt-19db3d5f-openshift-default-7f4c69b8fc-vjtzt" Oct 30 21:04:33.678002 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:04:33.677986 5378 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0406854a-c204-4495-bdc8-23afaad6457f-istio-envoy\") pod \"e2e-mt-19db3d5f-openshift-default-7f4c69b8fc-vjtzt\" (UID: \"0406854a-c204-4495-bdc8-23afaad6457f\") " pod="openshift-ingress/e2e-mt-19db3d5f-openshift-default-7f4c69b8fc-vjtzt" Oct 30 21:04:33.682594 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:04:33.678083 5378 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0406854a-c204-4495-bdc8-23afaad6457f-istio-token\") pod \"e2e-mt-19db3d5f-openshift-default-7f4c69b8fc-vjtzt\" (UID: \"0406854a-c204-4495-bdc8-23afaad6457f\") " pod="openshift-ingress/e2e-mt-19db3d5f-openshift-default-7f4c69b8fc-vjtzt" Oct 30 21:04:33.687636 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:04:33.682832 5378 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f6szh\" (UniqueName: \"kubernetes.io/projected/0406854a-c204-4495-bdc8-23afaad6457f-kube-api-access-f6szh\") pod \"e2e-mt-19db3d5f-openshift-default-7f4c69b8fc-vjtzt\" (UID: \"0406854a-c204-4495-bdc8-23afaad6457f\") " pod="openshift-ingress/e2e-mt-19db3d5f-openshift-default-7f4c69b8fc-vjtzt" Oct 30 21:04:33.725822 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:04:33.725727 5378 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0406854a-c204-4495-bdc8-23afaad6457f-workload-certs\") pod \"e2e-mt-19db3d5f-openshift-default-7f4c69b8fc-vjtzt\" (UID: \"0406854a-c204-4495-bdc8-23afaad6457f\") " pod="openshift-ingress/e2e-mt-19db3d5f-openshift-default-7f4c69b8fc-vjtzt" Oct 30 21:04:33.727743 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:04:33.727689 5378 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0406854a-c204-4495-bdc8-23afaad6457f-istio-data\") pod \"e2e-mt-19db3d5f-openshift-default-7f4c69b8fc-vjtzt\" (UID: \"0406854a-c204-4495-bdc8-23afaad6457f\") " pod="openshift-ingress/e2e-mt-19db3d5f-openshift-default-7f4c69b8fc-vjtzt" Oct 30 21:04:33.737376 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:04:33.737342 5378 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0406854a-c204-4495-bdc8-23afaad6457f-workload-socket\") pod \"e2e-mt-19db3d5f-openshift-default-7f4c69b8fc-vjtzt\" (UID: \"0406854a-c204-4495-bdc8-23afaad6457f\") " pod="openshift-ingress/e2e-mt-19db3d5f-openshift-default-7f4c69b8fc-vjtzt" Oct 30 21:04:33.741805 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:04:33.683330 5378 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0406854a-c204-4495-bdc8-23afaad6457f-istiod-ca-cert\") pod \"e2e-mt-19db3d5f-openshift-default-7f4c69b8fc-vjtzt\" (UID: \"0406854a-c204-4495-bdc8-23afaad6457f\") " pod="openshift-ingress/e2e-mt-19db3d5f-openshift-default-7f4c69b8fc-vjtzt" Oct 30 21:04:33.745311 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:04:33.647220 5378 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0406854a-c204-4495-bdc8-23afaad6457f-credential-socket\") pod \"e2e-mt-19db3d5f-openshift-default-7f4c69b8fc-vjtzt\" (UID: \"0406854a-c204-4495-bdc8-23afaad6457f\") " pod="openshift-ingress/e2e-mt-19db3d5f-openshift-default-7f4c69b8fc-vjtzt" Oct 30 21:04:33.745498 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:04:33.728699 5378 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0406854a-c204-4495-bdc8-23afaad6457f-istio-data\") pod \"e2e-mt-19db3d5f-openshift-default-7f4c69b8fc-vjtzt\" (UID: \"0406854a-c204-4495-bdc8-23afaad6457f\") " pod="openshift-ingress/e2e-mt-19db3d5f-openshift-default-7f4c69b8fc-vjtzt" Oct 30 21:04:33.778908 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:04:33.728389 5378 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0406854a-c204-4495-bdc8-23afaad6457f-workload-certs\") pod \"e2e-mt-19db3d5f-openshift-default-7f4c69b8fc-vjtzt\" (UID: \"0406854a-c204-4495-bdc8-23afaad6457f\") " pod="openshift-ingress/e2e-mt-19db3d5f-openshift-default-7f4c69b8fc-vjtzt" Oct 30 21:04:33.791409 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:04:33.744340 5378 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0406854a-c204-4495-bdc8-23afaad6457f-workload-socket\") pod \"e2e-mt-19db3d5f-openshift-default-7f4c69b8fc-vjtzt\" (UID: \"0406854a-c204-4495-bdc8-23afaad6457f\") " pod="openshift-ingress/e2e-mt-19db3d5f-openshift-default-7f4c69b8fc-vjtzt" Oct 30 21:04:33.856514 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:04:33.856475 5378 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0406854a-c204-4495-bdc8-23afaad6457f-istio-token\") pod \"e2e-mt-19db3d5f-openshift-default-7f4c69b8fc-vjtzt\" (UID: \"0406854a-c204-4495-bdc8-23afaad6457f\") " pod="openshift-ingress/e2e-mt-19db3d5f-openshift-default-7f4c69b8fc-vjtzt" Oct 30 21:04:33.881849 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:04:33.857744 5378 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0406854a-c204-4495-bdc8-23afaad6457f-istio-envoy\") pod \"e2e-mt-19db3d5f-openshift-default-7f4c69b8fc-vjtzt\" (UID: \"0406854a-c204-4495-bdc8-23afaad6457f\") " pod="openshift-ingress/e2e-mt-19db3d5f-openshift-default-7f4c69b8fc-vjtzt" Oct 30 21:04:33.923366 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:04:33.916606 5378 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f6szh\" (UniqueName: \"kubernetes.io/projected/0406854a-c204-4495-bdc8-23afaad6457f-kube-api-access-f6szh\") pod \"e2e-mt-19db3d5f-openshift-default-7f4c69b8fc-vjtzt\" (UID: \"0406854a-c204-4495-bdc8-23afaad6457f\") " pod="openshift-ingress/e2e-mt-19db3d5f-openshift-default-7f4c69b8fc-vjtzt" Oct 30 21:04:33.941752 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:04:33.940997 5378 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w2-de53c2-openshift-default-58f78dcbb7-lxxs8" Oct 30 21:04:33.941752 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:04:33.941105 5378 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w2-de53c2-openshift-default-58f78dcbb7-lxxs8" Oct 30 21:04:34.022035 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:04:33.994988 5378 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0406854a-c204-4495-bdc8-23afaad6457f-istio-podinfo\") pod \"e2e-mt-19db3d5f-openshift-default-7f4c69b8fc-vjtzt\" (UID: \"0406854a-c204-4495-bdc8-23afaad6457f\") " pod="openshift-ingress/e2e-mt-19db3d5f-openshift-default-7f4c69b8fc-vjtzt" Oct 30 21:04:34.031389 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:04:34.024215 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:04:34.133683 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:04:34.131183 5378 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-19db3d5f-openshift-default-7f4c69b8fc-vjtzt" Oct 30 21:04:34.637188 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:04:34.633481 5378 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-19db3d5f-openshift-default-7f4c69b8fc-vjtzt"] Oct 30 21:04:34.641843 ip-10-0-138-149 kubenswrapper[5378]: W1030 21:04:34.641788 5378 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0406854a_c204_4495_bdc8_23afaad6457f.slice/crio-49bbf42185959c1f4347f9f36083124555bf7182b71a3df89a875dc329bf792c WatchSource:0}: Error finding container 49bbf42185959c1f4347f9f36083124555bf7182b71a3df89a875dc329bf792c: Status 404 returned error can't find the container with id 49bbf42185959c1f4347f9f36083124555bf7182b71a3df89a875dc329bf792c Oct 30 21:04:34.656937 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:04:34.653956 5378 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 21:04:34.656937 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:04:34.654060 5378 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 21:04:34.656937 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:04:34.654101 5378 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 21:04:34.878717 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:04:34.878633 5378 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-19db3d5f-openshift-default-7f4c69b8fc-vjtzt" event={"ID":"0406854a-c204-4495-bdc8-23afaad6457f","Type":"ContainerStarted","Data":"8793bedb56a074407b42ba1d90aedb3e2ea15f4e71c7cdabf051ca723976299e"} Oct 30 21:04:34.878717 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:04:34.878683 5378 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-19db3d5f-openshift-default-7f4c69b8fc-vjtzt" event={"ID":"0406854a-c204-4495-bdc8-23afaad6457f","Type":"ContainerStarted","Data":"49bbf42185959c1f4347f9f36083124555bf7182b71a3df89a875dc329bf792c"} Oct 30 21:04:35.000145 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:04:35.000070 5378 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-19db3d5f-openshift-default-7f4c69b8fc-vjtzt" podStartSLOduration=2.000032774 podStartE2EDuration="2.000032774s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:04:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:04:34.99672058 +0000 UTC m=+1565.424834500" watchObservedRunningTime="2026-10-30 21:04:35.000032774 +0000 UTC m=+1565.428146720" Oct 30 21:04:35.133639 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:04:35.133128 5378 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-19db3d5f-openshift-default-7f4c69b8fc-vjtzt" Oct 30 21:04:35.136627 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:04:35.135972 5378 patch_prober.go:29] interesting pod/e2e-mt-19db3d5f-openshift-default-7f4c69b8fc-vjtzt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.114:15021/healthz/ready\": dial tcp 10.133.0.114:15021: connect: connection refused" start-of-body= Oct 30 21:04:35.136627 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:04:35.136033 5378 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-19db3d5f-openshift-default-7f4c69b8fc-vjtzt" podUID="0406854a-c204-4495-bdc8-23afaad6457f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.114:15021/healthz/ready\": dial tcp 10.133.0.114:15021: connect: connection refused" Oct 30 21:04:35.595646 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:04:35.595171 5378 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-2cf21005-7d898b7f7b-l7vts" Oct 30 21:04:35.704947 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:04:35.704886 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:04:35.924041 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:04:35.923927 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:04:36.071439 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:04:36.071402 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:04:36.136743 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:04:36.136709 5378 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-19db3d5f-openshift-default-7f4c69b8fc-vjtzt" Oct 30 21:04:36.892490 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:04:36.892461 5378 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-19db3d5f-openshift-default-7f4c69b8fc-vjtzt" Oct 30 21:04:36.893297 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:04:36.893277 5378 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-19db3d5f-openshift-default-7f4c69b8fc-vjtzt" Oct 30 21:04:37.857502 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:04:37.857471 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:04:39.121520 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:04:39.121487 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:04:39.319658 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:04:39.319619 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:04:39.451228 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:04:39.451138 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:04:40.991196 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:04:40.991163 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:04:41.307639 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:04:41.284405 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:04:41.566376 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:04:41.566289 5378 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-de53c2-fdf4f8fc5-8fjrg"] Oct 30 21:04:41.642044 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:04:41.642014 5378 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-de53c2-fdf4f8fc5-8fjrg"] Oct 30 21:04:41.642286 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:04:41.642273 5378 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-de53c2-6dc7f4bf96-b5cbp"] Oct 30 21:04:41.769595 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:04:41.765597 5378 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-de53c2-fdf4f8fc5-8fjrg" Oct 30 21:04:41.941593 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:04:41.908394 5378 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w2-de53c2\"" Oct 30 21:04:41.986548 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:04:41.960463 5378 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w2-de53c2-dockercfg-m54px\"" Oct 30 21:04:42.008666 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:04:42.008631 5378 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4tqb7\" (UniqueName: \"kubernetes.io/projected/4f0b0d35-65b6-4721-8939-8d2d6dbb8fc5-kube-api-access-4tqb7\") pod \"payload-pre-processing-e2e-shared-a-w2-de53c2-fdf4f8fc5-8fjrg\" (UID: \"4f0b0d35-65b6-4721-8939-8d2d6dbb8fc5\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-de53c2-fdf4f8fc5-8fjrg" Oct 30 21:04:42.013421 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:04:42.013233 5378 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4f0b0d35-65b6-4721-8939-8d2d6dbb8fc5-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w2-de53c2-fdf4f8fc5-8fjrg\" (UID: \"4f0b0d35-65b6-4721-8939-8d2d6dbb8fc5\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-de53c2-fdf4f8fc5-8fjrg" Oct 30 21:04:42.021314 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:04:42.021287 5378 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-de53c2-6dc7f4bf96-b5cbp" Oct 30 21:04:42.035714 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:04:42.035690 5378 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-de53c2-6dc7f4bf96-b5cbp"] Oct 30 21:04:42.118886 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:04:42.118852 5378 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d9131396-6236-4a21-8c77-0f78fcc851f8-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w2-de53c2-6dc7f4bf96-b5cbp\" (UID: \"d9131396-6236-4a21-8c77-0f78fcc851f8\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-de53c2-6dc7f4bf96-b5cbp" Oct 30 21:04:42.119046 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:04:42.118953 5378 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fgh58\" (UniqueName: \"kubernetes.io/projected/d9131396-6236-4a21-8c77-0f78fcc851f8-kube-api-access-fgh58\") pod \"payload-processing-e2e-shared-a-w2-de53c2-6dc7f4bf96-b5cbp\" (UID: \"d9131396-6236-4a21-8c77-0f78fcc851f8\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-de53c2-6dc7f4bf96-b5cbp" Oct 30 21:04:42.119046 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:04:42.118996 5378 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4f0b0d35-65b6-4721-8939-8d2d6dbb8fc5-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w2-de53c2-fdf4f8fc5-8fjrg\" (UID: \"4f0b0d35-65b6-4721-8939-8d2d6dbb8fc5\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-de53c2-fdf4f8fc5-8fjrg" Oct 30 21:04:42.119247 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:04:42.119226 5378 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4tqb7\" (UniqueName: \"kubernetes.io/projected/4f0b0d35-65b6-4721-8939-8d2d6dbb8fc5-kube-api-access-4tqb7\") pod \"payload-pre-processing-e2e-shared-a-w2-de53c2-fdf4f8fc5-8fjrg\" (UID: \"4f0b0d35-65b6-4721-8939-8d2d6dbb8fc5\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-de53c2-fdf4f8fc5-8fjrg" Oct 30 21:04:42.119823 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:04:42.119801 5378 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4f0b0d35-65b6-4721-8939-8d2d6dbb8fc5-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w2-de53c2-fdf4f8fc5-8fjrg\" (UID: \"4f0b0d35-65b6-4721-8939-8d2d6dbb8fc5\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-de53c2-fdf4f8fc5-8fjrg" Oct 30 21:04:42.177053 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:04:42.177023 5378 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4tqb7\" (UniqueName: \"kubernetes.io/projected/4f0b0d35-65b6-4721-8939-8d2d6dbb8fc5-kube-api-access-4tqb7\") pod \"payload-pre-processing-e2e-shared-a-w2-de53c2-fdf4f8fc5-8fjrg\" (UID: \"4f0b0d35-65b6-4721-8939-8d2d6dbb8fc5\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-de53c2-fdf4f8fc5-8fjrg" Oct 30 21:04:42.221205 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:04:42.221118 5378 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d9131396-6236-4a21-8c77-0f78fcc851f8-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w2-de53c2-6dc7f4bf96-b5cbp\" (UID: \"d9131396-6236-4a21-8c77-0f78fcc851f8\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-de53c2-6dc7f4bf96-b5cbp" Oct 30 21:04:42.221205 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:04:42.221172 5378 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fgh58\" (UniqueName: \"kubernetes.io/projected/d9131396-6236-4a21-8c77-0f78fcc851f8-kube-api-access-fgh58\") pod \"payload-processing-e2e-shared-a-w2-de53c2-6dc7f4bf96-b5cbp\" (UID: \"d9131396-6236-4a21-8c77-0f78fcc851f8\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-de53c2-6dc7f4bf96-b5cbp" Oct 30 21:04:42.221940 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:04:42.221916 5378 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d9131396-6236-4a21-8c77-0f78fcc851f8-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w2-de53c2-6dc7f4bf96-b5cbp\" (UID: \"d9131396-6236-4a21-8c77-0f78fcc851f8\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-de53c2-6dc7f4bf96-b5cbp" Oct 30 21:04:42.279198 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:04:42.279168 5378 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fgh58\" (UniqueName: \"kubernetes.io/projected/d9131396-6236-4a21-8c77-0f78fcc851f8-kube-api-access-fgh58\") pod \"payload-processing-e2e-shared-a-w2-de53c2-6dc7f4bf96-b5cbp\" (UID: \"d9131396-6236-4a21-8c77-0f78fcc851f8\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-de53c2-6dc7f4bf96-b5cbp" Oct 30 21:04:42.321443 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:04:42.321410 5378 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-de53c2-fdf4f8fc5-8fjrg" Oct 30 21:04:42.339112 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:04:42.339082 5378 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-de53c2-6dc7f4bf96-b5cbp" Oct 30 21:04:42.710610 ip-10-0-138-149 kubenswrapper[5378]: W1030 21:04:42.710579 5378 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4f0b0d35_65b6_4721_8939_8d2d6dbb8fc5.slice/crio-84c1ab1dd2b5659080fda374f62ae1c19e53db0e72977c1c7601565997120985 WatchSource:0}: Error finding container 84c1ab1dd2b5659080fda374f62ae1c19e53db0e72977c1c7601565997120985: Status 404 returned error can't find the container with id 84c1ab1dd2b5659080fda374f62ae1c19e53db0e72977c1c7601565997120985 Oct 30 21:04:42.711792 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:04:42.711765 5378 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-de53c2-fdf4f8fc5-8fjrg"] Oct 30 21:04:42.757072 ip-10-0-138-149 kubenswrapper[5378]: W1030 21:04:42.757043 5378 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd9131396_6236_4a21_8c77_0f78fcc851f8.slice/crio-d0b614b6f2d876c78da6098ef42d28727c39136acda5fd7b03e2190307be4b97 WatchSource:0}: Error finding container d0b614b6f2d876c78da6098ef42d28727c39136acda5fd7b03e2190307be4b97: Status 404 returned error can't find the container with id d0b614b6f2d876c78da6098ef42d28727c39136acda5fd7b03e2190307be4b97 Oct 30 21:04:42.759088 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:04:42.759066 5378 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-de53c2-6dc7f4bf96-b5cbp"] Oct 30 21:04:43.034935 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:04:43.034829 5378 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-de53c2-6dc7f4bf96-b5cbp" event={"ID":"d9131396-6236-4a21-8c77-0f78fcc851f8","Type":"ContainerStarted","Data":"d910afbb1d2150b7cbdb312e1517a33d01df6c475363dfee8ad46c9fbf4f0c17"} Oct 30 21:04:43.034935 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:04:43.034918 5378 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-de53c2-6dc7f4bf96-b5cbp" event={"ID":"d9131396-6236-4a21-8c77-0f78fcc851f8","Type":"ContainerStarted","Data":"d0b614b6f2d876c78da6098ef42d28727c39136acda5fd7b03e2190307be4b97"} Oct 30 21:04:43.035483 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:04:43.034998 5378 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-de53c2-6dc7f4bf96-b5cbp" Oct 30 21:04:43.036878 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:04:43.036844 5378 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-de53c2-fdf4f8fc5-8fjrg" event={"ID":"4f0b0d35-65b6-4721-8939-8d2d6dbb8fc5","Type":"ContainerStarted","Data":"5f4c9210b74cb706168c1d5e32e749971cb46563c40a0c9342f8bed2956cc472"} Oct 30 21:04:43.037012 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:04:43.036904 5378 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-de53c2-fdf4f8fc5-8fjrg" event={"ID":"4f0b0d35-65b6-4721-8939-8d2d6dbb8fc5","Type":"ContainerStarted","Data":"84c1ab1dd2b5659080fda374f62ae1c19e53db0e72977c1c7601565997120985"} Oct 30 21:04:43.037012 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:04:43.036931 5378 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-de53c2-fdf4f8fc5-8fjrg" Oct 30 21:04:43.153275 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:04:43.153212 5378 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-de53c2-6dc7f4bf96-b5cbp" podStartSLOduration=2.153198198 podStartE2EDuration="2.153198198s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:04:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:04:43.150430123 +0000 UTC m=+1573.578544042" watchObservedRunningTime="2026-10-30 21:04:43.153198198 +0000 UTC m=+1573.581312116" Oct 30 21:04:43.299793 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:04:43.293138 5378 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-de53c2-fdf4f8fc5-8fjrg" podStartSLOduration=2.293114413 podStartE2EDuration="2.293114413s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:04:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:04:43.276975977 +0000 UTC m=+1573.705089897" watchObservedRunningTime="2026-10-30 21:04:43.293114413 +0000 UTC m=+1573.721228335" Oct 30 21:04:43.362186 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:04:43.362141 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:04:43.810304 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:04:43.810274 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:04:44.017661 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:04:44.016213 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:04:44.613438 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:04:44.603057 5378 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-19db3d5f-744ff4fb64-blk7k"] Oct 30 21:04:44.613438 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:04:44.610100 5378 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-19db3d5f-744ff4fb64-blk7k" Oct 30 21:04:44.633590 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:04:44.630304 5378 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-19db3d5f-744ff4fb64-blk7k"] Oct 30 21:04:44.703075 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:04:44.688463 5378 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-19db3d5f\"" Oct 30 21:04:44.703075 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:04:44.702669 5378 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0c2654e1-7e67-4697-9613-4fa0aa8aa5d1-plugins-config-volume\") pod \"payload-processing-e2e-mt-19db3d5f-744ff4fb64-blk7k\" (UID: \"0c2654e1-7e67-4697-9613-4fa0aa8aa5d1\") " pod="openshift-ingress/payload-processing-e2e-mt-19db3d5f-744ff4fb64-blk7k" Oct 30 21:04:44.703075 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:04:44.702858 5378 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lbj2k\" (UniqueName: \"kubernetes.io/projected/0c2654e1-7e67-4697-9613-4fa0aa8aa5d1-kube-api-access-lbj2k\") pod \"payload-processing-e2e-mt-19db3d5f-744ff4fb64-blk7k\" (UID: \"0c2654e1-7e67-4697-9613-4fa0aa8aa5d1\") " pod="openshift-ingress/payload-processing-e2e-mt-19db3d5f-744ff4fb64-blk7k" Oct 30 21:04:44.727589 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:04:44.723879 5378 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-mt-19db3d5f-dockercfg-kjvj5\"" Oct 30 21:04:44.822162 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:04:44.822124 5378 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lbj2k\" (UniqueName: \"kubernetes.io/projected/0c2654e1-7e67-4697-9613-4fa0aa8aa5d1-kube-api-access-lbj2k\") pod \"payload-processing-e2e-mt-19db3d5f-744ff4fb64-blk7k\" (UID: \"0c2654e1-7e67-4697-9613-4fa0aa8aa5d1\") " pod="openshift-ingress/payload-processing-e2e-mt-19db3d5f-744ff4fb64-blk7k" Oct 30 21:04:44.829373 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:04:44.829337 5378 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0c2654e1-7e67-4697-9613-4fa0aa8aa5d1-plugins-config-volume\") pod \"payload-processing-e2e-mt-19db3d5f-744ff4fb64-blk7k\" (UID: \"0c2654e1-7e67-4697-9613-4fa0aa8aa5d1\") " pod="openshift-ingress/payload-processing-e2e-mt-19db3d5f-744ff4fb64-blk7k" Oct 30 21:04:44.831337 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:04:44.831307 5378 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0c2654e1-7e67-4697-9613-4fa0aa8aa5d1-plugins-config-volume\") pod \"payload-processing-e2e-mt-19db3d5f-744ff4fb64-blk7k\" (UID: \"0c2654e1-7e67-4697-9613-4fa0aa8aa5d1\") " pod="openshift-ingress/payload-processing-e2e-mt-19db3d5f-744ff4fb64-blk7k" Oct 30 21:04:44.886441 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:04:44.886361 5378 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lbj2k\" (UniqueName: \"kubernetes.io/projected/0c2654e1-7e67-4697-9613-4fa0aa8aa5d1-kube-api-access-lbj2k\") pod \"payload-processing-e2e-mt-19db3d5f-744ff4fb64-blk7k\" (UID: \"0c2654e1-7e67-4697-9613-4fa0aa8aa5d1\") " pod="openshift-ingress/payload-processing-e2e-mt-19db3d5f-744ff4fb64-blk7k" Oct 30 21:04:44.902867 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:04:44.902825 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:04:45.033744 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:04:45.033713 5378 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-19db3d5f-744ff4fb64-blk7k" Oct 30 21:04:45.418615 ip-10-0-138-149 kubenswrapper[5378]: W1030 21:04:45.418580 5378 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0c2654e1_7e67_4697_9613_4fa0aa8aa5d1.slice/crio-d1c05bc2b0b73b7d7262ff3466dd9984175d71a81f8c335f383cad66f6f8eff3 WatchSource:0}: Error finding container d1c05bc2b0b73b7d7262ff3466dd9984175d71a81f8c335f383cad66f6f8eff3: Status 404 returned error can't find the container with id d1c05bc2b0b73b7d7262ff3466dd9984175d71a81f8c335f383cad66f6f8eff3 Oct 30 21:04:45.419376 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:04:45.419353 5378 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-19db3d5f-744ff4fb64-blk7k"] Oct 30 21:04:46.174272 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:04:46.174232 5378 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-19db3d5f-744ff4fb64-blk7k" event={"ID":"0c2654e1-7e67-4697-9613-4fa0aa8aa5d1","Type":"ContainerStarted","Data":"f1e677937bfb755acbf760ddb590dfec3cc3919e2745531f12f31250f772bb79"} Oct 30 21:04:46.174780 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:04:46.174293 5378 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-19db3d5f-744ff4fb64-blk7k" event={"ID":"0c2654e1-7e67-4697-9613-4fa0aa8aa5d1","Type":"ContainerStarted","Data":"d1c05bc2b0b73b7d7262ff3466dd9984175d71a81f8c335f383cad66f6f8eff3"} Oct 30 21:04:46.174780 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:04:46.174329 5378 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-mt-19db3d5f-744ff4fb64-blk7k" Oct 30 21:04:46.292293 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:04:46.292227 5378 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-mt-19db3d5f-744ff4fb64-blk7k" podStartSLOduration=2.292211711 podStartE2EDuration="2.292211711s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:04:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:04:46.288411735 +0000 UTC m=+1576.716525659" watchObservedRunningTime="2026-10-30 21:04:46.292211711 +0000 UTC m=+1576.720325650" Oct 30 21:04:46.589576 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:04:46.587145 5378 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-r9g59"] Oct 30 21:04:48.950000 ip-10-0-138-149 kubenswrapper[5378]: E1030 21:04:48.949836 5378 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q6mx7" containerName="extractor" Oct 30 21:04:49.215752 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:04:49.213016 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:04:50.341830 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:04:50.340649 5378 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-8fa2a7/premium-w6-8fa2a7-kserve-64bf4bf6c-dxrdr" podUID="484cba5f-b415-430a-b52d-50b1293be948" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:04:50.410744 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:04:50.410699 5378 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-1fdaa7/premium-w4-1fdaa7-kserve-55dffbd5cd-gvpxq" podUID="ec282a4b-742f-4382-a296-335dcd2e9636" containerName="main" probeResult="failure" output="Get \"https://10.133.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:04:50.467591 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:04:50.463644 5378 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-8fa2a7/simulator-w6-8fa2a7-kserve-6cbf5bd894-lfpcq" podUID="f409f469-71fc-445b-bdc8-b8e84a4d1dc6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:04:50.467591 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:04:50.464486 5378 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-d061d0/distinct-w2-d061d0-kserve-679c7cd7c4-8czrv" podUID="2c139b9f-c7df-4675-b431-95d4136095e5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:04:50.503591 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:04:50.499654 5378 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-f9f277/simulator-w1-f9f277-kserve-5d9685f85c-2h78x" podUID="dc662e9d-914c-4dfb-a86f-0fb048f7b690" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:04:50.608742 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:04:50.608699 5378 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-1fdaa7/distinct-w4-1fdaa7-kserve-86dc9774b9-6x2gf" podUID="2e162c95-0527-4cb3-85a3-795aae212d29" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 21:04:50.759720 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:04:50.759670 5378 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-f9f277/distinct-2-w1-f9f277-kserve-6c49cf5dcb-h4x74" podUID="9eb6ff54-b77b-453c-a166-988b98cf8f49" containerName="main" probeResult="failure" output="Get \"https://10.133.0.76:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:04:50.798787 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:04:50.798745 5378 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-f9f277/distinct-2-w1-f9f277-kserve-6c49cf5dcb-h4x74" podUID="9eb6ff54-b77b-453c-a166-988b98cf8f49" containerName="main" probeResult="failure" output="Get \"https://10.133.0.76:8000/ready\": context deadline exceeded" Oct 30 21:04:50.799538 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:04:50.799052 5378 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-1fdaa7/simulator-w4-1fdaa7-kserve-9866d468c-fwmsk" podUID="09219832-010e-4ba8-bfb9-5a2884b358e1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:04:50.839689 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:04:50.839644 5378 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-d061d0/simulator-w2-d061d0-kserve-8484f9678b-lnszm" podUID="525fac02-b21c-4dd5-b553-72bcc36062c5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/ready\": context deadline exceeded" Oct 30 21:04:51.341703 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:04:51.341654 5378 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-8fa2a7/premium-w6-8fa2a7-kserve-64bf4bf6c-dxrdr" podUID="484cba5f-b415-430a-b52d-50b1293be948" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/ready\": context deadline exceeded" Oct 30 21:04:51.342465 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:04:51.342439 5378 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-8fa2a7/premium-w6-8fa2a7-kserve-64bf4bf6c-dxrdr" Oct 30 21:04:51.413638 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:04:51.412745 5378 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-1fdaa7/premium-w4-1fdaa7-kserve-55dffbd5cd-gvpxq" podUID="ec282a4b-742f-4382-a296-335dcd2e9636" containerName="main" probeResult="failure" output="Get \"https://10.133.0.59:8000/ready\": context deadline exceeded" Oct 30 21:04:51.413638 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:04:51.413586 5378 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-1fdaa7/premium-w4-1fdaa7-kserve-55dffbd5cd-gvpxq" Oct 30 21:04:51.434016 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:04:51.433656 5378 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-781058/distinct-w0-781058-kserve-8bdf9bfcd-hzdfk" podUID="a9678fc2-1c56-4d45-9007-eee5049e0f48" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:04:51.483590 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:04:51.474165 5378 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-d061d0/distinct-w2-d061d0-kserve-679c7cd7c4-8czrv" podUID="2c139b9f-c7df-4675-b431-95d4136095e5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:04:51.483590 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:04:51.474673 5378 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-d061d0/distinct-w2-d061d0-kserve-679c7cd7c4-8czrv" Oct 30 21:04:51.483590 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:04:51.477026 5378 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-8fa2a7/simulator-w6-8fa2a7-kserve-6cbf5bd894-lfpcq" podUID="f409f469-71fc-445b-bdc8-b8e84a4d1dc6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:04:51.483590 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:04:51.477415 5378 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-8fa2a7/simulator-w6-8fa2a7-kserve-6cbf5bd894-lfpcq" Oct 30 21:04:51.726343 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:04:51.663699 5378 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-f9f277/simulator-w1-f9f277-kserve-5d9685f85c-2h78x" podUID="dc662e9d-914c-4dfb-a86f-0fb048f7b690" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": context deadline exceeded" Oct 30 21:04:51.726343 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:04:51.724774 5378 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-f9f277/simulator-w1-f9f277-kserve-5d9685f85c-2h78x" Oct 30 21:04:51.741930 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:04:51.741887 5378 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-1fdaa7/distinct-w4-1fdaa7-kserve-86dc9774b9-6x2gf" podUID="2e162c95-0527-4cb3-85a3-795aae212d29" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:04:51.798141 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:04:51.798074 5378 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-1fdaa7/distinct-w4-1fdaa7-kserve-86dc9774b9-6x2gf" Oct 30 21:04:51.806596 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:04:51.805834 5378 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-1fdaa7/simulator-w4-1fdaa7-kserve-9866d468c-fwmsk" podUID="09219832-010e-4ba8-bfb9-5a2884b358e1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:04:51.806596 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:04:51.806245 5378 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-1fdaa7/simulator-w4-1fdaa7-kserve-9866d468c-fwmsk" Oct 30 21:04:51.810628 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:04:51.802814 5378 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-f9f277/distinct-2-w1-f9f277-kserve-6c49cf5dcb-h4x74" podUID="9eb6ff54-b77b-453c-a166-988b98cf8f49" containerName="main" probeResult="failure" output="Get \"https://10.133.0.76:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:04:51.896618 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:04:51.868032 5378 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-f9f277/distinct-2-w1-f9f277-kserve-6c49cf5dcb-h4x74" Oct 30 21:04:51.896618 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:04:51.880451 5378 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-d061d0/simulator-w2-d061d0-kserve-8484f9678b-lnszm" podUID="525fac02-b21c-4dd5-b553-72bcc36062c5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/ready\": context deadline exceeded" Oct 30 21:04:51.896618 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:04:51.881745 5378 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-d061d0/simulator-w2-d061d0-kserve-8484f9678b-lnszm" Oct 30 21:04:52.217384 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:04:52.203103 5378 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-781058/distinct-w0-781058-kserve-8bdf9bfcd-hzdfk" podUID="a9678fc2-1c56-4d45-9007-eee5049e0f48" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:04:52.442045 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:04:52.441989 5378 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-781058/distinct-w0-781058-kserve-8bdf9bfcd-hzdfk" podUID="a9678fc2-1c56-4d45-9007-eee5049e0f48" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/ready\": context deadline exceeded" Oct 30 21:04:52.448304 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:04:52.448279 5378 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-781058/distinct-w0-781058-kserve-8bdf9bfcd-hzdfk" Oct 30 21:04:52.476645 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:04:52.451445 5378 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-1fdaa7/premium-w4-1fdaa7-kserve-55dffbd5cd-gvpxq" podUID="ec282a4b-742f-4382-a296-335dcd2e9636" containerName="main" probeResult="failure" output="Get \"https://10.133.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:04:52.544588 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:04:52.457943 5378 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-8fa2a7/premium-w6-8fa2a7-kserve-64bf4bf6c-dxrdr" podUID="484cba5f-b415-430a-b52d-50b1293be948" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/ready\": context deadline exceeded" Oct 30 21:04:52.601586 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:04:52.508635 5378 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-d061d0/distinct-w2-d061d0-kserve-679c7cd7c4-8czrv" podUID="2c139b9f-c7df-4675-b431-95d4136095e5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:04:52.732632 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:04:52.730263 5378 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-f9f277/simulator-w1-f9f277-kserve-5d9685f85c-2h78x" podUID="dc662e9d-914c-4dfb-a86f-0fb048f7b690" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:04:52.741257 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:04:52.741203 5378 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-8fa2a7/simulator-w6-8fa2a7-kserve-6cbf5bd894-lfpcq" podUID="f409f469-71fc-445b-bdc8-b8e84a4d1dc6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 21:04:52.750925 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:04:52.750896 5378 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-f9f277/distinct-2-w1-f9f277-kserve-6c49cf5dcb-h4x74" Oct 30 21:04:52.751170 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:04:52.751159 5378 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-781058/distinct-w0-781058-kserve-8bdf9bfcd-hzdfk" Oct 30 21:04:52.751459 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:04:52.751435 5378 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-1fdaa7/distinct-w4-1fdaa7-kserve-86dc9774b9-6x2gf" podUID="2e162c95-0527-4cb3-85a3-795aae212d29" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:04:52.766618 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:04:52.760649 5378 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-d061d0/simulator-w2-d061d0-kserve-8484f9678b-lnszm" Oct 30 21:04:52.822928 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:04:52.821993 5378 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-d061d0/distinct-w2-d061d0-kserve-679c7cd7c4-8czrv" Oct 30 21:04:52.835365 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:04:52.835307 5378 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-1fdaa7/simulator-w4-1fdaa7-kserve-9866d468c-fwmsk" podUID="09219832-010e-4ba8-bfb9-5a2884b358e1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 21:04:52.838707 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:04:52.838688 5378 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-1fdaa7/premium-w4-1fdaa7-kserve-55dffbd5cd-gvpxq" Oct 30 21:04:52.838948 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:04:52.838938 5378 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-8fa2a7/premium-w6-8fa2a7-kserve-64bf4bf6c-dxrdr" Oct 30 21:04:52.850399 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:04:52.850378 5378 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-f9f277/simulator-w1-f9f277-kserve-5d9685f85c-2h78x" Oct 30 21:04:52.852070 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:04:52.852052 5378 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-8fa2a7/simulator-w6-8fa2a7-kserve-6cbf5bd894-lfpcq" Oct 30 21:04:52.873806 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:04:52.873778 5378 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-1fdaa7/distinct-w4-1fdaa7-kserve-86dc9774b9-6x2gf" Oct 30 21:04:52.908846 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:04:52.908807 5378 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-1fdaa7/simulator-w4-1fdaa7-kserve-9866d468c-fwmsk" Oct 30 21:04:54.177617 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:04:54.176321 5378 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-de53c2-fdf4f8fc5-8fjrg" Oct 30 21:04:54.177617 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:04:54.176474 5378 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-de53c2-6dc7f4bf96-b5cbp" Oct 30 21:04:54.619645 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:04:54.619608 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:04:57.184328 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:04:57.184297 5378 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-mt-19db3d5f-744ff4fb64-blk7k" Oct 30 21:04:58.896731 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:04:58.896680 5378 patch_prober.go:29] interesting pod/e2e-ait-2cf21005-openshift-default-5fdcf57455-c7jwb container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.106:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 21:04:58.897222 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:04:58.896758 5378 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ait-2cf21005-openshift-default-5fdcf57455-c7jwb" podUID="c7035375-d666-498d-b9e7-f699e1e86c99" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.106:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 21:04:59.618659 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:04:59.618620 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:04:59.904715 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:04:59.904680 5378 patch_prober.go:29] interesting pod/e2e-derive-bf9c0727-openshift-default-7fb479dd89-jzzdk container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.107:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 21:04:59.905115 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:04:59.904782 5378 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-derive-bf9c0727-openshift-default-7fb479dd89-jzzdk" podUID="eafa34dc-6d4b-4507-a2a3-989ad8cf7de2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.107:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 21:05:00.675639 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:05:00.675591 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:05:01.995401 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:05:01.995363 5378 generic.go:356] "Generic (PLEG): container finished" podID="eafa34dc-6d4b-4507-a2a3-989ad8cf7de2" containerID="c90939ef377fe76a9fbc3ed0753ebe88aaaad74d368b45d4a2658581909adb44" exitCode=0 Oct 30 21:05:01.995901 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:05:01.995437 5378 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-bf9c0727-openshift-default-7fb479dd89-jzzdk" event={"ID":"eafa34dc-6d4b-4507-a2a3-989ad8cf7de2","Type":"ContainerDied","Data":"c90939ef377fe76a9fbc3ed0753ebe88aaaad74d368b45d4a2658581909adb44"} Oct 30 21:05:01.995983 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:05:01.995968 5378 scope.go:122] "RemoveContainer" containerID="c90939ef377fe76a9fbc3ed0753ebe88aaaad74d368b45d4a2658581909adb44" Oct 30 21:05:01.997273 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:05:01.997248 5378 generic.go:356] "Generic (PLEG): container finished" podID="c7035375-d666-498d-b9e7-f699e1e86c99" containerID="59bfd2be3c8f8ea9773d7c0dfc4caacef2573c6729b786c89f506e047f94d370" exitCode=0 Oct 30 21:05:01.997394 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:05:01.997325 5378 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-2cf21005-openshift-default-5fdcf57455-c7jwb" event={"ID":"c7035375-d666-498d-b9e7-f699e1e86c99","Type":"ContainerDied","Data":"59bfd2be3c8f8ea9773d7c0dfc4caacef2573c6729b786c89f506e047f94d370"} Oct 30 21:05:01.997727 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:05:01.997709 5378 scope.go:122] "RemoveContainer" containerID="59bfd2be3c8f8ea9773d7c0dfc4caacef2573c6729b786c89f506e047f94d370" Oct 30 21:05:01.997816 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:05:01.997739 5378 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 21:05:01.997816 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:05:01.997807 5378 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 21:05:01.997887 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:05:01.997843 5378 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 21:05:01.999460 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:05:01.999433 5378 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 21:05:01.999535 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:05:01.999508 5378 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 21:05:01.999596 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:05:01.999554 5378 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 21:05:03.015353 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:05:03.015313 5378 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-bf9c0727-openshift-default-7fb479dd89-jzzdk" event={"ID":"eafa34dc-6d4b-4507-a2a3-989ad8cf7de2","Type":"ContainerStarted","Data":"36a0351f1605358413b4ed86dc714b151758b0961124deaed1360602b7804c2a"} Oct 30 21:05:03.018258 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:05:03.018220 5378 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-2cf21005-openshift-default-5fdcf57455-c7jwb" event={"ID":"c7035375-d666-498d-b9e7-f699e1e86c99","Type":"ContainerStarted","Data":"51c283842e2cefe34949229234561d3e96098541d98c1f3de02529a6df501442"} Oct 30 21:05:03.126537 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:05:03.126500 5378 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-f9f277-openshift-default-869d58fb77-wwkxn"] Oct 30 21:05:03.126995 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:05:03.126764 5378 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-f9f277-openshift-default-869d58fb77-wwkxn" podUID="3c98266e-ebfe-4c44-ae6e-079e5109bf9c" containerName="istio-proxy" containerID="cri-o://5b9e127f42ee4f6169792ae410b55f7f06a5197c2448b74af8fc3d46d48ceed2" gracePeriod=30 Oct 30 21:05:03.185601 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:05:03.180985 5378 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-2cf21005-openshift-default-5fdcf57455-c7jwb" Oct 30 21:05:03.185601 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:05:03.181121 5378 patch_prober.go:29] interesting pod/e2e-ait-2cf21005-openshift-default-5fdcf57455-c7jwb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.106:15021/healthz/ready\": dial tcp 10.133.0.106:15021: connect: connection refused" start-of-body= Oct 30 21:05:03.185601 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:05:03.181163 5378 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-2cf21005-openshift-default-5fdcf57455-c7jwb" podUID="c7035375-d666-498d-b9e7-f699e1e86c99" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.106:15021/healthz/ready\": dial tcp 10.133.0.106:15021: connect: connection refused" Oct 30 21:05:03.664066 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:05:03.664027 5378 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-derive-bf9c0727-openshift-default-7fb479dd89-jzzdk" Oct 30 21:05:03.664482 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:05:03.664383 5378 patch_prober.go:29] interesting pod/e2e-derive-bf9c0727-openshift-default-7fb479dd89-jzzdk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.107:15021/healthz/ready\": dial tcp 10.133.0.107:15021: connect: connection refused" start-of-body= Oct 30 21:05:03.664711 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:05:03.664684 5378 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-bf9c0727-openshift-default-7fb479dd89-jzzdk" podUID="eafa34dc-6d4b-4507-a2a3-989ad8cf7de2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.107:15021/healthz/ready\": dial tcp 10.133.0.107:15021: connect: connection refused" Oct 30 21:05:04.183994 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:05:04.183950 5378 patch_prober.go:29] interesting pod/e2e-ait-2cf21005-openshift-default-5fdcf57455-c7jwb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.106:15021/healthz/ready\": dial tcp 10.133.0.106:15021: connect: connection refused" start-of-body= Oct 30 21:05:04.184481 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:05:04.184023 5378 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-2cf21005-openshift-default-5fdcf57455-c7jwb" podUID="c7035375-d666-498d-b9e7-f699e1e86c99" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.106:15021/healthz/ready\": dial tcp 10.133.0.106:15021: connect: connection refused" Oct 30 21:05:04.605876 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:05:04.595475 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:05:04.662990 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:05:04.662951 5378 patch_prober.go:29] interesting pod/e2e-derive-bf9c0727-openshift-default-7fb479dd89-jzzdk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.107:15021/healthz/ready\": dial tcp 10.133.0.107:15021: connect: connection refused" start-of-body= Oct 30 21:05:04.663297 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:05:04.663265 5378 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-bf9c0727-openshift-default-7fb479dd89-jzzdk" podUID="eafa34dc-6d4b-4507-a2a3-989ad8cf7de2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.107:15021/healthz/ready\": dial tcp 10.133.0.107:15021: connect: connection refused" Oct 30 21:05:05.182957 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:05:05.182917 5378 patch_prober.go:29] interesting pod/e2e-ait-2cf21005-openshift-default-5fdcf57455-c7jwb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.106:15021/healthz/ready\": dial tcp 10.133.0.106:15021: connect: connection refused" start-of-body= Oct 30 21:05:05.183783 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:05:05.183726 5378 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-2cf21005-openshift-default-5fdcf57455-c7jwb" podUID="c7035375-d666-498d-b9e7-f699e1e86c99" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.106:15021/healthz/ready\": dial tcp 10.133.0.106:15021: connect: connection refused" Oct 30 21:05:05.666605 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:05:05.666541 5378 patch_prober.go:29] interesting pod/e2e-derive-bf9c0727-openshift-default-7fb479dd89-jzzdk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.107:15021/healthz/ready\": dial tcp 10.133.0.107:15021: connect: connection refused" start-of-body= Oct 30 21:05:05.667214 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:05:05.667173 5378 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-bf9c0727-openshift-default-7fb479dd89-jzzdk" podUID="eafa34dc-6d4b-4507-a2a3-989ad8cf7de2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.107:15021/healthz/ready\": dial tcp 10.133.0.107:15021: connect: connection refused" Oct 30 21:05:06.186440 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:05:06.186404 5378 patch_prober.go:29] interesting pod/e2e-ait-2cf21005-openshift-default-5fdcf57455-c7jwb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.106:15021/healthz/ready\": dial tcp 10.133.0.106:15021: connect: connection refused" start-of-body= Oct 30 21:05:06.186759 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:05:06.186729 5378 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-2cf21005-openshift-default-5fdcf57455-c7jwb" podUID="c7035375-d666-498d-b9e7-f699e1e86c99" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.106:15021/healthz/ready\": dial tcp 10.133.0.106:15021: connect: connection refused" Oct 30 21:05:06.589644 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:05:06.584780 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:05:06.664982 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:05:06.664941 5378 patch_prober.go:29] interesting pod/e2e-derive-bf9c0727-openshift-default-7fb479dd89-jzzdk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.107:15021/healthz/ready\": dial tcp 10.133.0.107:15021: connect: connection refused" start-of-body= Oct 30 21:05:06.665291 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:05:06.665250 5378 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-bf9c0727-openshift-default-7fb479dd89-jzzdk" podUID="eafa34dc-6d4b-4507-a2a3-989ad8cf7de2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.107:15021/healthz/ready\": dial tcp 10.133.0.107:15021: connect: connection refused" Oct 30 21:05:07.181589 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:05:07.180967 5378 patch_prober.go:29] interesting pod/e2e-ait-2cf21005-openshift-default-5fdcf57455-c7jwb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.106:15021/healthz/ready\": dial tcp 10.133.0.106:15021: connect: connection refused" start-of-body= Oct 30 21:05:07.181589 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:05:07.181046 5378 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-2cf21005-openshift-default-5fdcf57455-c7jwb" podUID="c7035375-d666-498d-b9e7-f699e1e86c99" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.106:15021/healthz/ready\": dial tcp 10.133.0.106:15021: connect: connection refused" Oct 30 21:05:07.674979 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:05:07.674942 5378 patch_prober.go:29] interesting pod/e2e-derive-bf9c0727-openshift-default-7fb479dd89-jzzdk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.107:15021/healthz/ready\": dial tcp 10.133.0.107:15021: connect: connection refused" start-of-body= Oct 30 21:05:07.675263 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:05:07.675234 5378 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-bf9c0727-openshift-default-7fb479dd89-jzzdk" podUID="eafa34dc-6d4b-4507-a2a3-989ad8cf7de2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.107:15021/healthz/ready\": dial tcp 10.133.0.107:15021: connect: connection refused" Oct 30 21:05:08.007523 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:05:08.007433 5378 patch_prober.go:29] interesting pod/e2e-worker-w1-f9f277-openshift-default-869d58fb77-wwkxn container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 21:05:08.007811 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:05:08.007780 5378 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-f9f277-openshift-default-869d58fb77-wwkxn" podUID="3c98266e-ebfe-4c44-ae6e-079e5109bf9c" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 21:05:08.048968 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:05:08.048925 5378 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bd507c85-vz2br"] Oct 30 21:05:08.111158 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:05:08.111124 5378 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bd507c85-vz2br"] Oct 30 21:05:08.181239 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:05:08.181195 5378 patch_prober.go:29] interesting pod/e2e-ait-2cf21005-openshift-default-5fdcf57455-c7jwb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.106:15021/healthz/ready\": dial tcp 10.133.0.106:15021: connect: connection refused" start-of-body= Oct 30 21:05:08.181416 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:05:08.181286 5378 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-2cf21005-openshift-default-5fdcf57455-c7jwb" podUID="c7035375-d666-498d-b9e7-f699e1e86c99" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.106:15021/healthz/ready\": dial tcp 10.133.0.106:15021: connect: connection refused" Oct 30 21:05:08.448216 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:05:08.448191 5378 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-f9f277-openshift-default-869d58fb77-wwkxn" Oct 30 21:05:08.537825 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:05:08.537796 5378 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3c98266e-ebfe-4c44-ae6e-079e5109bf9c-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3c98266e-ebfe-4c44-ae6e-079e5109bf9c-workload-socket\") pod \"3c98266e-ebfe-4c44-ae6e-079e5109bf9c\" (UID: \"3c98266e-ebfe-4c44-ae6e-079e5109bf9c\") " Oct 30 21:05:08.537982 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:05:08.537872 5378 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/3c98266e-ebfe-4c44-ae6e-079e5109bf9c-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3c98266e-ebfe-4c44-ae6e-079e5109bf9c-istio-podinfo\") pod \"3c98266e-ebfe-4c44-ae6e-079e5109bf9c\" (UID: \"3c98266e-ebfe-4c44-ae6e-079e5109bf9c\") " Oct 30 21:05:08.537982 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:05:08.537897 5378 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3c98266e-ebfe-4c44-ae6e-079e5109bf9c-kube-api-access-v9nct\" (UniqueName: \"kubernetes.io/projected/3c98266e-ebfe-4c44-ae6e-079e5109bf9c-kube-api-access-v9nct\") pod \"3c98266e-ebfe-4c44-ae6e-079e5109bf9c\" (UID: \"3c98266e-ebfe-4c44-ae6e-079e5109bf9c\") " Oct 30 21:05:08.537982 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:05:08.537940 5378 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3c98266e-ebfe-4c44-ae6e-079e5109bf9c-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3c98266e-ebfe-4c44-ae6e-079e5109bf9c-istio-data\") pod \"3c98266e-ebfe-4c44-ae6e-079e5109bf9c\" (UID: \"3c98266e-ebfe-4c44-ae6e-079e5109bf9c\") " Oct 30 21:05:08.537982 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:05:08.537971 5378 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3c98266e-ebfe-4c44-ae6e-079e5109bf9c-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3c98266e-ebfe-4c44-ae6e-079e5109bf9c-istiod-ca-cert\") pod \"3c98266e-ebfe-4c44-ae6e-079e5109bf9c\" (UID: \"3c98266e-ebfe-4c44-ae6e-079e5109bf9c\") " Oct 30 21:05:08.538255 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:05:08.537998 5378 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3c98266e-ebfe-4c44-ae6e-079e5109bf9c-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3c98266e-ebfe-4c44-ae6e-079e5109bf9c-credential-socket\") pod \"3c98266e-ebfe-4c44-ae6e-079e5109bf9c\" (UID: \"3c98266e-ebfe-4c44-ae6e-079e5109bf9c\") " Oct 30 21:05:08.538255 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:05:08.538040 5378 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3c98266e-ebfe-4c44-ae6e-079e5109bf9c-workload-socket" pod "3c98266e-ebfe-4c44-ae6e-079e5109bf9c" (UID: "3c98266e-ebfe-4c44-ae6e-079e5109bf9c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:05:08.538255 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:05:08.538226 5378 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3c98266e-ebfe-4c44-ae6e-079e5109bf9c-istio-data" pod "3c98266e-ebfe-4c44-ae6e-079e5109bf9c" (UID: "3c98266e-ebfe-4c44-ae6e-079e5109bf9c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:05:08.538433 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:05:08.538292 5378 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3c98266e-ebfe-4c44-ae6e-079e5109bf9c-credential-socket" pod "3c98266e-ebfe-4c44-ae6e-079e5109bf9c" (UID: "3c98266e-ebfe-4c44-ae6e-079e5109bf9c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:05:08.538433 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:05:08.538414 5378 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3c98266e-ebfe-4c44-ae6e-079e5109bf9c-istiod-ca-cert" pod "3c98266e-ebfe-4c44-ae6e-079e5109bf9c" (UID: "3c98266e-ebfe-4c44-ae6e-079e5109bf9c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:05:08.538821 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:05:08.538800 5378 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3c98266e-ebfe-4c44-ae6e-079e5109bf9c-workload-socket\") on node \"ip-10-0-138-149.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:05:08.538908 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:05:08.538823 5378 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3c98266e-ebfe-4c44-ae6e-079e5109bf9c-istio-data\") on node \"ip-10-0-138-149.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:05:08.538908 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:05:08.538839 5378 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3c98266e-ebfe-4c44-ae6e-079e5109bf9c-istiod-ca-cert\") on node \"ip-10-0-138-149.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:05:08.538908 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:05:08.538852 5378 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3c98266e-ebfe-4c44-ae6e-079e5109bf9c-credential-socket\") on node \"ip-10-0-138-149.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:05:08.540183 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:05:08.540162 5378 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/3c98266e-ebfe-4c44-ae6e-079e5109bf9c-istio-podinfo" pod "3c98266e-ebfe-4c44-ae6e-079e5109bf9c" (UID: "3c98266e-ebfe-4c44-ae6e-079e5109bf9c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 21:05:08.540349 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:05:08.540187 5378 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3c98266e-ebfe-4c44-ae6e-079e5109bf9c-kube-api-access-v9nct" pod "3c98266e-ebfe-4c44-ae6e-079e5109bf9c" (UID: "3c98266e-ebfe-4c44-ae6e-079e5109bf9c"). InnerVolumeSpecName "kube-api-access-v9nct". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:05:08.640670 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:05:08.640636 5378 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3c98266e-ebfe-4c44-ae6e-079e5109bf9c-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3c98266e-ebfe-4c44-ae6e-079e5109bf9c-istio-envoy\") pod \"3c98266e-ebfe-4c44-ae6e-079e5109bf9c\" (UID: \"3c98266e-ebfe-4c44-ae6e-079e5109bf9c\") " Oct 30 21:05:08.640853 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:05:08.640703 5378 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3c98266e-ebfe-4c44-ae6e-079e5109bf9c-istio-token\" (UniqueName: \"kubernetes.io/projected/3c98266e-ebfe-4c44-ae6e-079e5109bf9c-istio-token\") pod \"3c98266e-ebfe-4c44-ae6e-079e5109bf9c\" (UID: \"3c98266e-ebfe-4c44-ae6e-079e5109bf9c\") " Oct 30 21:05:08.640853 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:05:08.640741 5378 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3c98266e-ebfe-4c44-ae6e-079e5109bf9c-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3c98266e-ebfe-4c44-ae6e-079e5109bf9c-workload-certs\") pod \"3c98266e-ebfe-4c44-ae6e-079e5109bf9c\" (UID: \"3c98266e-ebfe-4c44-ae6e-079e5109bf9c\") " Oct 30 21:05:08.641165 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:05:08.641137 5378 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3c98266e-ebfe-4c44-ae6e-079e5109bf9c-workload-certs" pod "3c98266e-ebfe-4c44-ae6e-079e5109bf9c" (UID: "3c98266e-ebfe-4c44-ae6e-079e5109bf9c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:05:08.641638 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:05:08.641614 5378 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3c98266e-ebfe-4c44-ae6e-079e5109bf9c-istio-podinfo\") on node \"ip-10-0-138-149.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:05:08.641638 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:05:08.641638 5378 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v9nct\" (UniqueName: \"kubernetes.io/projected/3c98266e-ebfe-4c44-ae6e-079e5109bf9c-kube-api-access-v9nct\") on node \"ip-10-0-138-149.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:05:08.641847 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:05:08.641649 5378 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3c98266e-ebfe-4c44-ae6e-079e5109bf9c-workload-certs\") on node \"ip-10-0-138-149.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:05:08.642895 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:05:08.642873 5378 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3c98266e-ebfe-4c44-ae6e-079e5109bf9c-istio-envoy" pod "3c98266e-ebfe-4c44-ae6e-079e5109bf9c" (UID: "3c98266e-ebfe-4c44-ae6e-079e5109bf9c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:05:08.643000 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:05:08.642986 5378 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3c98266e-ebfe-4c44-ae6e-079e5109bf9c-istio-token" pod "3c98266e-ebfe-4c44-ae6e-079e5109bf9c" (UID: "3c98266e-ebfe-4c44-ae6e-079e5109bf9c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:05:08.668049 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:05:08.668028 5378 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-derive-bf9c0727-openshift-default-7fb479dd89-jzzdk" Oct 30 21:05:08.668373 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:05:08.668354 5378 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-derive-bf9c0727-openshift-default-7fb479dd89-jzzdk" Oct 30 21:05:08.668981 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:05:08.668965 5378 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-derive-bf9c0727-openshift-default-7fb479dd89-jzzdk" Oct 30 21:05:08.743402 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:05:08.743360 5378 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3c98266e-ebfe-4c44-ae6e-079e5109bf9c-istio-envoy\") on node \"ip-10-0-138-149.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:05:08.743402 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:05:08.743396 5378 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3c98266e-ebfe-4c44-ae6e-079e5109bf9c-istio-token\") on node \"ip-10-0-138-149.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:05:09.116660 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:05:09.116627 5378 generic.go:356] "Generic (PLEG): container finished" podID="3c98266e-ebfe-4c44-ae6e-079e5109bf9c" containerID="5b9e127f42ee4f6169792ae410b55f7f06a5197c2448b74af8fc3d46d48ceed2" exitCode=0 Oct 30 21:05:09.116822 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:05:09.116683 5378 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-f9f277-openshift-default-869d58fb77-wwkxn" Oct 30 21:05:09.116822 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:05:09.116716 5378 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-f9f277-openshift-default-869d58fb77-wwkxn" event={"ID":"3c98266e-ebfe-4c44-ae6e-079e5109bf9c","Type":"ContainerDied","Data":"5b9e127f42ee4f6169792ae410b55f7f06a5197c2448b74af8fc3d46d48ceed2"} Oct 30 21:05:09.116822 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:05:09.116781 5378 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-f9f277-openshift-default-869d58fb77-wwkxn" event={"ID":"3c98266e-ebfe-4c44-ae6e-079e5109bf9c","Type":"ContainerDied","Data":"e0afb729e20be762809cc3a816360d11af514657d991c81038b1447fc0e13870"} Oct 30 21:05:09.116822 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:05:09.116811 5378 scope.go:122] "RemoveContainer" containerID="5b9e127f42ee4f6169792ae410b55f7f06a5197c2448b74af8fc3d46d48ceed2" Oct 30 21:05:09.127968 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:05:09.127952 5378 scope.go:122] "RemoveContainer" containerID="5b9e127f42ee4f6169792ae410b55f7f06a5197c2448b74af8fc3d46d48ceed2" Oct 30 21:05:09.128245 ip-10-0-138-149 kubenswrapper[5378]: E1030 21:05:09.128227 5378 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5b9e127f42ee4f6169792ae410b55f7f06a5197c2448b74af8fc3d46d48ceed2\": container with ID starting with 5b9e127f42ee4f6169792ae410b55f7f06a5197c2448b74af8fc3d46d48ceed2 not found: ID does not exist" containerID="5b9e127f42ee4f6169792ae410b55f7f06a5197c2448b74af8fc3d46d48ceed2" Oct 30 21:05:09.128294 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:05:09.128254 5378 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5b9e127f42ee4f6169792ae410b55f7f06a5197c2448b74af8fc3d46d48ceed2"} err="failed to get container status \"5b9e127f42ee4f6169792ae410b55f7f06a5197c2448b74af8fc3d46d48ceed2\": rpc error: code = NotFound desc = could not find container \"5b9e127f42ee4f6169792ae410b55f7f06a5197c2448b74af8fc3d46d48ceed2\": container with ID starting with 5b9e127f42ee4f6169792ae410b55f7f06a5197c2448b74af8fc3d46d48ceed2 not found: ID does not exist" Oct 30 21:05:09.185423 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:05:09.185395 5378 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-2cf21005-openshift-default-5fdcf57455-c7jwb" Oct 30 21:05:09.185679 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:05:09.185660 5378 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-2cf21005-openshift-default-5fdcf57455-c7jwb" Oct 30 21:05:09.186293 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:05:09.186278 5378 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-2cf21005-openshift-default-5fdcf57455-c7jwb" Oct 30 21:05:09.284677 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:05:09.284650 5378 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-f9f277-openshift-default-869d58fb77-wwkxn"] Oct 30 21:05:09.335463 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:05:09.335434 5378 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-f9f277-openshift-default-869d58fb77-wwkxn"] Oct 30 21:05:09.593048 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:05:09.592974 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:05:09.783358 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:05:09.781813 5378 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2b1ddcab-5592-429c-9329-b7c2a9bbdfe9" path="/var/lib/kubelet/pods/2b1ddcab-5592-429c-9329-b7c2a9bbdfe9/volumes" Oct 30 21:05:09.783358 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:05:09.782019 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:05:09.783358 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:05:09.782880 5378 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3c98266e-ebfe-4c44-ae6e-079e5109bf9c" path="/var/lib/kubelet/pods/3c98266e-ebfe-4c44-ae6e-079e5109bf9c/volumes" Oct 30 21:05:09.907603 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:05:09.907547 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:05:11.048824 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:05:11.048791 5378 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b95e32e5-kzw5p"] Oct 30 21:05:11.098560 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:05:11.098528 5378 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b95e32e5-kzw5p"] Oct 30 21:05:11.743140 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:05:11.743108 5378 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="03973ac1-3f01-42f4-bc88-fa8ae0ddf93b" path="/var/lib/kubelet/pods/03973ac1-3f01-42f4-bc88-fa8ae0ddf93b/volumes" Oct 30 21:05:14.781241 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:05:14.781204 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:05:17.050623 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:05:17.050590 5378 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a44eaf05-cfrpj"] Oct 30 21:05:17.100539 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:05:17.100506 5378 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a44eaf05-cfrpj"] Oct 30 21:05:17.744828 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:05:17.744797 5378 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6107aaf1-6393-49ce-b59a-5c00a36ab3aa" path="/var/lib/kubelet/pods/6107aaf1-6393-49ce-b59a-5c00a36ab3aa/volumes" Oct 30 21:05:18.838682 ip-10-0-138-149 kubenswrapper[5378]: E1030 21:05:18.838641 5378 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q6mx7" containerName="extractor" Oct 30 21:05:19.759349 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:05:19.759309 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:05:21.050978 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:05:21.050942 5378 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ca08f19a-td59w"] Oct 30 21:05:21.101407 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:05:21.101380 5378 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ca08f19a-td59w"] Oct 30 21:05:21.104909 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:05:21.104885 5378 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-31baea91-bjrgz"] Oct 30 21:05:21.110883 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:05:21.110860 5378 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-31baea91-bjrgz"] Oct 30 21:05:21.743743 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:05:21.743706 5378 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8ce46c99-ec66-4e61-8b6f-135d76c640e1" path="/var/lib/kubelet/pods/8ce46c99-ec66-4e61-8b6f-135d76c640e1/volumes" Oct 30 21:05:21.744197 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:05:21.744182 5378 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="965b8faf-458e-499d-8cf2-4001dc4f97ef" path="/var/lib/kubelet/pods/965b8faf-458e-499d-8cf2-4001dc4f97ef/volumes" Oct 30 21:05:24.527463 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:05:24.527426 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:05:24.944382 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:05:24.944345 5378 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-r9g59"] Oct 30 21:05:28.047773 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:05:28.047735 5378 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ea167fdf-769pg"] Oct 30 21:05:28.098390 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:05:28.098359 5378 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ea167fdf-769pg"] Oct 30 21:05:29.743942 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:05:29.743904 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:05:29.749235 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:05:29.749205 5378 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4b8072df-d2bb-4709-82c5-6e6a9ccd9de6" path="/var/lib/kubelet/pods/4b8072df-d2bb-4709-82c5-6e6a9ccd9de6/volumes" Oct 30 21:05:34.111383 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:05:34.111350 5378 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-781058-openshift-default-55fb495b7-5zx7k"] Oct 30 21:05:34.111799 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:05:34.111556 5378 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-781058-openshift-default-55fb495b7-5zx7k" podUID="48a0901f-8b2b-4278-8422-226bb6bc6dd3" containerName="istio-proxy" containerID="cri-o://e636fc8115299f0916b8822ff29832fec0ea4a10f340562fe04ee4014ab985c5" gracePeriod=30 Oct 30 21:05:34.161430 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:05:34.161387 5378 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-781058-openshift-default-55fb495b7-2c7gl"] Oct 30 21:05:34.161642 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:05:34.161606 5378 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-781058-openshift-default-55fb495b7-2c7gl" podUID="507cd7ee-c139-489c-a520-292d8ebd91e4" containerName="istio-proxy" containerID="cri-o://6b92315664f5650233d2906cc50f29a6d230ad23d5fe304db0b0025ba3483c3e" gracePeriod=30 Oct 30 21:05:34.165160 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:05:34.165137 5378 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-781058-openshift-default-55fb495b7-hpgv5"] Oct 30 21:05:34.165336 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:05:34.165306 5378 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-781058-openshift-default-55fb495b7-hpgv5" podUID="d43ee748-feb2-4556-be54-3662f8dfa98d" containerName="istio-proxy" containerID="cri-o://2dfda93c6c9252b225103f327e91f54fc6eded5a91be4d9f9d7e8893160a3184" gracePeriod=30 Oct 30 21:05:34.211887 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:05:34.211857 5378 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-781058-openshift-default-55fb495b7-khvpl"] Oct 30 21:05:34.212066 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:05:34.212047 5378 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-781058-openshift-default-55fb495b7-khvpl" podUID="afcb5db1-70aa-4758-90e6-e21b112d48f4" containerName="istio-proxy" containerID="cri-o://c682db6caa42bbefaa74a7509ac7c21897323acff7831c3caf7369bba15a262f" gracePeriod=30 Oct 30 21:05:34.927552 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:05:34.927517 5378 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-de53c2-697d9756c6-t4xvj"] Oct 30 21:05:34.928602 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:05:34.928582 5378 cpu_manager.go:414] "containerMap: removing container" podUID="3c98266e-ebfe-4c44-ae6e-079e5109bf9c" containerName="istio-proxy" Oct 30 21:05:34.928689 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:05:34.928602 5378 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3c98266e-ebfe-4c44-ae6e-079e5109bf9c" containerName="istio-proxy" Oct 30 21:05:34.928828 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:05:34.928817 5378 memory_manager.go:365] "RemoveStaleState removing state" podUID="3c98266e-ebfe-4c44-ae6e-079e5109bf9c" containerName="istio-proxy" Oct 30 21:05:34.932762 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:05:34.932744 5378 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-de53c2-697d9756c6-t4xvj" Oct 30 21:05:34.985477 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:05:34.985446 5378 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-w2-de53c2-serving-cert\"" Oct 30 21:05:34.987435 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:05:34.987398 5378 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-de53c2-697d9756c6-t4xvj"] Oct 30 21:05:35.075273 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:05:35.075239 5378 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b3981dd6-e944-47d1-83a4-1b4689d74bcb-maas-api-tls\") pod \"maas-api-e2e-shared-a-w2-de53c2-697d9756c6-t4xvj\" (UID: \"b3981dd6-e944-47d1-83a4-1b4689d74bcb\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-de53c2-697d9756c6-t4xvj" Oct 30 21:05:35.075431 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:05:35.075368 5378 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b3981dd6-e944-47d1-83a4-1b4689d74bcb-metrics-certs\") pod \"maas-api-e2e-shared-a-w2-de53c2-697d9756c6-t4xvj\" (UID: \"b3981dd6-e944-47d1-83a4-1b4689d74bcb\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-de53c2-697d9756c6-t4xvj" Oct 30 21:05:35.075621 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:05:35.075599 5378 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tqqtq\" (UniqueName: \"kubernetes.io/projected/b3981dd6-e944-47d1-83a4-1b4689d74bcb-kube-api-access-tqqtq\") pod \"maas-api-e2e-shared-a-w2-de53c2-697d9756c6-t4xvj\" (UID: \"b3981dd6-e944-47d1-83a4-1b4689d74bcb\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-de53c2-697d9756c6-t4xvj" Oct 30 21:05:35.075731 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:05:35.075714 5378 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b3981dd6-e944-47d1-83a4-1b4689d74bcb-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w2-de53c2-697d9756c6-t4xvj\" (UID: \"b3981dd6-e944-47d1-83a4-1b4689d74bcb\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-de53c2-697d9756c6-t4xvj" Oct 30 21:05:35.156442 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:05:35.156399 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:05:35.176882 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:05:35.176853 5378 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b3981dd6-e944-47d1-83a4-1b4689d74bcb-maas-api-tls\") pod \"maas-api-e2e-shared-a-w2-de53c2-697d9756c6-t4xvj\" (UID: \"b3981dd6-e944-47d1-83a4-1b4689d74bcb\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-de53c2-697d9756c6-t4xvj" Oct 30 21:05:35.177041 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:05:35.176948 5378 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b3981dd6-e944-47d1-83a4-1b4689d74bcb-metrics-certs\") pod \"maas-api-e2e-shared-a-w2-de53c2-697d9756c6-t4xvj\" (UID: \"b3981dd6-e944-47d1-83a4-1b4689d74bcb\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-de53c2-697d9756c6-t4xvj" Oct 30 21:05:35.177041 ip-10-0-138-149 kubenswrapper[5378]: E1030 21:05:35.177020 5378 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-de53c2-serving-cert: secret "maas-api-e2e-shared-a-w2-de53c2-serving-cert" not found Oct 30 21:05:35.177139 ip-10-0-138-149 kubenswrapper[5378]: E1030 21:05:35.177084 5378 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b3981dd6-e944-47d1-83a4-1b4689d74bcb-maas-api-tls podName:b3981dd6-e944-47d1-83a4-1b4689d74bcb nodeName:}" failed. No retries permitted until 2026-10-30 21:05:35.677066294 +0000 UTC m=+1626.105180204 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/b3981dd6-e944-47d1-83a4-1b4689d74bcb-maas-api-tls") pod "maas-api-e2e-shared-a-w2-de53c2-697d9756c6-t4xvj" (UID: "b3981dd6-e944-47d1-83a4-1b4689d74bcb") : secret "maas-api-e2e-shared-a-w2-de53c2-serving-cert" not found Oct 30 21:05:35.177198 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:05:35.177182 5378 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tqqtq\" (UniqueName: \"kubernetes.io/projected/b3981dd6-e944-47d1-83a4-1b4689d74bcb-kube-api-access-tqqtq\") pod \"maas-api-e2e-shared-a-w2-de53c2-697d9756c6-t4xvj\" (UID: \"b3981dd6-e944-47d1-83a4-1b4689d74bcb\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-de53c2-697d9756c6-t4xvj" Oct 30 21:05:35.177305 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:05:35.177287 5378 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b3981dd6-e944-47d1-83a4-1b4689d74bcb-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w2-de53c2-697d9756c6-t4xvj\" (UID: \"b3981dd6-e944-47d1-83a4-1b4689d74bcb\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-de53c2-697d9756c6-t4xvj" Oct 30 21:05:35.178032 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:05:35.177980 5378 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b3981dd6-e944-47d1-83a4-1b4689d74bcb-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w2-de53c2-697d9756c6-t4xvj\" (UID: \"b3981dd6-e944-47d1-83a4-1b4689d74bcb\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-de53c2-697d9756c6-t4xvj" Oct 30 21:05:35.179369 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:05:35.179348 5378 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b3981dd6-e944-47d1-83a4-1b4689d74bcb-metrics-certs\") pod \"maas-api-e2e-shared-a-w2-de53c2-697d9756c6-t4xvj\" (UID: \"b3981dd6-e944-47d1-83a4-1b4689d74bcb\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-de53c2-697d9756c6-t4xvj" Oct 30 21:05:35.279609 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:05:35.275542 5378 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tqqtq\" (UniqueName: \"kubernetes.io/projected/b3981dd6-e944-47d1-83a4-1b4689d74bcb-kube-api-access-tqqtq\") pod \"maas-api-e2e-shared-a-w2-de53c2-697d9756c6-t4xvj\" (UID: \"b3981dd6-e944-47d1-83a4-1b4689d74bcb\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-de53c2-697d9756c6-t4xvj" Oct 30 21:05:35.339990 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:05:35.318499 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:05:35.445496 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:05:35.445396 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:05:35.690530 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:05:35.690494 5378 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b3981dd6-e944-47d1-83a4-1b4689d74bcb-maas-api-tls\") pod \"maas-api-e2e-shared-a-w2-de53c2-697d9756c6-t4xvj\" (UID: \"b3981dd6-e944-47d1-83a4-1b4689d74bcb\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-de53c2-697d9756c6-t4xvj" Oct 30 21:05:35.692958 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:05:35.692934 5378 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b3981dd6-e944-47d1-83a4-1b4689d74bcb-maas-api-tls\") pod \"maas-api-e2e-shared-a-w2-de53c2-697d9756c6-t4xvj\" (UID: \"b3981dd6-e944-47d1-83a4-1b4689d74bcb\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-de53c2-697d9756c6-t4xvj" Oct 30 21:05:35.848129 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:05:35.848097 5378 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-de53c2-697d9756c6-t4xvj" Oct 30 21:05:36.229391 ip-10-0-138-149 kubenswrapper[5378]: W1030 21:05:36.229362 5378 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb3981dd6_e944_47d1_83a4_1b4689d74bcb.slice/crio-fd4073727d235cca1689eb406ded5358e4bb3853ebfad97963b4604f8590bf1f WatchSource:0}: Error finding container fd4073727d235cca1689eb406ded5358e4bb3853ebfad97963b4604f8590bf1f: Status 404 returned error can't find the container with id fd4073727d235cca1689eb406ded5358e4bb3853ebfad97963b4604f8590bf1f Oct 30 21:05:36.229884 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:05:36.229855 5378 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-de53c2-697d9756c6-t4xvj"] Oct 30 21:05:36.418122 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:05:36.418087 5378 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-de53c2-697d9756c6-t4xvj" event={"ID":"b3981dd6-e944-47d1-83a4-1b4689d74bcb","Type":"ContainerStarted","Data":"fd4073727d235cca1689eb406ded5358e4bb3853ebfad97963b4604f8590bf1f"} Oct 30 21:05:37.429303 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:05:37.429267 5378 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-de53c2-697d9756c6-t4xvj" event={"ID":"b3981dd6-e944-47d1-83a4-1b4689d74bcb","Type":"ContainerStarted","Data":"2eb97482976b47870badffb3d92c28732a708331b9ed83781b7f261f78cb7a25"} Oct 30 21:05:37.429734 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:05:37.429359 5378 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-de53c2-697d9756c6-t4xvj" Oct 30 21:05:37.544637 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:05:37.544560 5378 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-de53c2-697d9756c6-t4xvj" podStartSLOduration=2.681305962 podStartE2EDuration="3.544546909s" totalImagesPullingTime="863.240947ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:05:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:05:37.544273092 +0000 UTC m=+1627.972387013" watchObservedRunningTime="2026-10-30 21:05:37.544546909 +0000 UTC m=+1627.972660827" Oct 30 21:05:39.449202 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:05:39.449160 5378 generic.go:356] "Generic (PLEG): container finished" podID="afcb5db1-70aa-4758-90e6-e21b112d48f4" containerID="c682db6caa42bbefaa74a7509ac7c21897323acff7831c3caf7369bba15a262f" exitCode=0 Oct 30 21:05:39.449663 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:05:39.449288 5378 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-781058-openshift-default-55fb495b7-khvpl" event={"ID":"afcb5db1-70aa-4758-90e6-e21b112d48f4","Type":"ContainerDied","Data":"c682db6caa42bbefaa74a7509ac7c21897323acff7831c3caf7369bba15a262f"} Oct 30 21:05:39.451672 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:05:39.451443 5378 generic.go:356] "Generic (PLEG): container finished" podID="48a0901f-8b2b-4278-8422-226bb6bc6dd3" containerID="e636fc8115299f0916b8822ff29832fec0ea4a10f340562fe04ee4014ab985c5" exitCode=0 Oct 30 21:05:39.451672 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:05:39.451522 5378 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-781058-openshift-default-55fb495b7-5zx7k" event={"ID":"48a0901f-8b2b-4278-8422-226bb6bc6dd3","Type":"ContainerDied","Data":"e636fc8115299f0916b8822ff29832fec0ea4a10f340562fe04ee4014ab985c5"} Oct 30 21:05:39.451672 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:05:39.451583 5378 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-781058-openshift-default-55fb495b7-5zx7k" event={"ID":"48a0901f-8b2b-4278-8422-226bb6bc6dd3","Type":"ContainerDied","Data":"68447e73ada5c838ae757208013800e46def98e0e5e8ce7d87383a3cd27279b3"} Oct 30 21:05:39.451672 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:05:39.451611 5378 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="68447e73ada5c838ae757208013800e46def98e0e5e8ce7d87383a3cd27279b3" Oct 30 21:05:39.459116 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:05:39.459089 5378 generic.go:356] "Generic (PLEG): container finished" podID="d43ee748-feb2-4556-be54-3662f8dfa98d" containerID="2dfda93c6c9252b225103f327e91f54fc6eded5a91be4d9f9d7e8893160a3184" exitCode=0 Oct 30 21:05:39.459300 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:05:39.459279 5378 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-781058-openshift-default-55fb495b7-hpgv5" event={"ID":"d43ee748-feb2-4556-be54-3662f8dfa98d","Type":"ContainerDied","Data":"2dfda93c6c9252b225103f327e91f54fc6eded5a91be4d9f9d7e8893160a3184"} Oct 30 21:05:39.465604 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:05:39.464030 5378 generic.go:356] "Generic (PLEG): container finished" podID="507cd7ee-c139-489c-a520-292d8ebd91e4" containerID="6b92315664f5650233d2906cc50f29a6d230ad23d5fe304db0b0025ba3483c3e" exitCode=0 Oct 30 21:05:39.465604 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:05:39.464174 5378 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-781058-openshift-default-55fb495b7-2c7gl" event={"ID":"507cd7ee-c139-489c-a520-292d8ebd91e4","Type":"ContainerDied","Data":"6b92315664f5650233d2906cc50f29a6d230ad23d5fe304db0b0025ba3483c3e"} Oct 30 21:05:39.470141 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:05:39.470125 5378 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-781058-openshift-default-55fb495b7-5zx7k" Oct 30 21:05:39.562986 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:05:39.562962 5378 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-781058-openshift-default-55fb495b7-hpgv5" Oct 30 21:05:39.563449 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:05:39.563430 5378 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/48a0901f-8b2b-4278-8422-226bb6bc6dd3-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/48a0901f-8b2b-4278-8422-226bb6bc6dd3-workload-certs\") pod \"48a0901f-8b2b-4278-8422-226bb6bc6dd3\" (UID: \"48a0901f-8b2b-4278-8422-226bb6bc6dd3\") " Oct 30 21:05:39.563530 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:05:39.563499 5378 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/48a0901f-8b2b-4278-8422-226bb6bc6dd3-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/48a0901f-8b2b-4278-8422-226bb6bc6dd3-workload-socket\") pod \"48a0901f-8b2b-4278-8422-226bb6bc6dd3\" (UID: \"48a0901f-8b2b-4278-8422-226bb6bc6dd3\") " Oct 30 21:05:39.563783 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:05:39.563766 5378 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/48a0901f-8b2b-4278-8422-226bb6bc6dd3-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/48a0901f-8b2b-4278-8422-226bb6bc6dd3-credential-socket\") pod \"48a0901f-8b2b-4278-8422-226bb6bc6dd3\" (UID: \"48a0901f-8b2b-4278-8422-226bb6bc6dd3\") " Oct 30 21:05:39.563898 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:05:39.563760 5378 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/48a0901f-8b2b-4278-8422-226bb6bc6dd3-workload-certs" pod "48a0901f-8b2b-4278-8422-226bb6bc6dd3" (UID: "48a0901f-8b2b-4278-8422-226bb6bc6dd3"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:05:39.563898 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:05:39.563810 5378 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/48a0901f-8b2b-4278-8422-226bb6bc6dd3-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/48a0901f-8b2b-4278-8422-226bb6bc6dd3-istiod-ca-cert\") pod \"48a0901f-8b2b-4278-8422-226bb6bc6dd3\" (UID: \"48a0901f-8b2b-4278-8422-226bb6bc6dd3\") " Oct 30 21:05:39.563898 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:05:39.563821 5378 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/48a0901f-8b2b-4278-8422-226bb6bc6dd3-workload-socket" pod "48a0901f-8b2b-4278-8422-226bb6bc6dd3" (UID: "48a0901f-8b2b-4278-8422-226bb6bc6dd3"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:05:39.563898 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:05:39.563832 5378 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/48a0901f-8b2b-4278-8422-226bb6bc6dd3-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/48a0901f-8b2b-4278-8422-226bb6bc6dd3-istio-data\") pod \"48a0901f-8b2b-4278-8422-226bb6bc6dd3\" (UID: \"48a0901f-8b2b-4278-8422-226bb6bc6dd3\") " Oct 30 21:05:39.564103 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:05:39.563924 5378 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/48a0901f-8b2b-4278-8422-226bb6bc6dd3-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/48a0901f-8b2b-4278-8422-226bb6bc6dd3-istio-podinfo\") pod \"48a0901f-8b2b-4278-8422-226bb6bc6dd3\" (UID: \"48a0901f-8b2b-4278-8422-226bb6bc6dd3\") " Oct 30 21:05:39.564103 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:05:39.564001 5378 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/48a0901f-8b2b-4278-8422-226bb6bc6dd3-istio-data" pod "48a0901f-8b2b-4278-8422-226bb6bc6dd3" (UID: "48a0901f-8b2b-4278-8422-226bb6bc6dd3"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:05:39.564103 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:05:39.564018 5378 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/48a0901f-8b2b-4278-8422-226bb6bc6dd3-credential-socket" pod "48a0901f-8b2b-4278-8422-226bb6bc6dd3" (UID: "48a0901f-8b2b-4278-8422-226bb6bc6dd3"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:05:39.564103 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:05:39.564090 5378 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/48a0901f-8b2b-4278-8422-226bb6bc6dd3-kube-api-access-c9xkz\" (UniqueName: \"kubernetes.io/projected/48a0901f-8b2b-4278-8422-226bb6bc6dd3-kube-api-access-c9xkz\") pod \"48a0901f-8b2b-4278-8422-226bb6bc6dd3\" (UID: \"48a0901f-8b2b-4278-8422-226bb6bc6dd3\") " Oct 30 21:05:39.564297 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:05:39.564126 5378 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/48a0901f-8b2b-4278-8422-226bb6bc6dd3-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/48a0901f-8b2b-4278-8422-226bb6bc6dd3-istio-envoy\") pod \"48a0901f-8b2b-4278-8422-226bb6bc6dd3\" (UID: \"48a0901f-8b2b-4278-8422-226bb6bc6dd3\") " Oct 30 21:05:39.564297 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:05:39.564204 5378 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/48a0901f-8b2b-4278-8422-226bb6bc6dd3-istio-token\" (UniqueName: \"kubernetes.io/projected/48a0901f-8b2b-4278-8422-226bb6bc6dd3-istio-token\") pod \"48a0901f-8b2b-4278-8422-226bb6bc6dd3\" (UID: \"48a0901f-8b2b-4278-8422-226bb6bc6dd3\") " Oct 30 21:05:39.564297 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:05:39.564239 5378 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/48a0901f-8b2b-4278-8422-226bb6bc6dd3-istiod-ca-cert" pod "48a0901f-8b2b-4278-8422-226bb6bc6dd3" (UID: "48a0901f-8b2b-4278-8422-226bb6bc6dd3"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:05:39.564974 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:05:39.564948 5378 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/48a0901f-8b2b-4278-8422-226bb6bc6dd3-credential-socket\") on node \"ip-10-0-138-149.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:05:39.565152 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:05:39.564976 5378 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/48a0901f-8b2b-4278-8422-226bb6bc6dd3-istiod-ca-cert\") on node \"ip-10-0-138-149.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:05:39.565152 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:05:39.564990 5378 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/48a0901f-8b2b-4278-8422-226bb6bc6dd3-istio-data\") on node \"ip-10-0-138-149.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:05:39.565152 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:05:39.565001 5378 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/48a0901f-8b2b-4278-8422-226bb6bc6dd3-workload-certs\") on node \"ip-10-0-138-149.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:05:39.565152 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:05:39.565014 5378 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/48a0901f-8b2b-4278-8422-226bb6bc6dd3-workload-socket\") on node \"ip-10-0-138-149.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:05:39.567349 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:05:39.567309 5378 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/48a0901f-8b2b-4278-8422-226bb6bc6dd3-istio-envoy" pod "48a0901f-8b2b-4278-8422-226bb6bc6dd3" (UID: "48a0901f-8b2b-4278-8422-226bb6bc6dd3"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:05:39.567547 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:05:39.567531 5378 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/48a0901f-8b2b-4278-8422-226bb6bc6dd3-kube-api-access-c9xkz" pod "48a0901f-8b2b-4278-8422-226bb6bc6dd3" (UID: "48a0901f-8b2b-4278-8422-226bb6bc6dd3"). InnerVolumeSpecName "kube-api-access-c9xkz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:05:39.567713 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:05:39.567680 5378 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/48a0901f-8b2b-4278-8422-226bb6bc6dd3-istio-token" pod "48a0901f-8b2b-4278-8422-226bb6bc6dd3" (UID: "48a0901f-8b2b-4278-8422-226bb6bc6dd3"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:05:39.567884 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:05:39.567865 5378 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/48a0901f-8b2b-4278-8422-226bb6bc6dd3-istio-podinfo" pod "48a0901f-8b2b-4278-8422-226bb6bc6dd3" (UID: "48a0901f-8b2b-4278-8422-226bb6bc6dd3"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 21:05:39.624713 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:05:39.624689 5378 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-781058-openshift-default-55fb495b7-khvpl" Oct 30 21:05:39.634965 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:05:39.634924 5378 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-781058-openshift-default-55fb495b7-2c7gl" Oct 30 21:05:39.666252 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:05:39.666179 5378 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/afcb5db1-70aa-4758-90e6-e21b112d48f4-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/afcb5db1-70aa-4758-90e6-e21b112d48f4-istio-data\") pod \"afcb5db1-70aa-4758-90e6-e21b112d48f4\" (UID: \"afcb5db1-70aa-4758-90e6-e21b112d48f4\") " Oct 30 21:05:39.666252 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:05:39.666237 5378 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/507cd7ee-c139-489c-a520-292d8ebd91e4-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/507cd7ee-c139-489c-a520-292d8ebd91e4-istiod-ca-cert\") pod \"507cd7ee-c139-489c-a520-292d8ebd91e4\" (UID: \"507cd7ee-c139-489c-a520-292d8ebd91e4\") " Oct 30 21:05:39.666460 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:05:39.666319 5378 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/d43ee748-feb2-4556-be54-3662f8dfa98d-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d43ee748-feb2-4556-be54-3662f8dfa98d-istio-podinfo\") pod \"d43ee748-feb2-4556-be54-3662f8dfa98d\" (UID: \"d43ee748-feb2-4556-be54-3662f8dfa98d\") " Oct 30 21:05:39.666460 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:05:39.666365 5378 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/507cd7ee-c139-489c-a520-292d8ebd91e4-kube-api-access-5tzd9\" (UniqueName: \"kubernetes.io/projected/507cd7ee-c139-489c-a520-292d8ebd91e4-kube-api-access-5tzd9\") pod \"507cd7ee-c139-489c-a520-292d8ebd91e4\" (UID: \"507cd7ee-c139-489c-a520-292d8ebd91e4\") " Oct 30 21:05:39.666460 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:05:39.666398 5378 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d43ee748-feb2-4556-be54-3662f8dfa98d-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d43ee748-feb2-4556-be54-3662f8dfa98d-workload-socket\") pod \"d43ee748-feb2-4556-be54-3662f8dfa98d\" (UID: \"d43ee748-feb2-4556-be54-3662f8dfa98d\") " Oct 30 21:05:39.666460 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:05:39.666436 5378 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/afcb5db1-70aa-4758-90e6-e21b112d48f4-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/afcb5db1-70aa-4758-90e6-e21b112d48f4-credential-socket\") pod \"afcb5db1-70aa-4758-90e6-e21b112d48f4\" (UID: \"afcb5db1-70aa-4758-90e6-e21b112d48f4\") " Oct 30 21:05:39.666690 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:05:39.666439 5378 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/afcb5db1-70aa-4758-90e6-e21b112d48f4-istio-data" pod "afcb5db1-70aa-4758-90e6-e21b112d48f4" (UID: "afcb5db1-70aa-4758-90e6-e21b112d48f4"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:05:39.666690 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:05:39.666480 5378 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/507cd7ee-c139-489c-a520-292d8ebd91e4-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/507cd7ee-c139-489c-a520-292d8ebd91e4-workload-socket\") pod \"507cd7ee-c139-489c-a520-292d8ebd91e4\" (UID: \"507cd7ee-c139-489c-a520-292d8ebd91e4\") " Oct 30 21:05:39.666690 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:05:39.666518 5378 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/afcb5db1-70aa-4758-90e6-e21b112d48f4-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/afcb5db1-70aa-4758-90e6-e21b112d48f4-istio-podinfo\") pod \"afcb5db1-70aa-4758-90e6-e21b112d48f4\" (UID: \"afcb5db1-70aa-4758-90e6-e21b112d48f4\") " Oct 30 21:05:39.666690 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:05:39.666609 5378 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d43ee748-feb2-4556-be54-3662f8dfa98d-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d43ee748-feb2-4556-be54-3662f8dfa98d-workload-certs\") pod \"d43ee748-feb2-4556-be54-3662f8dfa98d\" (UID: \"d43ee748-feb2-4556-be54-3662f8dfa98d\") " Oct 30 21:05:39.666690 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:05:39.666656 5378 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/507cd7ee-c139-489c-a520-292d8ebd91e4-istio-token\" (UniqueName: \"kubernetes.io/projected/507cd7ee-c139-489c-a520-292d8ebd91e4-istio-token\") pod \"507cd7ee-c139-489c-a520-292d8ebd91e4\" (UID: \"507cd7ee-c139-489c-a520-292d8ebd91e4\") " Oct 30 21:05:39.666992 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:05:39.666695 5378 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/afcb5db1-70aa-4758-90e6-e21b112d48f4-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/afcb5db1-70aa-4758-90e6-e21b112d48f4-workload-certs\") pod \"afcb5db1-70aa-4758-90e6-e21b112d48f4\" (UID: \"afcb5db1-70aa-4758-90e6-e21b112d48f4\") " Oct 30 21:05:39.666992 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:05:39.666741 5378 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/afcb5db1-70aa-4758-90e6-e21b112d48f4-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/afcb5db1-70aa-4758-90e6-e21b112d48f4-istiod-ca-cert\") pod \"afcb5db1-70aa-4758-90e6-e21b112d48f4\" (UID: \"afcb5db1-70aa-4758-90e6-e21b112d48f4\") " Oct 30 21:05:39.666992 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:05:39.666745 5378 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/afcb5db1-70aa-4758-90e6-e21b112d48f4-credential-socket" pod "afcb5db1-70aa-4758-90e6-e21b112d48f4" (UID: "afcb5db1-70aa-4758-90e6-e21b112d48f4"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:05:39.666992 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:05:39.666626 5378 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/507cd7ee-c139-489c-a520-292d8ebd91e4-istiod-ca-cert" pod "507cd7ee-c139-489c-a520-292d8ebd91e4" (UID: "507cd7ee-c139-489c-a520-292d8ebd91e4"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:05:39.666992 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:05:39.666774 5378 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d43ee748-feb2-4556-be54-3662f8dfa98d-workload-socket" pod "d43ee748-feb2-4556-be54-3662f8dfa98d" (UID: "d43ee748-feb2-4556-be54-3662f8dfa98d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:05:39.666992 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:05:39.666848 5378 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/507cd7ee-c139-489c-a520-292d8ebd91e4-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/507cd7ee-c139-489c-a520-292d8ebd91e4-istio-podinfo\") pod \"507cd7ee-c139-489c-a520-292d8ebd91e4\" (UID: \"507cd7ee-c139-489c-a520-292d8ebd91e4\") " Oct 30 21:05:39.666992 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:05:39.666909 5378 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/afcb5db1-70aa-4758-90e6-e21b112d48f4-istio-token\" (UniqueName: \"kubernetes.io/projected/afcb5db1-70aa-4758-90e6-e21b112d48f4-istio-token\") pod \"afcb5db1-70aa-4758-90e6-e21b112d48f4\" (UID: \"afcb5db1-70aa-4758-90e6-e21b112d48f4\") " Oct 30 21:05:39.666992 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:05:39.666940 5378 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d43ee748-feb2-4556-be54-3662f8dfa98d-workload-certs" pod "d43ee748-feb2-4556-be54-3662f8dfa98d" (UID: "d43ee748-feb2-4556-be54-3662f8dfa98d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:05:39.666992 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:05:39.666946 5378 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d43ee748-feb2-4556-be54-3662f8dfa98d-kube-api-access-4fgvd\" (UniqueName: \"kubernetes.io/projected/d43ee748-feb2-4556-be54-3662f8dfa98d-kube-api-access-4fgvd\") pod \"d43ee748-feb2-4556-be54-3662f8dfa98d\" (UID: \"d43ee748-feb2-4556-be54-3662f8dfa98d\") " Oct 30 21:05:39.667441 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:05:39.667046 5378 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/507cd7ee-c139-489c-a520-292d8ebd91e4-workload-socket" pod "507cd7ee-c139-489c-a520-292d8ebd91e4" (UID: "507cd7ee-c139-489c-a520-292d8ebd91e4"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:05:39.667441 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:05:39.667083 5378 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/507cd7ee-c139-489c-a520-292d8ebd91e4-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/507cd7ee-c139-489c-a520-292d8ebd91e4-credential-socket\") pod \"507cd7ee-c139-489c-a520-292d8ebd91e4\" (UID: \"507cd7ee-c139-489c-a520-292d8ebd91e4\") " Oct 30 21:05:39.667441 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:05:39.667145 5378 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/afcb5db1-70aa-4758-90e6-e21b112d48f4-istiod-ca-cert" pod "afcb5db1-70aa-4758-90e6-e21b112d48f4" (UID: "afcb5db1-70aa-4758-90e6-e21b112d48f4"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:05:39.667441 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:05:39.667160 5378 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d43ee748-feb2-4556-be54-3662f8dfa98d-istio-token\" (UniqueName: \"kubernetes.io/projected/d43ee748-feb2-4556-be54-3662f8dfa98d-istio-token\") pod \"d43ee748-feb2-4556-be54-3662f8dfa98d\" (UID: \"d43ee748-feb2-4556-be54-3662f8dfa98d\") " Oct 30 21:05:39.667441 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:05:39.667236 5378 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/507cd7ee-c139-489c-a520-292d8ebd91e4-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/507cd7ee-c139-489c-a520-292d8ebd91e4-istio-envoy\") pod \"507cd7ee-c139-489c-a520-292d8ebd91e4\" (UID: \"507cd7ee-c139-489c-a520-292d8ebd91e4\") " Oct 30 21:05:39.667441 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:05:39.667323 5378 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/afcb5db1-70aa-4758-90e6-e21b112d48f4-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/afcb5db1-70aa-4758-90e6-e21b112d48f4-istio-envoy\") pod \"afcb5db1-70aa-4758-90e6-e21b112d48f4\" (UID: \"afcb5db1-70aa-4758-90e6-e21b112d48f4\") " Oct 30 21:05:39.667441 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:05:39.667352 5378 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/507cd7ee-c139-489c-a520-292d8ebd91e4-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/507cd7ee-c139-489c-a520-292d8ebd91e4-istio-data\") pod \"507cd7ee-c139-489c-a520-292d8ebd91e4\" (UID: \"507cd7ee-c139-489c-a520-292d8ebd91e4\") " Oct 30 21:05:39.667441 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:05:39.667400 5378 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/afcb5db1-70aa-4758-90e6-e21b112d48f4-workload-certs" pod "afcb5db1-70aa-4758-90e6-e21b112d48f4" (UID: "afcb5db1-70aa-4758-90e6-e21b112d48f4"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:05:39.667441 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:05:39.667408 5378 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d43ee748-feb2-4556-be54-3662f8dfa98d-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d43ee748-feb2-4556-be54-3662f8dfa98d-istiod-ca-cert\") pod \"d43ee748-feb2-4556-be54-3662f8dfa98d\" (UID: \"d43ee748-feb2-4556-be54-3662f8dfa98d\") " Oct 30 21:05:39.667912 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:05:39.667499 5378 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/afcb5db1-70aa-4758-90e6-e21b112d48f4-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/afcb5db1-70aa-4758-90e6-e21b112d48f4-workload-socket\") pod \"afcb5db1-70aa-4758-90e6-e21b112d48f4\" (UID: \"afcb5db1-70aa-4758-90e6-e21b112d48f4\") " Oct 30 21:05:39.667912 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:05:39.667597 5378 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/507cd7ee-c139-489c-a520-292d8ebd91e4-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/507cd7ee-c139-489c-a520-292d8ebd91e4-workload-certs\") pod \"507cd7ee-c139-489c-a520-292d8ebd91e4\" (UID: \"507cd7ee-c139-489c-a520-292d8ebd91e4\") " Oct 30 21:05:39.667912 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:05:39.667632 5378 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d43ee748-feb2-4556-be54-3662f8dfa98d-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d43ee748-feb2-4556-be54-3662f8dfa98d-istio-data\") pod \"d43ee748-feb2-4556-be54-3662f8dfa98d\" (UID: \"d43ee748-feb2-4556-be54-3662f8dfa98d\") " Oct 30 21:05:39.667912 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:05:39.667704 5378 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d43ee748-feb2-4556-be54-3662f8dfa98d-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d43ee748-feb2-4556-be54-3662f8dfa98d-credential-socket\") pod \"d43ee748-feb2-4556-be54-3662f8dfa98d\" (UID: \"d43ee748-feb2-4556-be54-3662f8dfa98d\") " Oct 30 21:05:39.667912 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:05:39.667783 5378 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d43ee748-feb2-4556-be54-3662f8dfa98d-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d43ee748-feb2-4556-be54-3662f8dfa98d-istio-envoy\") pod \"d43ee748-feb2-4556-be54-3662f8dfa98d\" (UID: \"d43ee748-feb2-4556-be54-3662f8dfa98d\") " Oct 30 21:05:39.667912 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:05:39.667870 5378 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/afcb5db1-70aa-4758-90e6-e21b112d48f4-kube-api-access-x6cdp\" (UniqueName: \"kubernetes.io/projected/afcb5db1-70aa-4758-90e6-e21b112d48f4-kube-api-access-x6cdp\") pod \"afcb5db1-70aa-4758-90e6-e21b112d48f4\" (UID: \"afcb5db1-70aa-4758-90e6-e21b112d48f4\") " Oct 30 21:05:39.668857 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:05:39.668827 5378 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/afcb5db1-70aa-4758-90e6-e21b112d48f4-istio-data\") on node \"ip-10-0-138-149.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:05:39.668964 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:05:39.668861 5378 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/507cd7ee-c139-489c-a520-292d8ebd91e4-istiod-ca-cert\") on node \"ip-10-0-138-149.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:05:39.668964 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:05:39.668878 5378 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c9xkz\" (UniqueName: \"kubernetes.io/projected/48a0901f-8b2b-4278-8422-226bb6bc6dd3-kube-api-access-c9xkz\") on node \"ip-10-0-138-149.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:05:39.668964 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:05:39.668895 5378 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/48a0901f-8b2b-4278-8422-226bb6bc6dd3-istio-envoy\") on node \"ip-10-0-138-149.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:05:39.668964 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:05:39.668911 5378 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d43ee748-feb2-4556-be54-3662f8dfa98d-workload-socket\") on node \"ip-10-0-138-149.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:05:39.668964 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:05:39.668925 5378 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/afcb5db1-70aa-4758-90e6-e21b112d48f4-credential-socket\") on node \"ip-10-0-138-149.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:05:39.668964 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:05:39.668942 5378 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/48a0901f-8b2b-4278-8422-226bb6bc6dd3-istio-token\") on node \"ip-10-0-138-149.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:05:39.668964 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:05:39.668956 5378 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/507cd7ee-c139-489c-a520-292d8ebd91e4-workload-socket\") on node \"ip-10-0-138-149.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:05:39.669288 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:05:39.668970 5378 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d43ee748-feb2-4556-be54-3662f8dfa98d-workload-certs\") on node \"ip-10-0-138-149.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:05:39.669288 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:05:39.668986 5378 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/afcb5db1-70aa-4758-90e6-e21b112d48f4-workload-certs\") on node \"ip-10-0-138-149.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:05:39.669288 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:05:39.669000 5378 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/afcb5db1-70aa-4758-90e6-e21b112d48f4-istiod-ca-cert\") on node \"ip-10-0-138-149.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:05:39.669288 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:05:39.669015 5378 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/48a0901f-8b2b-4278-8422-226bb6bc6dd3-istio-podinfo\") on node \"ip-10-0-138-149.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:05:39.669288 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:05:39.669203 5378 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d43ee748-feb2-4556-be54-3662f8dfa98d-istio-podinfo" pod "d43ee748-feb2-4556-be54-3662f8dfa98d" (UID: "d43ee748-feb2-4556-be54-3662f8dfa98d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 21:05:39.669288 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:05:39.669278 5378 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/507cd7ee-c139-489c-a520-292d8ebd91e4-kube-api-access-5tzd9" pod "507cd7ee-c139-489c-a520-292d8ebd91e4" (UID: "507cd7ee-c139-489c-a520-292d8ebd91e4"). InnerVolumeSpecName "kube-api-access-5tzd9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:05:39.669656 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:05:39.669554 5378 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d43ee748-feb2-4556-be54-3662f8dfa98d-istiod-ca-cert" pod "d43ee748-feb2-4556-be54-3662f8dfa98d" (UID: "d43ee748-feb2-4556-be54-3662f8dfa98d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:05:39.670250 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:05:39.670037 5378 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d43ee748-feb2-4556-be54-3662f8dfa98d-kube-api-access-4fgvd" pod "d43ee748-feb2-4556-be54-3662f8dfa98d" (UID: "d43ee748-feb2-4556-be54-3662f8dfa98d"). InnerVolumeSpecName "kube-api-access-4fgvd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:05:39.670360 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:05:39.670287 5378 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/afcb5db1-70aa-4758-90e6-e21b112d48f4-istio-podinfo" pod "afcb5db1-70aa-4758-90e6-e21b112d48f4" (UID: "afcb5db1-70aa-4758-90e6-e21b112d48f4"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 21:05:39.670606 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:05:39.670449 5378 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/507cd7ee-c139-489c-a520-292d8ebd91e4-credential-socket" pod "507cd7ee-c139-489c-a520-292d8ebd91e4" (UID: "507cd7ee-c139-489c-a520-292d8ebd91e4"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:05:39.670606 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:05:39.670447 5378 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/afcb5db1-70aa-4758-90e6-e21b112d48f4-istio-token" pod "afcb5db1-70aa-4758-90e6-e21b112d48f4" (UID: "afcb5db1-70aa-4758-90e6-e21b112d48f4"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:05:39.670606 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:05:39.670470 5378 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d43ee748-feb2-4556-be54-3662f8dfa98d-istio-data" pod "d43ee748-feb2-4556-be54-3662f8dfa98d" (UID: "d43ee748-feb2-4556-be54-3662f8dfa98d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:05:39.670811 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:05:39.670680 5378 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/afcb5db1-70aa-4758-90e6-e21b112d48f4-workload-socket" pod "afcb5db1-70aa-4758-90e6-e21b112d48f4" (UID: "afcb5db1-70aa-4758-90e6-e21b112d48f4"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:05:39.671011 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:05:39.670974 5378 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/507cd7ee-c139-489c-a520-292d8ebd91e4-istio-data" pod "507cd7ee-c139-489c-a520-292d8ebd91e4" (UID: "507cd7ee-c139-489c-a520-292d8ebd91e4"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:05:39.671114 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:05:39.671034 5378 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/507cd7ee-c139-489c-a520-292d8ebd91e4-workload-certs" pod "507cd7ee-c139-489c-a520-292d8ebd91e4" (UID: "507cd7ee-c139-489c-a520-292d8ebd91e4"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:05:39.671205 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:05:39.671188 5378 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d43ee748-feb2-4556-be54-3662f8dfa98d-credential-socket" pod "d43ee748-feb2-4556-be54-3662f8dfa98d" (UID: "d43ee748-feb2-4556-be54-3662f8dfa98d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:05:39.671710 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:05:39.671682 5378 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/507cd7ee-c139-489c-a520-292d8ebd91e4-istio-podinfo" pod "507cd7ee-c139-489c-a520-292d8ebd91e4" (UID: "507cd7ee-c139-489c-a520-292d8ebd91e4"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 21:05:39.672068 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:05:39.672042 5378 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/afcb5db1-70aa-4758-90e6-e21b112d48f4-kube-api-access-x6cdp" pod "afcb5db1-70aa-4758-90e6-e21b112d48f4" (UID: "afcb5db1-70aa-4758-90e6-e21b112d48f4"). InnerVolumeSpecName "kube-api-access-x6cdp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:05:39.672380 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:05:39.672343 5378 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/507cd7ee-c139-489c-a520-292d8ebd91e4-istio-envoy" pod "507cd7ee-c139-489c-a520-292d8ebd91e4" (UID: "507cd7ee-c139-489c-a520-292d8ebd91e4"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:05:39.672446 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:05:39.672378 5378 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/507cd7ee-c139-489c-a520-292d8ebd91e4-istio-token" pod "507cd7ee-c139-489c-a520-292d8ebd91e4" (UID: "507cd7ee-c139-489c-a520-292d8ebd91e4"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:05:39.672446 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:05:39.672385 5378 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/afcb5db1-70aa-4758-90e6-e21b112d48f4-istio-envoy" pod "afcb5db1-70aa-4758-90e6-e21b112d48f4" (UID: "afcb5db1-70aa-4758-90e6-e21b112d48f4"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:05:39.672711 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:05:39.672694 5378 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d43ee748-feb2-4556-be54-3662f8dfa98d-istio-token" pod "d43ee748-feb2-4556-be54-3662f8dfa98d" (UID: "d43ee748-feb2-4556-be54-3662f8dfa98d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:05:39.673158 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:05:39.673135 5378 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d43ee748-feb2-4556-be54-3662f8dfa98d-istio-envoy" pod "d43ee748-feb2-4556-be54-3662f8dfa98d" (UID: "d43ee748-feb2-4556-be54-3662f8dfa98d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:05:39.771521 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:05:39.771487 5378 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d43ee748-feb2-4556-be54-3662f8dfa98d-credential-socket\") on node \"ip-10-0-138-149.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:05:39.771706 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:05:39.771525 5378 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d43ee748-feb2-4556-be54-3662f8dfa98d-istio-envoy\") on node \"ip-10-0-138-149.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:05:39.771706 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:05:39.771541 5378 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x6cdp\" (UniqueName: \"kubernetes.io/projected/afcb5db1-70aa-4758-90e6-e21b112d48f4-kube-api-access-x6cdp\") on node \"ip-10-0-138-149.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:05:39.771706 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:05:39.771558 5378 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d43ee748-feb2-4556-be54-3662f8dfa98d-istio-podinfo\") on node \"ip-10-0-138-149.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:05:39.771706 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:05:39.771588 5378 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5tzd9\" (UniqueName: \"kubernetes.io/projected/507cd7ee-c139-489c-a520-292d8ebd91e4-kube-api-access-5tzd9\") on node \"ip-10-0-138-149.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:05:39.771706 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:05:39.771602 5378 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/afcb5db1-70aa-4758-90e6-e21b112d48f4-istio-podinfo\") on node \"ip-10-0-138-149.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:05:39.771706 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:05:39.771617 5378 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/507cd7ee-c139-489c-a520-292d8ebd91e4-istio-token\") on node \"ip-10-0-138-149.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:05:39.771706 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:05:39.771630 5378 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/507cd7ee-c139-489c-a520-292d8ebd91e4-istio-podinfo\") on node \"ip-10-0-138-149.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:05:39.771706 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:05:39.771644 5378 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/afcb5db1-70aa-4758-90e6-e21b112d48f4-istio-token\") on node \"ip-10-0-138-149.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:05:39.771706 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:05:39.771657 5378 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4fgvd\" (UniqueName: \"kubernetes.io/projected/d43ee748-feb2-4556-be54-3662f8dfa98d-kube-api-access-4fgvd\") on node \"ip-10-0-138-149.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:05:39.771706 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:05:39.771673 5378 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/507cd7ee-c139-489c-a520-292d8ebd91e4-credential-socket\") on node \"ip-10-0-138-149.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:05:39.771706 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:05:39.771686 5378 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d43ee748-feb2-4556-be54-3662f8dfa98d-istio-token\") on node \"ip-10-0-138-149.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:05:39.771706 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:05:39.771699 5378 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/507cd7ee-c139-489c-a520-292d8ebd91e4-istio-envoy\") on node \"ip-10-0-138-149.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:05:39.772079 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:05:39.771712 5378 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/afcb5db1-70aa-4758-90e6-e21b112d48f4-istio-envoy\") on node \"ip-10-0-138-149.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:05:39.772079 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:05:39.771725 5378 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/507cd7ee-c139-489c-a520-292d8ebd91e4-istio-data\") on node \"ip-10-0-138-149.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:05:39.772079 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:05:39.771738 5378 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d43ee748-feb2-4556-be54-3662f8dfa98d-istiod-ca-cert\") on node \"ip-10-0-138-149.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:05:39.772079 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:05:39.771752 5378 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/afcb5db1-70aa-4758-90e6-e21b112d48f4-workload-socket\") on node \"ip-10-0-138-149.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:05:39.772079 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:05:39.771765 5378 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/507cd7ee-c139-489c-a520-292d8ebd91e4-workload-certs\") on node \"ip-10-0-138-149.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:05:39.772079 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:05:39.771778 5378 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d43ee748-feb2-4556-be54-3662f8dfa98d-istio-data\") on node \"ip-10-0-138-149.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:05:40.327661 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:05:40.327626 5378 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-19db3d5f-55c59f9b9d-h4zbn"] Oct 30 21:05:40.328333 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:05:40.328317 5378 cpu_manager.go:414] "containerMap: removing container" podUID="d43ee748-feb2-4556-be54-3662f8dfa98d" containerName="istio-proxy" Oct 30 21:05:40.328380 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:05:40.328335 5378 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d43ee748-feb2-4556-be54-3662f8dfa98d" containerName="istio-proxy" Oct 30 21:05:40.328380 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:05:40.328373 5378 cpu_manager.go:414] "containerMap: removing container" podUID="48a0901f-8b2b-4278-8422-226bb6bc6dd3" containerName="istio-proxy" Oct 30 21:05:40.328380 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:05:40.328378 5378 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="48a0901f-8b2b-4278-8422-226bb6bc6dd3" containerName="istio-proxy" Oct 30 21:05:40.328473 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:05:40.328388 5378 cpu_manager.go:414] "containerMap: removing container" podUID="afcb5db1-70aa-4758-90e6-e21b112d48f4" containerName="istio-proxy" Oct 30 21:05:40.328473 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:05:40.328393 5378 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="afcb5db1-70aa-4758-90e6-e21b112d48f4" containerName="istio-proxy" Oct 30 21:05:40.328473 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:05:40.328413 5378 cpu_manager.go:414] "containerMap: removing container" podUID="507cd7ee-c139-489c-a520-292d8ebd91e4" containerName="istio-proxy" Oct 30 21:05:40.328473 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:05:40.328417 5378 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="507cd7ee-c139-489c-a520-292d8ebd91e4" containerName="istio-proxy" Oct 30 21:05:40.328618 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:05:40.328507 5378 memory_manager.go:365] "RemoveStaleState removing state" podUID="507cd7ee-c139-489c-a520-292d8ebd91e4" containerName="istio-proxy" Oct 30 21:05:40.328618 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:05:40.328518 5378 memory_manager.go:365] "RemoveStaleState removing state" podUID="d43ee748-feb2-4556-be54-3662f8dfa98d" containerName="istio-proxy" Oct 30 21:05:40.328618 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:05:40.328528 5378 memory_manager.go:365] "RemoveStaleState removing state" podUID="48a0901f-8b2b-4278-8422-226bb6bc6dd3" containerName="istio-proxy" Oct 30 21:05:40.328618 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:05:40.328536 5378 memory_manager.go:365] "RemoveStaleState removing state" podUID="afcb5db1-70aa-4758-90e6-e21b112d48f4" containerName="istio-proxy" Oct 30 21:05:40.331696 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:05:40.331682 5378 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-19db3d5f-55c59f9b9d-h4zbn" Oct 30 21:05:40.382146 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:05:40.382120 5378 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6bb33be9-e451-42ba-a7f9-c9a4ebbbf169-metrics-certs\") pod \"maas-api-e2e-mt-19db3d5f-55c59f9b9d-h4zbn\" (UID: \"6bb33be9-e451-42ba-a7f9-c9a4ebbbf169\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-19db3d5f-55c59f9b9d-h4zbn" Oct 30 21:05:40.382292 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:05:40.382247 5378 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mjb2k\" (UniqueName: \"kubernetes.io/projected/6bb33be9-e451-42ba-a7f9-c9a4ebbbf169-kube-api-access-mjb2k\") pod \"maas-api-e2e-mt-19db3d5f-55c59f9b9d-h4zbn\" (UID: \"6bb33be9-e451-42ba-a7f9-c9a4ebbbf169\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-19db3d5f-55c59f9b9d-h4zbn" Oct 30 21:05:40.382405 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:05:40.382386 5378 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6bb33be9-e451-42ba-a7f9-c9a4ebbbf169-openshift-service-ca\") pod \"maas-api-e2e-mt-19db3d5f-55c59f9b9d-h4zbn\" (UID: \"6bb33be9-e451-42ba-a7f9-c9a4ebbbf169\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-19db3d5f-55c59f9b9d-h4zbn" Oct 30 21:05:40.382455 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:05:40.382432 5378 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6bb33be9-e451-42ba-a7f9-c9a4ebbbf169-maas-api-tls\") pod \"maas-api-e2e-mt-19db3d5f-55c59f9b9d-h4zbn\" (UID: \"6bb33be9-e451-42ba-a7f9-c9a4ebbbf169\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-19db3d5f-55c59f9b9d-h4zbn" Oct 30 21:05:40.389030 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:05:40.388970 5378 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-19db3d5f-55c59f9b9d-h4zbn"] Oct 30 21:05:40.389552 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:05:40.389537 5378 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-mt-19db3d5f-serving-cert\"" Oct 30 21:05:40.472892 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:05:40.472866 5378 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-781058-openshift-default-55fb495b7-hpgv5" Oct 30 21:05:40.473309 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:05:40.472906 5378 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-781058-openshift-default-55fb495b7-hpgv5" event={"ID":"d43ee748-feb2-4556-be54-3662f8dfa98d","Type":"ContainerDied","Data":"3bf9d6cf7c660dcd7f135f5c46ad910548ceae44eef66e936654ff2ead417481"} Oct 30 21:05:40.473309 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:05:40.472969 5378 scope.go:122] "RemoveContainer" containerID="2dfda93c6c9252b225103f327e91f54fc6eded5a91be4d9f9d7e8893160a3184" Oct 30 21:05:40.474515 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:05:40.474489 5378 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-781058-openshift-default-55fb495b7-2c7gl" Oct 30 21:05:40.474659 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:05:40.474518 5378 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-781058-openshift-default-55fb495b7-2c7gl" event={"ID":"507cd7ee-c139-489c-a520-292d8ebd91e4","Type":"ContainerDied","Data":"2f027f9441458a04ae2b974849a14470830c98de45ace5fbdeaae974418a267c"} Oct 30 21:05:40.476348 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:05:40.476310 5378 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-781058-openshift-default-55fb495b7-khvpl" event={"ID":"afcb5db1-70aa-4758-90e6-e21b112d48f4","Type":"ContainerDied","Data":"7904f41e16b8f8abff2b5e054eb6e4b1147c84c9ff11305442c79027fbf7d00e"} Oct 30 21:05:40.476463 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:05:40.476323 5378 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-781058-openshift-default-55fb495b7-khvpl" Oct 30 21:05:40.476463 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:05:40.476327 5378 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-781058-openshift-default-55fb495b7-5zx7k" Oct 30 21:05:40.483808 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:05:40.483787 5378 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6bb33be9-e451-42ba-a7f9-c9a4ebbbf169-openshift-service-ca\") pod \"maas-api-e2e-mt-19db3d5f-55c59f9b9d-h4zbn\" (UID: \"6bb33be9-e451-42ba-a7f9-c9a4ebbbf169\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-19db3d5f-55c59f9b9d-h4zbn" Oct 30 21:05:40.483913 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:05:40.483847 5378 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6bb33be9-e451-42ba-a7f9-c9a4ebbbf169-maas-api-tls\") pod \"maas-api-e2e-mt-19db3d5f-55c59f9b9d-h4zbn\" (UID: \"6bb33be9-e451-42ba-a7f9-c9a4ebbbf169\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-19db3d5f-55c59f9b9d-h4zbn" Oct 30 21:05:40.484014 ip-10-0-138-149 kubenswrapper[5378]: E1030 21:05:40.483995 5378 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-mt-19db3d5f-serving-cert: secret "maas-api-e2e-mt-19db3d5f-serving-cert" not found Oct 30 21:05:40.484082 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:05:40.483996 5378 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6bb33be9-e451-42ba-a7f9-c9a4ebbbf169-metrics-certs\") pod \"maas-api-e2e-mt-19db3d5f-55c59f9b9d-h4zbn\" (UID: \"6bb33be9-e451-42ba-a7f9-c9a4ebbbf169\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-19db3d5f-55c59f9b9d-h4zbn" Oct 30 21:05:40.484082 ip-10-0-138-149 kubenswrapper[5378]: E1030 21:05:40.484065 5378 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6bb33be9-e451-42ba-a7f9-c9a4ebbbf169-maas-api-tls podName:6bb33be9-e451-42ba-a7f9-c9a4ebbbf169 nodeName:}" failed. No retries permitted until 2026-10-30 21:05:40.984045194 +0000 UTC m=+1631.412159096 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/6bb33be9-e451-42ba-a7f9-c9a4ebbbf169-maas-api-tls") pod "maas-api-e2e-mt-19db3d5f-55c59f9b9d-h4zbn" (UID: "6bb33be9-e451-42ba-a7f9-c9a4ebbbf169") : secret "maas-api-e2e-mt-19db3d5f-serving-cert" not found Oct 30 21:05:40.484222 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:05:40.484203 5378 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mjb2k\" (UniqueName: \"kubernetes.io/projected/6bb33be9-e451-42ba-a7f9-c9a4ebbbf169-kube-api-access-mjb2k\") pod \"maas-api-e2e-mt-19db3d5f-55c59f9b9d-h4zbn\" (UID: \"6bb33be9-e451-42ba-a7f9-c9a4ebbbf169\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-19db3d5f-55c59f9b9d-h4zbn" Oct 30 21:05:40.484505 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:05:40.484481 5378 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6bb33be9-e451-42ba-a7f9-c9a4ebbbf169-openshift-service-ca\") pod \"maas-api-e2e-mt-19db3d5f-55c59f9b9d-h4zbn\" (UID: \"6bb33be9-e451-42ba-a7f9-c9a4ebbbf169\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-19db3d5f-55c59f9b9d-h4zbn" Oct 30 21:05:40.484910 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:05:40.484892 5378 scope.go:122] "RemoveContainer" containerID="6b92315664f5650233d2906cc50f29a6d230ad23d5fe304db0b0025ba3483c3e" Oct 30 21:05:40.486405 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:05:40.486388 5378 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6bb33be9-e451-42ba-a7f9-c9a4ebbbf169-metrics-certs\") pod \"maas-api-e2e-mt-19db3d5f-55c59f9b9d-h4zbn\" (UID: \"6bb33be9-e451-42ba-a7f9-c9a4ebbbf169\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-19db3d5f-55c59f9b9d-h4zbn" Oct 30 21:05:40.508315 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:05:40.507816 5378 scope.go:122] "RemoveContainer" containerID="c682db6caa42bbefaa74a7509ac7c21897323acff7831c3caf7369bba15a262f" Oct 30 21:05:40.544646 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:05:40.544618 5378 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mjb2k\" (UniqueName: \"kubernetes.io/projected/6bb33be9-e451-42ba-a7f9-c9a4ebbbf169-kube-api-access-mjb2k\") pod \"maas-api-e2e-mt-19db3d5f-55c59f9b9d-h4zbn\" (UID: \"6bb33be9-e451-42ba-a7f9-c9a4ebbbf169\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-19db3d5f-55c59f9b9d-h4zbn" Oct 30 21:05:40.565549 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:05:40.565520 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:05:40.683713 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:05:40.683621 5378 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-781058-openshift-default-55fb495b7-hpgv5"] Oct 30 21:05:40.709856 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:05:40.707743 5378 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-781058-openshift-default-55fb495b7-hpgv5"] Oct 30 21:05:40.729127 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:05:40.729093 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:05:40.781508 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:05:40.781478 5378 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-781058-openshift-default-55fb495b7-khvpl"] Oct 30 21:05:40.792059 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:05:40.792008 5378 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-781058-openshift-default-55fb495b7-khvpl"] Oct 30 21:05:40.914074 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:05:40.914044 5378 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-781058-openshift-default-55fb495b7-5zx7k"] Oct 30 21:05:40.918854 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:05:40.918833 5378 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-781058-openshift-default-55fb495b7-5zx7k"] Oct 30 21:05:40.992219 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:05:40.992160 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:05:40.999701 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:05:40.999680 5378 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6bb33be9-e451-42ba-a7f9-c9a4ebbbf169-maas-api-tls\") pod \"maas-api-e2e-mt-19db3d5f-55c59f9b9d-h4zbn\" (UID: \"6bb33be9-e451-42ba-a7f9-c9a4ebbbf169\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-19db3d5f-55c59f9b9d-h4zbn" Oct 30 21:05:41.002098 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:05:41.002078 5378 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6bb33be9-e451-42ba-a7f9-c9a4ebbbf169-maas-api-tls\") pod \"maas-api-e2e-mt-19db3d5f-55c59f9b9d-h4zbn\" (UID: \"6bb33be9-e451-42ba-a7f9-c9a4ebbbf169\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-19db3d5f-55c59f9b9d-h4zbn" Oct 30 21:05:41.022361 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:05:41.022340 5378 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-781058-openshift-default-55fb495b7-2c7gl"] Oct 30 21:05:41.025699 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:05:41.025677 5378 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-781058-openshift-default-55fb495b7-2c7gl"] Oct 30 21:05:41.244615 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:05:41.244522 5378 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-19db3d5f-55c59f9b9d-h4zbn" Oct 30 21:05:41.623676 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:05:41.623648 5378 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-19db3d5f-55c59f9b9d-h4zbn"] Oct 30 21:05:41.624105 ip-10-0-138-149 kubenswrapper[5378]: W1030 21:05:41.624080 5378 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6bb33be9_e451_42ba_a7f9_c9a4ebbbf169.slice/crio-3f9928697fc5c4fa17c4212524033d5c380408716294236a138d377d18f4b695 WatchSource:0}: Error finding container 3f9928697fc5c4fa17c4212524033d5c380408716294236a138d377d18f4b695: Status 404 returned error can't find the container with id 3f9928697fc5c4fa17c4212524033d5c380408716294236a138d377d18f4b695 Oct 30 21:05:41.745276 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:05:41.745239 5378 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="48a0901f-8b2b-4278-8422-226bb6bc6dd3" path="/var/lib/kubelet/pods/48a0901f-8b2b-4278-8422-226bb6bc6dd3/volumes" Oct 30 21:05:41.745786 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:05:41.745771 5378 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="507cd7ee-c139-489c-a520-292d8ebd91e4" path="/var/lib/kubelet/pods/507cd7ee-c139-489c-a520-292d8ebd91e4/volumes" Oct 30 21:05:41.746240 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:05:41.746226 5378 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="afcb5db1-70aa-4758-90e6-e21b112d48f4" path="/var/lib/kubelet/pods/afcb5db1-70aa-4758-90e6-e21b112d48f4/volumes" Oct 30 21:05:41.747026 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:05:41.747009 5378 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d43ee748-feb2-4556-be54-3662f8dfa98d" path="/var/lib/kubelet/pods/d43ee748-feb2-4556-be54-3662f8dfa98d/volumes" Oct 30 21:05:41.779442 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:05:41.779414 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:05:41.940138 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:05:41.940067 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:05:42.070975 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:05:42.070942 5378 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-derive-bf9c0727-openshift-default-7fb479dd89-lrsjm"] Oct 30 21:05:42.072202 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:05:42.072176 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:05:42.075109 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:05:42.075091 5378 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-bf9c0727-openshift-default-7fb479dd89-lrsjm" Oct 30 21:05:42.115944 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:05:42.115908 5378 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/88bbd407-197c-4ba6-b31f-fc097ec5fbf3-istio-token\") pod \"e2e-derive-bf9c0727-openshift-default-7fb479dd89-lrsjm\" (UID: \"88bbd407-197c-4ba6-b31f-fc097ec5fbf3\") " pod="openshift-ingress/e2e-derive-bf9c0727-openshift-default-7fb479dd89-lrsjm" Oct 30 21:05:42.116060 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:05:42.115949 5378 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/88bbd407-197c-4ba6-b31f-fc097ec5fbf3-istiod-ca-cert\") pod \"e2e-derive-bf9c0727-openshift-default-7fb479dd89-lrsjm\" (UID: \"88bbd407-197c-4ba6-b31f-fc097ec5fbf3\") " pod="openshift-ingress/e2e-derive-bf9c0727-openshift-default-7fb479dd89-lrsjm" Oct 30 21:05:42.116188 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:05:42.116168 5378 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/88bbd407-197c-4ba6-b31f-fc097ec5fbf3-istio-envoy\") pod \"e2e-derive-bf9c0727-openshift-default-7fb479dd89-lrsjm\" (UID: \"88bbd407-197c-4ba6-b31f-fc097ec5fbf3\") " pod="openshift-ingress/e2e-derive-bf9c0727-openshift-default-7fb479dd89-lrsjm" Oct 30 21:05:42.116487 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:05:42.116312 5378 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6sfdj\" (UniqueName: \"kubernetes.io/projected/88bbd407-197c-4ba6-b31f-fc097ec5fbf3-kube-api-access-6sfdj\") pod \"e2e-derive-bf9c0727-openshift-default-7fb479dd89-lrsjm\" (UID: \"88bbd407-197c-4ba6-b31f-fc097ec5fbf3\") " pod="openshift-ingress/e2e-derive-bf9c0727-openshift-default-7fb479dd89-lrsjm" Oct 30 21:05:42.116487 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:05:42.116373 5378 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/88bbd407-197c-4ba6-b31f-fc097ec5fbf3-workload-certs\") pod \"e2e-derive-bf9c0727-openshift-default-7fb479dd89-lrsjm\" (UID: \"88bbd407-197c-4ba6-b31f-fc097ec5fbf3\") " pod="openshift-ingress/e2e-derive-bf9c0727-openshift-default-7fb479dd89-lrsjm" Oct 30 21:05:42.116487 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:05:42.116425 5378 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/88bbd407-197c-4ba6-b31f-fc097ec5fbf3-credential-socket\") pod \"e2e-derive-bf9c0727-openshift-default-7fb479dd89-lrsjm\" (UID: \"88bbd407-197c-4ba6-b31f-fc097ec5fbf3\") " pod="openshift-ingress/e2e-derive-bf9c0727-openshift-default-7fb479dd89-lrsjm" Oct 30 21:05:42.116705 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:05:42.116515 5378 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/88bbd407-197c-4ba6-b31f-fc097ec5fbf3-istio-data\") pod \"e2e-derive-bf9c0727-openshift-default-7fb479dd89-lrsjm\" (UID: \"88bbd407-197c-4ba6-b31f-fc097ec5fbf3\") " pod="openshift-ingress/e2e-derive-bf9c0727-openshift-default-7fb479dd89-lrsjm" Oct 30 21:05:42.116705 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:05:42.116561 5378 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/88bbd407-197c-4ba6-b31f-fc097ec5fbf3-workload-socket\") pod \"e2e-derive-bf9c0727-openshift-default-7fb479dd89-lrsjm\" (UID: \"88bbd407-197c-4ba6-b31f-fc097ec5fbf3\") " pod="openshift-ingress/e2e-derive-bf9c0727-openshift-default-7fb479dd89-lrsjm" Oct 30 21:05:42.116705 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:05:42.116625 5378 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/88bbd407-197c-4ba6-b31f-fc097ec5fbf3-istio-podinfo\") pod \"e2e-derive-bf9c0727-openshift-default-7fb479dd89-lrsjm\" (UID: \"88bbd407-197c-4ba6-b31f-fc097ec5fbf3\") " pod="openshift-ingress/e2e-derive-bf9c0727-openshift-default-7fb479dd89-lrsjm" Oct 30 21:05:42.186181 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:05:42.186150 5378 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-derive-bf9c0727-openshift-default-7fb479dd89-lrsjm"] Oct 30 21:05:42.218422 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:05:42.218347 5378 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/88bbd407-197c-4ba6-b31f-fc097ec5fbf3-istio-token\") pod \"e2e-derive-bf9c0727-openshift-default-7fb479dd89-lrsjm\" (UID: \"88bbd407-197c-4ba6-b31f-fc097ec5fbf3\") " pod="openshift-ingress/e2e-derive-bf9c0727-openshift-default-7fb479dd89-lrsjm" Oct 30 21:05:42.218422 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:05:42.218387 5378 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/88bbd407-197c-4ba6-b31f-fc097ec5fbf3-istiod-ca-cert\") pod \"e2e-derive-bf9c0727-openshift-default-7fb479dd89-lrsjm\" (UID: \"88bbd407-197c-4ba6-b31f-fc097ec5fbf3\") " pod="openshift-ingress/e2e-derive-bf9c0727-openshift-default-7fb479dd89-lrsjm" Oct 30 21:05:42.218674 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:05:42.218492 5378 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/88bbd407-197c-4ba6-b31f-fc097ec5fbf3-istio-envoy\") pod \"e2e-derive-bf9c0727-openshift-default-7fb479dd89-lrsjm\" (UID: \"88bbd407-197c-4ba6-b31f-fc097ec5fbf3\") " pod="openshift-ingress/e2e-derive-bf9c0727-openshift-default-7fb479dd89-lrsjm" Oct 30 21:05:42.218674 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:05:42.218596 5378 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6sfdj\" (UniqueName: \"kubernetes.io/projected/88bbd407-197c-4ba6-b31f-fc097ec5fbf3-kube-api-access-6sfdj\") pod \"e2e-derive-bf9c0727-openshift-default-7fb479dd89-lrsjm\" (UID: \"88bbd407-197c-4ba6-b31f-fc097ec5fbf3\") " pod="openshift-ingress/e2e-derive-bf9c0727-openshift-default-7fb479dd89-lrsjm" Oct 30 21:05:42.218674 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:05:42.218651 5378 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/88bbd407-197c-4ba6-b31f-fc097ec5fbf3-workload-certs\") pod \"e2e-derive-bf9c0727-openshift-default-7fb479dd89-lrsjm\" (UID: \"88bbd407-197c-4ba6-b31f-fc097ec5fbf3\") " pod="openshift-ingress/e2e-derive-bf9c0727-openshift-default-7fb479dd89-lrsjm" Oct 30 21:05:42.218840 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:05:42.218698 5378 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/88bbd407-197c-4ba6-b31f-fc097ec5fbf3-credential-socket\") pod \"e2e-derive-bf9c0727-openshift-default-7fb479dd89-lrsjm\" (UID: \"88bbd407-197c-4ba6-b31f-fc097ec5fbf3\") " pod="openshift-ingress/e2e-derive-bf9c0727-openshift-default-7fb479dd89-lrsjm" Oct 30 21:05:42.218840 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:05:42.218738 5378 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/88bbd407-197c-4ba6-b31f-fc097ec5fbf3-istio-data\") pod \"e2e-derive-bf9c0727-openshift-default-7fb479dd89-lrsjm\" (UID: \"88bbd407-197c-4ba6-b31f-fc097ec5fbf3\") " pod="openshift-ingress/e2e-derive-bf9c0727-openshift-default-7fb479dd89-lrsjm" Oct 30 21:05:42.218840 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:05:42.218774 5378 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/88bbd407-197c-4ba6-b31f-fc097ec5fbf3-workload-socket\") pod \"e2e-derive-bf9c0727-openshift-default-7fb479dd89-lrsjm\" (UID: \"88bbd407-197c-4ba6-b31f-fc097ec5fbf3\") " pod="openshift-ingress/e2e-derive-bf9c0727-openshift-default-7fb479dd89-lrsjm" Oct 30 21:05:42.218840 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:05:42.218823 5378 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/88bbd407-197c-4ba6-b31f-fc097ec5fbf3-istio-podinfo\") pod \"e2e-derive-bf9c0727-openshift-default-7fb479dd89-lrsjm\" (UID: \"88bbd407-197c-4ba6-b31f-fc097ec5fbf3\") " pod="openshift-ingress/e2e-derive-bf9c0727-openshift-default-7fb479dd89-lrsjm" Oct 30 21:05:42.219108 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:05:42.219080 5378 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/88bbd407-197c-4ba6-b31f-fc097ec5fbf3-workload-certs\") pod \"e2e-derive-bf9c0727-openshift-default-7fb479dd89-lrsjm\" (UID: \"88bbd407-197c-4ba6-b31f-fc097ec5fbf3\") " pod="openshift-ingress/e2e-derive-bf9c0727-openshift-default-7fb479dd89-lrsjm" Oct 30 21:05:42.219229 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:05:42.219201 5378 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/88bbd407-197c-4ba6-b31f-fc097ec5fbf3-workload-socket\") pod \"e2e-derive-bf9c0727-openshift-default-7fb479dd89-lrsjm\" (UID: \"88bbd407-197c-4ba6-b31f-fc097ec5fbf3\") " pod="openshift-ingress/e2e-derive-bf9c0727-openshift-default-7fb479dd89-lrsjm" Oct 30 21:05:42.219295 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:05:42.219231 5378 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/88bbd407-197c-4ba6-b31f-fc097ec5fbf3-istiod-ca-cert\") pod \"e2e-derive-bf9c0727-openshift-default-7fb479dd89-lrsjm\" (UID: \"88bbd407-197c-4ba6-b31f-fc097ec5fbf3\") " pod="openshift-ingress/e2e-derive-bf9c0727-openshift-default-7fb479dd89-lrsjm" Oct 30 21:05:42.219295 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:05:42.219212 5378 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/88bbd407-197c-4ba6-b31f-fc097ec5fbf3-istio-data\") pod \"e2e-derive-bf9c0727-openshift-default-7fb479dd89-lrsjm\" (UID: \"88bbd407-197c-4ba6-b31f-fc097ec5fbf3\") " pod="openshift-ingress/e2e-derive-bf9c0727-openshift-default-7fb479dd89-lrsjm" Oct 30 21:05:42.219392 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:05:42.219374 5378 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/88bbd407-197c-4ba6-b31f-fc097ec5fbf3-credential-socket\") pod \"e2e-derive-bf9c0727-openshift-default-7fb479dd89-lrsjm\" (UID: \"88bbd407-197c-4ba6-b31f-fc097ec5fbf3\") " pod="openshift-ingress/e2e-derive-bf9c0727-openshift-default-7fb479dd89-lrsjm" Oct 30 21:05:42.220936 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:05:42.220911 5378 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/88bbd407-197c-4ba6-b31f-fc097ec5fbf3-istio-envoy\") pod \"e2e-derive-bf9c0727-openshift-default-7fb479dd89-lrsjm\" (UID: \"88bbd407-197c-4ba6-b31f-fc097ec5fbf3\") " pod="openshift-ingress/e2e-derive-bf9c0727-openshift-default-7fb479dd89-lrsjm" Oct 30 21:05:42.221247 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:05:42.221227 5378 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/88bbd407-197c-4ba6-b31f-fc097ec5fbf3-istio-podinfo\") pod \"e2e-derive-bf9c0727-openshift-default-7fb479dd89-lrsjm\" (UID: \"88bbd407-197c-4ba6-b31f-fc097ec5fbf3\") " pod="openshift-ingress/e2e-derive-bf9c0727-openshift-default-7fb479dd89-lrsjm" Oct 30 21:05:42.276139 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:05:42.276101 5378 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/88bbd407-197c-4ba6-b31f-fc097ec5fbf3-istio-token\") pod \"e2e-derive-bf9c0727-openshift-default-7fb479dd89-lrsjm\" (UID: \"88bbd407-197c-4ba6-b31f-fc097ec5fbf3\") " pod="openshift-ingress/e2e-derive-bf9c0727-openshift-default-7fb479dd89-lrsjm" Oct 30 21:05:42.276379 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:05:42.276354 5378 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6sfdj\" (UniqueName: \"kubernetes.io/projected/88bbd407-197c-4ba6-b31f-fc097ec5fbf3-kube-api-access-6sfdj\") pod \"e2e-derive-bf9c0727-openshift-default-7fb479dd89-lrsjm\" (UID: \"88bbd407-197c-4ba6-b31f-fc097ec5fbf3\") " pod="openshift-ingress/e2e-derive-bf9c0727-openshift-default-7fb479dd89-lrsjm" Oct 30 21:05:42.388747 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:05:42.388716 5378 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-bf9c0727-openshift-default-7fb479dd89-lrsjm" Oct 30 21:05:42.498546 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:05:42.498508 5378 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-19db3d5f-55c59f9b9d-h4zbn" event={"ID":"6bb33be9-e451-42ba-a7f9-c9a4ebbbf169","Type":"ContainerStarted","Data":"3f9928697fc5c4fa17c4212524033d5c380408716294236a138d377d18f4b695"} Oct 30 21:05:42.827532 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:05:42.827452 5378 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-derive-bf9c0727-openshift-default-7fb479dd89-lrsjm"] Oct 30 21:05:42.830630 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:05:42.830593 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:05:42.866121 ip-10-0-138-149 kubenswrapper[5378]: W1030 21:05:42.866091 5378 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod88bbd407_197c_4ba6_b31f_fc097ec5fbf3.slice/crio-0199052dac8b0bd2cbb89c06de4f16cfae80a146fed6046f5f824a58ab2a37c8 WatchSource:0}: Error finding container 0199052dac8b0bd2cbb89c06de4f16cfae80a146fed6046f5f824a58ab2a37c8: Status 404 returned error can't find the container with id 0199052dac8b0bd2cbb89c06de4f16cfae80a146fed6046f5f824a58ab2a37c8 Oct 30 21:05:42.869345 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:05:42.869313 5378 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 21:05:42.869453 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:05:42.869379 5378 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 21:05:42.869453 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:05:42.869410 5378 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 21:05:43.442211 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:05:43.442180 5378 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-de53c2-697d9756c6-t4xvj" Oct 30 21:05:43.507505 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:05:43.507472 5378 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-bf9c0727-openshift-default-7fb479dd89-lrsjm" event={"ID":"88bbd407-197c-4ba6-b31f-fc097ec5fbf3","Type":"ContainerStarted","Data":"434ca2e3b8666ac602b32ac4097c0bb6d841dcdeac21e53e8c262b96fa49f606"} Oct 30 21:05:43.507732 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:05:43.507535 5378 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-bf9c0727-openshift-default-7fb479dd89-lrsjm" event={"ID":"88bbd407-197c-4ba6-b31f-fc097ec5fbf3","Type":"ContainerStarted","Data":"0199052dac8b0bd2cbb89c06de4f16cfae80a146fed6046f5f824a58ab2a37c8"} Oct 30 21:05:43.509211 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:05:43.509182 5378 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-19db3d5f-55c59f9b9d-h4zbn" event={"ID":"6bb33be9-e451-42ba-a7f9-c9a4ebbbf169","Type":"ContainerStarted","Data":"d3b98d5bee535121103d0993e8805a8d43afaaed8d2cd2ccab6ddd54f009c5ec"} Oct 30 21:05:43.509337 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:05:43.509308 5378 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-19db3d5f-55c59f9b9d-h4zbn" Oct 30 21:05:43.704869 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:05:43.704763 5378 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-derive-bf9c0727-openshift-default-7fb479dd89-lrsjm" podStartSLOduration=1.7047454480000002 podStartE2EDuration="1.704745448s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:05:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:05:43.701041006 +0000 UTC m=+1634.129154926" watchObservedRunningTime="2026-10-30 21:05:43.704745448 +0000 UTC m=+1634.132859431" Oct 30 21:05:43.878588 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:05:43.873616 5378 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-mt-19db3d5f-55c59f9b9d-h4zbn" podStartSLOduration=3.051246481 podStartE2EDuration="3.87359464s" totalImagesPullingTime="822.348159ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:05:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:05:43.871203463 +0000 UTC m=+1634.299317382" watchObservedRunningTime="2026-10-30 21:05:43.87359464 +0000 UTC m=+1634.301708559" Oct 30 21:05:44.389345 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:05:44.389310 5378 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-derive-bf9c0727-openshift-default-7fb479dd89-lrsjm" Oct 30 21:05:44.391275 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:05:44.391242 5378 patch_prober.go:29] interesting pod/e2e-derive-bf9c0727-openshift-default-7fb479dd89-lrsjm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.120:15021/healthz/ready\": dial tcp 10.133.0.120:15021: connect: connection refused" start-of-body= Oct 30 21:05:44.391419 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:05:44.391310 5378 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-bf9c0727-openshift-default-7fb479dd89-lrsjm" podUID="88bbd407-197c-4ba6-b31f-fc097ec5fbf3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.120:15021/healthz/ready\": dial tcp 10.133.0.120:15021: connect: connection refused" Oct 30 21:05:45.277591 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:05:45.275173 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:05:45.389836 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:05:45.389799 5378 patch_prober.go:29] interesting pod/e2e-derive-bf9c0727-openshift-default-7fb479dd89-lrsjm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.120:15021/healthz/ready\": dial tcp 10.133.0.120:15021: connect: connection refused" start-of-body= Oct 30 21:05:45.390023 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:05:45.389874 5378 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-bf9c0727-openshift-default-7fb479dd89-lrsjm" podUID="88bbd407-197c-4ba6-b31f-fc097ec5fbf3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.120:15021/healthz/ready\": dial tcp 10.133.0.120:15021: connect: connection refused" Oct 30 21:05:46.179662 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:05:46.179614 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:05:46.394010 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:05:46.393975 5378 patch_prober.go:29] interesting pod/e2e-derive-bf9c0727-openshift-default-7fb479dd89-lrsjm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.120:15021/healthz/ready\": dial tcp 10.133.0.120:15021: connect: connection refused" start-of-body= Oct 30 21:05:46.394646 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:05:46.394605 5378 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-bf9c0727-openshift-default-7fb479dd89-lrsjm" podUID="88bbd407-197c-4ba6-b31f-fc097ec5fbf3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.120:15021/healthz/ready\": dial tcp 10.133.0.120:15021: connect: connection refused" Oct 30 21:05:47.389128 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:05:47.389095 5378 patch_prober.go:29] interesting pod/e2e-derive-bf9c0727-openshift-default-7fb479dd89-lrsjm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.120:15021/healthz/ready\": dial tcp 10.133.0.120:15021: connect: connection refused" start-of-body= Oct 30 21:05:47.389311 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:05:47.389150 5378 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-bf9c0727-openshift-default-7fb479dd89-lrsjm" podUID="88bbd407-197c-4ba6-b31f-fc097ec5fbf3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.120:15021/healthz/ready\": dial tcp 10.133.0.120:15021: connect: connection refused" Oct 30 21:05:48.389548 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:05:48.389514 5378 patch_prober.go:29] interesting pod/e2e-derive-bf9c0727-openshift-default-7fb479dd89-lrsjm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.120:15021/healthz/ready\": dial tcp 10.133.0.120:15021: connect: connection refused" start-of-body= Oct 30 21:05:48.389960 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:05:48.389603 5378 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-bf9c0727-openshift-default-7fb479dd89-lrsjm" podUID="88bbd407-197c-4ba6-b31f-fc097ec5fbf3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.120:15021/healthz/ready\": dial tcp 10.133.0.120:15021: connect: connection refused" Oct 30 21:05:48.581469 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:05:48.581433 5378 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-bf9c0727-55f74857d7-m98lb"] Oct 30 21:05:48.810687 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:05:48.808739 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:05:48.844504 ip-10-0-138-149 kubenswrapper[5378]: E1030 21:05:48.844469 5378 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q6mx7" containerName="extractor" Oct 30 21:05:49.022242 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:05:49.022146 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:05:49.395780 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:05:49.395750 5378 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-derive-bf9c0727-openshift-default-7fb479dd89-lrsjm" Oct 30 21:05:49.396249 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:05:49.396223 5378 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-derive-bf9c0727-openshift-default-7fb479dd89-lrsjm" Oct 30 21:05:49.397084 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:05:49.397059 5378 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-derive-bf9c0727-openshift-default-7fb479dd89-lrsjm" Oct 30 21:05:49.522508 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:05:49.522481 5378 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-19db3d5f-55c59f9b9d-h4zbn" Oct 30 21:05:51.485429 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:05:51.485396 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:05:53.048677 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:05:53.048639 5378 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-23c672dc-5cccz"] Oct 30 21:05:53.099215 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:05:53.099187 5378 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-23c672dc-5cccz"] Oct 30 21:05:53.582048 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:05:53.581996 5378 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-derive-bf9c0727-55f74857d7-m98lb" podUID="9723840f-d122-4d53-8600-346cd5f2db0d" containerName="payload-pre-processing" containerID="cri-o://37a55991a0ab07211368ac400f64601b9d244b9b8b202b41e2c37d3ee08d72e8" gracePeriod=25 Oct 30 21:05:53.744353 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:05:53.744316 5378 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d5c2205b-0665-490d-83c9-203f04e51d93" path="/var/lib/kubelet/pods/d5c2205b-0665-490d-83c9-203f04e51d93/volumes" Oct 30 21:05:53.871210 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:05:53.871186 5378 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-bf9c0727-55f74857d7-m98lb" Oct 30 21:05:53.965317 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:05:53.965282 5378 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9723840f-d122-4d53-8600-346cd5f2db0d-kube-api-access-jvvnf\" (UniqueName: \"kubernetes.io/projected/9723840f-d122-4d53-8600-346cd5f2db0d-kube-api-access-jvvnf\") pod \"9723840f-d122-4d53-8600-346cd5f2db0d\" (UID: \"9723840f-d122-4d53-8600-346cd5f2db0d\") " Oct 30 21:05:53.965513 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:05:53.965393 5378 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9723840f-d122-4d53-8600-346cd5f2db0d-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9723840f-d122-4d53-8600-346cd5f2db0d-plugins-config-volume\") pod \"9723840f-d122-4d53-8600-346cd5f2db0d\" (UID: \"9723840f-d122-4d53-8600-346cd5f2db0d\") " Oct 30 21:05:53.965789 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:05:53.965764 5378 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9723840f-d122-4d53-8600-346cd5f2db0d-plugins-config-volume" pod "9723840f-d122-4d53-8600-346cd5f2db0d" (UID: "9723840f-d122-4d53-8600-346cd5f2db0d"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:05:53.966010 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:05:53.965995 5378 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9723840f-d122-4d53-8600-346cd5f2db0d-plugins-config-volume\") on node \"ip-10-0-138-149.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:05:53.967450 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:05:53.967418 5378 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9723840f-d122-4d53-8600-346cd5f2db0d-kube-api-access-jvvnf" pod "9723840f-d122-4d53-8600-346cd5f2db0d" (UID: "9723840f-d122-4d53-8600-346cd5f2db0d"). InnerVolumeSpecName "kube-api-access-jvvnf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:05:54.067028 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:05:54.066998 5378 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jvvnf\" (UniqueName: \"kubernetes.io/projected/9723840f-d122-4d53-8600-346cd5f2db0d-kube-api-access-jvvnf\") on node \"ip-10-0-138-149.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:05:54.632675 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:05:54.632640 5378 generic.go:356] "Generic (PLEG): container finished" podID="9723840f-d122-4d53-8600-346cd5f2db0d" containerID="37a55991a0ab07211368ac400f64601b9d244b9b8b202b41e2c37d3ee08d72e8" exitCode=0 Oct 30 21:05:54.632887 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:05:54.632692 5378 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-bf9c0727-55f74857d7-m98lb" Oct 30 21:05:54.632887 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:05:54.632700 5378 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-bf9c0727-55f74857d7-m98lb" event={"ID":"9723840f-d122-4d53-8600-346cd5f2db0d","Type":"ContainerDied","Data":"37a55991a0ab07211368ac400f64601b9d244b9b8b202b41e2c37d3ee08d72e8"} Oct 30 21:05:54.632887 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:05:54.632751 5378 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-bf9c0727-55f74857d7-m98lb" event={"ID":"9723840f-d122-4d53-8600-346cd5f2db0d","Type":"ContainerDied","Data":"456af6ca162cff6c24cd8b7a0485325802b5a9739e04414b0080c65fe15ab06f"} Oct 30 21:05:54.632887 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:05:54.632779 5378 scope.go:122] "RemoveContainer" containerID="37a55991a0ab07211368ac400f64601b9d244b9b8b202b41e2c37d3ee08d72e8" Oct 30 21:05:54.643556 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:05:54.643538 5378 scope.go:122] "RemoveContainer" containerID="37a55991a0ab07211368ac400f64601b9d244b9b8b202b41e2c37d3ee08d72e8" Oct 30 21:05:54.643917 ip-10-0-138-149 kubenswrapper[5378]: E1030 21:05:54.643899 5378 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"37a55991a0ab07211368ac400f64601b9d244b9b8b202b41e2c37d3ee08d72e8\": container with ID starting with 37a55991a0ab07211368ac400f64601b9d244b9b8b202b41e2c37d3ee08d72e8 not found: ID does not exist" containerID="37a55991a0ab07211368ac400f64601b9d244b9b8b202b41e2c37d3ee08d72e8" Oct 30 21:05:54.643958 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:05:54.643927 5378 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"37a55991a0ab07211368ac400f64601b9d244b9b8b202b41e2c37d3ee08d72e8"} err="failed to get container status \"37a55991a0ab07211368ac400f64601b9d244b9b8b202b41e2c37d3ee08d72e8\": rpc error: code = NotFound desc = could not find container \"37a55991a0ab07211368ac400f64601b9d244b9b8b202b41e2c37d3ee08d72e8\": container with ID starting with 37a55991a0ab07211368ac400f64601b9d244b9b8b202b41e2c37d3ee08d72e8 not found: ID does not exist" Oct 30 21:05:54.799876 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:05:54.799842 5378 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-bf9c0727-55f74857d7-m98lb"] Oct 30 21:05:54.850451 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:05:54.850422 5378 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-bf9c0727-55f74857d7-m98lb"] Oct 30 21:05:55.742884 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:05:55.742852 5378 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9723840f-d122-4d53-8600-346cd5f2db0d" path="/var/lib/kubelet/pods/9723840f-d122-4d53-8600-346cd5f2db0d/volumes" Oct 30 21:05:56.253803 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:05:56.253769 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:05:56.379579 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:05:56.379528 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:05:56.625796 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:05:56.625768 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:06:01.568905 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:06:01.568869 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:06:04.047240 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:06:04.047207 5378 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-eac736c7-24cr2"] Oct 30 21:06:04.097987 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:06:04.097958 5378 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-eac736c7-24cr2"] Oct 30 21:06:04.367614 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:06:04.363322 5378 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-2b425d-openshift-default-f547dbcbf-wllbd"] Oct 30 21:06:04.367614 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:06:04.363613 5378 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w5-2b425d-openshift-default-f547dbcbf-wllbd" podUID="ce8609a7-1869-4d3e-b1d7-adab0908fbb9" containerName="istio-proxy" containerID="cri-o://9e0befd66939484f1b072e143f08f7c1abdabd630a986edd3396fb5315945ddb" gracePeriod=30 Oct 30 21:06:04.388166 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:06:04.388131 5378 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-2b425d-openshift-default-f547dbcbf-j99m9"] Oct 30 21:06:04.388748 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:06:04.388687 5378 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w5-2b425d-openshift-default-f547dbcbf-j99m9" podUID="ad099f6e-3035-4a6d-8fef-aea2e4bd1f3b" containerName="istio-proxy" containerID="cri-o://3119e4b6f8ac9712c0e43d657660719bcf26b0d85397424196cf757a1ad2f077" gracePeriod=30 Oct 30 21:06:04.592589 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:06:04.577834 5378 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-682999-openshift-default-54dcb797b5-r27vc"] Oct 30 21:06:04.592589 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:06:04.578191 5378 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-682999-openshift-default-54dcb797b5-r27vc" podUID="feae394a-bbbf-4624-8057-cf046f9ffb93" containerName="istio-proxy" containerID="cri-o://92f037c57c81aec3aac423afef2a5de0ad66cac77aad6a892e410e96d93b5543" gracePeriod=30 Oct 30 21:06:04.784676 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:06:04.782609 5378 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-682999-openshift-default-54dcb797b5-xhh97"] Oct 30 21:06:04.784676 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:06:04.782692 5378 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-682999-openshift-default-54dcb797b5-zhslb"] Oct 30 21:06:04.785882 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:06:04.785844 5378 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-682999-openshift-default-54dcb797b5-xhh97" podUID="95e52f08-4390-4924-89e1-c520ced3743f" containerName="istio-proxy" containerID="cri-o://4820660cffc38c84c6a5bb8f793ea741daa1d57b1c65c2e8ae8dcc6211617a48" gracePeriod=30 Oct 30 21:06:04.807951 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:06:04.796726 5378 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-682999-openshift-default-54dcb797b5-zhslb" podUID="afdb629d-0850-4d80-a6b9-3556bd0d706e" containerName="istio-proxy" containerID="cri-o://1b8acf4813b04532a32225114288d63035b23d2beca277eb36e69cec3a33c818" gracePeriod=30 Oct 30 21:06:04.807951 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:06:04.807913 5378 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-r9g59"] Oct 30 21:06:05.767782 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:06:05.767659 5378 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="10b2d79a-d60b-4624-bf6d-6ac75c235de0" path="/var/lib/kubelet/pods/10b2d79a-d60b-4624-bf6d-6ac75c235de0/volumes" Oct 30 21:06:06.869313 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:06:06.869223 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:06:07.938834 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:06:07.938803 5378 patch_prober.go:29] interesting pod/e2e-worker-w3-682999-openshift-default-54dcb797b5-xhh97 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 21:06:07.939257 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:06:07.939233 5378 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-682999-openshift-default-54dcb797b5-xhh97" podUID="95e52f08-4390-4924-89e1-c520ced3743f" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 21:06:08.895591 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:06:08.894714 5378 patch_prober.go:29] interesting pod/e2e-worker-w3-682999-openshift-default-54dcb797b5-zhslb container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 21:06:08.895591 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:06:08.894777 5378 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-682999-openshift-default-54dcb797b5-zhslb" podUID="afdb629d-0850-4d80-a6b9-3556bd0d706e" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 21:06:09.250390 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:06:09.250297 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:06:09.946269 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:06:09.939792 5378 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-2b425d-openshift-default-f547dbcbf-wllbd" Oct 30 21:06:10.013038 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:06:10.012479 5378 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ce8609a7-1869-4d3e-b1d7-adab0908fbb9-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ce8609a7-1869-4d3e-b1d7-adab0908fbb9-workload-certs\") pod \"ce8609a7-1869-4d3e-b1d7-adab0908fbb9\" (UID: \"ce8609a7-1869-4d3e-b1d7-adab0908fbb9\") " Oct 30 21:06:10.013038 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:06:10.012559 5378 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ce8609a7-1869-4d3e-b1d7-adab0908fbb9-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ce8609a7-1869-4d3e-b1d7-adab0908fbb9-istio-data\") pod \"ce8609a7-1869-4d3e-b1d7-adab0908fbb9\" (UID: \"ce8609a7-1869-4d3e-b1d7-adab0908fbb9\") " Oct 30 21:06:10.013038 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:06:10.012647 5378 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/ce8609a7-1869-4d3e-b1d7-adab0908fbb9-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ce8609a7-1869-4d3e-b1d7-adab0908fbb9-istio-podinfo\") pod \"ce8609a7-1869-4d3e-b1d7-adab0908fbb9\" (UID: \"ce8609a7-1869-4d3e-b1d7-adab0908fbb9\") " Oct 30 21:06:10.013038 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:06:10.012756 5378 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ce8609a7-1869-4d3e-b1d7-adab0908fbb9-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ce8609a7-1869-4d3e-b1d7-adab0908fbb9-istiod-ca-cert\") pod \"ce8609a7-1869-4d3e-b1d7-adab0908fbb9\" (UID: \"ce8609a7-1869-4d3e-b1d7-adab0908fbb9\") " Oct 30 21:06:10.013038 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:06:10.012827 5378 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ce8609a7-1869-4d3e-b1d7-adab0908fbb9-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ce8609a7-1869-4d3e-b1d7-adab0908fbb9-workload-socket\") pod \"ce8609a7-1869-4d3e-b1d7-adab0908fbb9\" (UID: \"ce8609a7-1869-4d3e-b1d7-adab0908fbb9\") " Oct 30 21:06:10.013038 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:06:10.012993 5378 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ce8609a7-1869-4d3e-b1d7-adab0908fbb9-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ce8609a7-1869-4d3e-b1d7-adab0908fbb9-istio-envoy\") pod \"ce8609a7-1869-4d3e-b1d7-adab0908fbb9\" (UID: \"ce8609a7-1869-4d3e-b1d7-adab0908fbb9\") " Oct 30 21:06:10.013512 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:06:10.013136 5378 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ce8609a7-1869-4d3e-b1d7-adab0908fbb9-istio-token\" (UniqueName: \"kubernetes.io/projected/ce8609a7-1869-4d3e-b1d7-adab0908fbb9-istio-token\") pod \"ce8609a7-1869-4d3e-b1d7-adab0908fbb9\" (UID: \"ce8609a7-1869-4d3e-b1d7-adab0908fbb9\") " Oct 30 21:06:10.013512 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:06:10.013173 5378 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ce8609a7-1869-4d3e-b1d7-adab0908fbb9-kube-api-access-z8w2w\" (UniqueName: \"kubernetes.io/projected/ce8609a7-1869-4d3e-b1d7-adab0908fbb9-kube-api-access-z8w2w\") pod \"ce8609a7-1869-4d3e-b1d7-adab0908fbb9\" (UID: \"ce8609a7-1869-4d3e-b1d7-adab0908fbb9\") " Oct 30 21:06:10.013512 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:06:10.013256 5378 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ce8609a7-1869-4d3e-b1d7-adab0908fbb9-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ce8609a7-1869-4d3e-b1d7-adab0908fbb9-credential-socket\") pod \"ce8609a7-1869-4d3e-b1d7-adab0908fbb9\" (UID: \"ce8609a7-1869-4d3e-b1d7-adab0908fbb9\") " Oct 30 21:06:10.025627 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:06:10.014189 5378 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ce8609a7-1869-4d3e-b1d7-adab0908fbb9-credential-socket" pod "ce8609a7-1869-4d3e-b1d7-adab0908fbb9" (UID: "ce8609a7-1869-4d3e-b1d7-adab0908fbb9"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:06:10.025627 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:06:10.014434 5378 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ce8609a7-1869-4d3e-b1d7-adab0908fbb9-workload-certs" pod "ce8609a7-1869-4d3e-b1d7-adab0908fbb9" (UID: "ce8609a7-1869-4d3e-b1d7-adab0908fbb9"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:06:10.025627 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:06:10.014640 5378 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ce8609a7-1869-4d3e-b1d7-adab0908fbb9-istio-data" pod "ce8609a7-1869-4d3e-b1d7-adab0908fbb9" (UID: "ce8609a7-1869-4d3e-b1d7-adab0908fbb9"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:06:10.025627 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:06:10.019344 5378 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ce8609a7-1869-4d3e-b1d7-adab0908fbb9-istio-podinfo" pod "ce8609a7-1869-4d3e-b1d7-adab0908fbb9" (UID: "ce8609a7-1869-4d3e-b1d7-adab0908fbb9"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 21:06:10.025627 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:06:10.020187 5378 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ce8609a7-1869-4d3e-b1d7-adab0908fbb9-istiod-ca-cert" pod "ce8609a7-1869-4d3e-b1d7-adab0908fbb9" (UID: "ce8609a7-1869-4d3e-b1d7-adab0908fbb9"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:06:10.025627 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:06:10.020419 5378 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ce8609a7-1869-4d3e-b1d7-adab0908fbb9-workload-socket" pod "ce8609a7-1869-4d3e-b1d7-adab0908fbb9" (UID: "ce8609a7-1869-4d3e-b1d7-adab0908fbb9"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:06:10.025627 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:06:10.020473 5378 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ce8609a7-1869-4d3e-b1d7-adab0908fbb9-istio-envoy" pod "ce8609a7-1869-4d3e-b1d7-adab0908fbb9" (UID: "ce8609a7-1869-4d3e-b1d7-adab0908fbb9"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:06:10.025627 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:06:10.022953 5378 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ce8609a7-1869-4d3e-b1d7-adab0908fbb9-istio-token" pod "ce8609a7-1869-4d3e-b1d7-adab0908fbb9" (UID: "ce8609a7-1869-4d3e-b1d7-adab0908fbb9"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:06:10.025627 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:06:10.025047 5378 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ce8609a7-1869-4d3e-b1d7-adab0908fbb9-kube-api-access-z8w2w" pod "ce8609a7-1869-4d3e-b1d7-adab0908fbb9" (UID: "ce8609a7-1869-4d3e-b1d7-adab0908fbb9"). InnerVolumeSpecName "kube-api-access-z8w2w". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:06:10.122136 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:06:10.121966 5378 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ce8609a7-1869-4d3e-b1d7-adab0908fbb9-istio-token\") on node \"ip-10-0-138-149.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:06:10.122136 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:06:10.121996 5378 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z8w2w\" (UniqueName: \"kubernetes.io/projected/ce8609a7-1869-4d3e-b1d7-adab0908fbb9-kube-api-access-z8w2w\") on node \"ip-10-0-138-149.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:06:10.122136 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:06:10.122007 5378 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ce8609a7-1869-4d3e-b1d7-adab0908fbb9-credential-socket\") on node \"ip-10-0-138-149.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:06:10.122136 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:06:10.122018 5378 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ce8609a7-1869-4d3e-b1d7-adab0908fbb9-workload-certs\") on node \"ip-10-0-138-149.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:06:10.122136 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:06:10.122026 5378 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ce8609a7-1869-4d3e-b1d7-adab0908fbb9-istio-data\") on node \"ip-10-0-138-149.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:06:10.122136 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:06:10.122035 5378 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ce8609a7-1869-4d3e-b1d7-adab0908fbb9-istio-podinfo\") on node \"ip-10-0-138-149.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:06:10.122136 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:06:10.122044 5378 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ce8609a7-1869-4d3e-b1d7-adab0908fbb9-istiod-ca-cert\") on node \"ip-10-0-138-149.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:06:10.122136 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:06:10.122054 5378 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ce8609a7-1869-4d3e-b1d7-adab0908fbb9-workload-socket\") on node \"ip-10-0-138-149.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:06:10.122136 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:06:10.122062 5378 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ce8609a7-1869-4d3e-b1d7-adab0908fbb9-istio-envoy\") on node \"ip-10-0-138-149.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:06:10.124411 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:06:10.123935 5378 generic.go:356] "Generic (PLEG): container finished" podID="ad099f6e-3035-4a6d-8fef-aea2e4bd1f3b" containerID="3119e4b6f8ac9712c0e43d657660719bcf26b0d85397424196cf757a1ad2f077" exitCode=0 Oct 30 21:06:10.124411 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:06:10.124064 5378 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-2b425d-openshift-default-f547dbcbf-j99m9" event={"ID":"ad099f6e-3035-4a6d-8fef-aea2e4bd1f3b","Type":"ContainerDied","Data":"3119e4b6f8ac9712c0e43d657660719bcf26b0d85397424196cf757a1ad2f077"} Oct 30 21:06:10.126781 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:06:10.126216 5378 generic.go:356] "Generic (PLEG): container finished" podID="feae394a-bbbf-4624-8057-cf046f9ffb93" containerID="92f037c57c81aec3aac423afef2a5de0ad66cac77aad6a892e410e96d93b5543" exitCode=0 Oct 30 21:06:10.126781 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:06:10.126343 5378 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-682999-openshift-default-54dcb797b5-r27vc" event={"ID":"feae394a-bbbf-4624-8057-cf046f9ffb93","Type":"ContainerDied","Data":"92f037c57c81aec3aac423afef2a5de0ad66cac77aad6a892e410e96d93b5543"} Oct 30 21:06:10.129617 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:06:10.129445 5378 generic.go:356] "Generic (PLEG): container finished" podID="95e52f08-4390-4924-89e1-c520ced3743f" containerID="4820660cffc38c84c6a5bb8f793ea741daa1d57b1c65c2e8ae8dcc6211617a48" exitCode=0 Oct 30 21:06:10.129617 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:06:10.129535 5378 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-682999-openshift-default-54dcb797b5-xhh97" event={"ID":"95e52f08-4390-4924-89e1-c520ced3743f","Type":"ContainerDied","Data":"4820660cffc38c84c6a5bb8f793ea741daa1d57b1c65c2e8ae8dcc6211617a48"} Oct 30 21:06:10.132312 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:06:10.132177 5378 generic.go:356] "Generic (PLEG): container finished" podID="afdb629d-0850-4d80-a6b9-3556bd0d706e" containerID="1b8acf4813b04532a32225114288d63035b23d2beca277eb36e69cec3a33c818" exitCode=0 Oct 30 21:06:10.132312 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:06:10.132254 5378 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-682999-openshift-default-54dcb797b5-zhslb" event={"ID":"afdb629d-0850-4d80-a6b9-3556bd0d706e","Type":"ContainerDied","Data":"1b8acf4813b04532a32225114288d63035b23d2beca277eb36e69cec3a33c818"} Oct 30 21:06:10.137198 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:06:10.133534 5378 generic.go:356] "Generic (PLEG): container finished" podID="ce8609a7-1869-4d3e-b1d7-adab0908fbb9" containerID="9e0befd66939484f1b072e143f08f7c1abdabd630a986edd3396fb5315945ddb" exitCode=0 Oct 30 21:06:10.137198 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:06:10.133686 5378 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-2b425d-openshift-default-f547dbcbf-wllbd" event={"ID":"ce8609a7-1869-4d3e-b1d7-adab0908fbb9","Type":"ContainerDied","Data":"9e0befd66939484f1b072e143f08f7c1abdabd630a986edd3396fb5315945ddb"} Oct 30 21:06:10.137198 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:06:10.133713 5378 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-2b425d-openshift-default-f547dbcbf-wllbd" event={"ID":"ce8609a7-1869-4d3e-b1d7-adab0908fbb9","Type":"ContainerDied","Data":"825c240375b0ed6aed7fa079d19dcf7d31217064d1e8ec1e8e29af5f4f180d86"} Oct 30 21:06:10.137198 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:06:10.133733 5378 scope.go:122] "RemoveContainer" containerID="9e0befd66939484f1b072e143f08f7c1abdabd630a986edd3396fb5315945ddb" Oct 30 21:06:10.137198 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:06:10.133854 5378 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-2b425d-openshift-default-f547dbcbf-wllbd" Oct 30 21:06:10.152557 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:06:10.152534 5378 scope.go:122] "RemoveContainer" containerID="9e0befd66939484f1b072e143f08f7c1abdabd630a986edd3396fb5315945ddb" Oct 30 21:06:10.152557 ip-10-0-138-149 kubenswrapper[5378]: E1030 21:06:10.159806 5378 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9e0befd66939484f1b072e143f08f7c1abdabd630a986edd3396fb5315945ddb\": container with ID starting with 9e0befd66939484f1b072e143f08f7c1abdabd630a986edd3396fb5315945ddb not found: ID does not exist" containerID="9e0befd66939484f1b072e143f08f7c1abdabd630a986edd3396fb5315945ddb" Oct 30 21:06:10.152557 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:06:10.159855 5378 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9e0befd66939484f1b072e143f08f7c1abdabd630a986edd3396fb5315945ddb"} err="failed to get container status \"9e0befd66939484f1b072e143f08f7c1abdabd630a986edd3396fb5315945ddb\": rpc error: code = NotFound desc = could not find container \"9e0befd66939484f1b072e143f08f7c1abdabd630a986edd3396fb5315945ddb\": container with ID starting with 9e0befd66939484f1b072e143f08f7c1abdabd630a986edd3396fb5315945ddb not found: ID does not exist" Oct 30 21:06:10.303858 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:06:10.303744 5378 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-682999-openshift-default-54dcb797b5-r27vc" Oct 30 21:06:10.334217 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:06:10.332191 5378 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/feae394a-bbbf-4624-8057-cf046f9ffb93-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/feae394a-bbbf-4624-8057-cf046f9ffb93-istio-envoy\") pod \"feae394a-bbbf-4624-8057-cf046f9ffb93\" (UID: \"feae394a-bbbf-4624-8057-cf046f9ffb93\") " Oct 30 21:06:10.334217 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:06:10.332629 5378 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/feae394a-bbbf-4624-8057-cf046f9ffb93-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/feae394a-bbbf-4624-8057-cf046f9ffb93-workload-certs\") pod \"feae394a-bbbf-4624-8057-cf046f9ffb93\" (UID: \"feae394a-bbbf-4624-8057-cf046f9ffb93\") " Oct 30 21:06:10.334217 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:06:10.332834 5378 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/feae394a-bbbf-4624-8057-cf046f9ffb93-workload-certs" pod "feae394a-bbbf-4624-8057-cf046f9ffb93" (UID: "feae394a-bbbf-4624-8057-cf046f9ffb93"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:06:10.334217 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:06:10.332881 5378 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/feae394a-bbbf-4624-8057-cf046f9ffb93-istio-token\" (UniqueName: \"kubernetes.io/projected/feae394a-bbbf-4624-8057-cf046f9ffb93-istio-token\") pod \"feae394a-bbbf-4624-8057-cf046f9ffb93\" (UID: \"feae394a-bbbf-4624-8057-cf046f9ffb93\") " Oct 30 21:06:10.334217 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:06:10.333150 5378 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/feae394a-bbbf-4624-8057-cf046f9ffb93-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/feae394a-bbbf-4624-8057-cf046f9ffb93-workload-socket\") pod \"feae394a-bbbf-4624-8057-cf046f9ffb93\" (UID: \"feae394a-bbbf-4624-8057-cf046f9ffb93\") " Oct 30 21:06:10.334217 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:06:10.333205 5378 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/feae394a-bbbf-4624-8057-cf046f9ffb93-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/feae394a-bbbf-4624-8057-cf046f9ffb93-credential-socket\") pod \"feae394a-bbbf-4624-8057-cf046f9ffb93\" (UID: \"feae394a-bbbf-4624-8057-cf046f9ffb93\") " Oct 30 21:06:10.334217 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:06:10.333263 5378 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/feae394a-bbbf-4624-8057-cf046f9ffb93-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/feae394a-bbbf-4624-8057-cf046f9ffb93-istiod-ca-cert\") pod \"feae394a-bbbf-4624-8057-cf046f9ffb93\" (UID: \"feae394a-bbbf-4624-8057-cf046f9ffb93\") " Oct 30 21:06:10.334217 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:06:10.333314 5378 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/feae394a-bbbf-4624-8057-cf046f9ffb93-kube-api-access-s9mtl\" (UniqueName: \"kubernetes.io/projected/feae394a-bbbf-4624-8057-cf046f9ffb93-kube-api-access-s9mtl\") pod \"feae394a-bbbf-4624-8057-cf046f9ffb93\" (UID: \"feae394a-bbbf-4624-8057-cf046f9ffb93\") " Oct 30 21:06:10.334217 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:06:10.333390 5378 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/feae394a-bbbf-4624-8057-cf046f9ffb93-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/feae394a-bbbf-4624-8057-cf046f9ffb93-istio-data\") pod \"feae394a-bbbf-4624-8057-cf046f9ffb93\" (UID: \"feae394a-bbbf-4624-8057-cf046f9ffb93\") " Oct 30 21:06:10.334217 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:06:10.333391 5378 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/feae394a-bbbf-4624-8057-cf046f9ffb93-workload-socket" pod "feae394a-bbbf-4624-8057-cf046f9ffb93" (UID: "feae394a-bbbf-4624-8057-cf046f9ffb93"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:06:10.334217 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:06:10.333495 5378 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/feae394a-bbbf-4624-8057-cf046f9ffb93-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/feae394a-bbbf-4624-8057-cf046f9ffb93-istio-podinfo\") pod \"feae394a-bbbf-4624-8057-cf046f9ffb93\" (UID: \"feae394a-bbbf-4624-8057-cf046f9ffb93\") " Oct 30 21:06:10.334217 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:06:10.333716 5378 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/feae394a-bbbf-4624-8057-cf046f9ffb93-istiod-ca-cert" pod "feae394a-bbbf-4624-8057-cf046f9ffb93" (UID: "feae394a-bbbf-4624-8057-cf046f9ffb93"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:06:10.334217 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:06:10.333934 5378 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/feae394a-bbbf-4624-8057-cf046f9ffb93-credential-socket" pod "feae394a-bbbf-4624-8057-cf046f9ffb93" (UID: "feae394a-bbbf-4624-8057-cf046f9ffb93"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:06:10.335021 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:06:10.334279 5378 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/feae394a-bbbf-4624-8057-cf046f9ffb93-workload-socket\") on node \"ip-10-0-138-149.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:06:10.335021 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:06:10.334299 5378 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/feae394a-bbbf-4624-8057-cf046f9ffb93-credential-socket\") on node \"ip-10-0-138-149.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:06:10.335021 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:06:10.334322 5378 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/feae394a-bbbf-4624-8057-cf046f9ffb93-istiod-ca-cert\") on node \"ip-10-0-138-149.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:06:10.335021 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:06:10.334335 5378 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/feae394a-bbbf-4624-8057-cf046f9ffb93-workload-certs\") on node \"ip-10-0-138-149.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:06:10.335021 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:06:10.334730 5378 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/feae394a-bbbf-4624-8057-cf046f9ffb93-istio-data" pod "feae394a-bbbf-4624-8057-cf046f9ffb93" (UID: "feae394a-bbbf-4624-8057-cf046f9ffb93"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:06:10.335021 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:06:10.334904 5378 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/feae394a-bbbf-4624-8057-cf046f9ffb93-istio-envoy" pod "feae394a-bbbf-4624-8057-cf046f9ffb93" (UID: "feae394a-bbbf-4624-8057-cf046f9ffb93"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:06:10.336178 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:06:10.336153 5378 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/feae394a-bbbf-4624-8057-cf046f9ffb93-istio-token" pod "feae394a-bbbf-4624-8057-cf046f9ffb93" (UID: "feae394a-bbbf-4624-8057-cf046f9ffb93"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:06:10.337378 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:06:10.337353 5378 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/feae394a-bbbf-4624-8057-cf046f9ffb93-kube-api-access-s9mtl" pod "feae394a-bbbf-4624-8057-cf046f9ffb93" (UID: "feae394a-bbbf-4624-8057-cf046f9ffb93"). InnerVolumeSpecName "kube-api-access-s9mtl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:06:10.337462 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:06:10.337362 5378 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/feae394a-bbbf-4624-8057-cf046f9ffb93-istio-podinfo" pod "feae394a-bbbf-4624-8057-cf046f9ffb93" (UID: "feae394a-bbbf-4624-8057-cf046f9ffb93"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 21:06:10.344203 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:06:10.344159 5378 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-2b425d-openshift-default-f547dbcbf-wllbd"] Oct 30 21:06:10.350778 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:06:10.350453 5378 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w5-2b425d-openshift-default-f547dbcbf-wllbd"] Oct 30 21:06:10.435671 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:06:10.435638 5378 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/feae394a-bbbf-4624-8057-cf046f9ffb93-istio-data\") on node \"ip-10-0-138-149.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:06:10.435797 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:06:10.435676 5378 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/feae394a-bbbf-4624-8057-cf046f9ffb93-istio-podinfo\") on node \"ip-10-0-138-149.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:06:10.435797 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:06:10.435691 5378 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/feae394a-bbbf-4624-8057-cf046f9ffb93-istio-envoy\") on node \"ip-10-0-138-149.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:06:10.435797 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:06:10.435708 5378 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/feae394a-bbbf-4624-8057-cf046f9ffb93-istio-token\") on node \"ip-10-0-138-149.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:06:10.435797 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:06:10.435723 5378 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s9mtl\" (UniqueName: \"kubernetes.io/projected/feae394a-bbbf-4624-8057-cf046f9ffb93-kube-api-access-s9mtl\") on node \"ip-10-0-138-149.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:06:10.639206 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:06:10.638861 5378 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-682999-openshift-default-54dcb797b5-zhslb" Oct 30 21:06:10.646914 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:06:10.646882 5378 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-2b425d-openshift-default-f547dbcbf-j99m9" Oct 30 21:06:10.720943 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:06:10.720917 5378 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-682999-openshift-default-54dcb797b5-xhh97" Oct 30 21:06:10.745109 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:06:10.745079 5378 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/afdb629d-0850-4d80-a6b9-3556bd0d706e-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/afdb629d-0850-4d80-a6b9-3556bd0d706e-workload-socket\") pod \"afdb629d-0850-4d80-a6b9-3556bd0d706e\" (UID: \"afdb629d-0850-4d80-a6b9-3556bd0d706e\") " Oct 30 21:06:10.745267 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:06:10.745195 5378 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/afdb629d-0850-4d80-a6b9-3556bd0d706e-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/afdb629d-0850-4d80-a6b9-3556bd0d706e-istiod-ca-cert\") pod \"afdb629d-0850-4d80-a6b9-3556bd0d706e\" (UID: \"afdb629d-0850-4d80-a6b9-3556bd0d706e\") " Oct 30 21:06:10.745340 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:06:10.745274 5378 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/afdb629d-0850-4d80-a6b9-3556bd0d706e-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/afdb629d-0850-4d80-a6b9-3556bd0d706e-workload-certs\") pod \"afdb629d-0850-4d80-a6b9-3556bd0d706e\" (UID: \"afdb629d-0850-4d80-a6b9-3556bd0d706e\") " Oct 30 21:06:10.745340 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:06:10.745321 5378 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/afdb629d-0850-4d80-a6b9-3556bd0d706e-istio-token\" (UniqueName: \"kubernetes.io/projected/afdb629d-0850-4d80-a6b9-3556bd0d706e-istio-token\") pod \"afdb629d-0850-4d80-a6b9-3556bd0d706e\" (UID: \"afdb629d-0850-4d80-a6b9-3556bd0d706e\") " Oct 30 21:06:10.745467 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:06:10.745379 5378 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/afdb629d-0850-4d80-a6b9-3556bd0d706e-kube-api-access-zngmr\" (UniqueName: \"kubernetes.io/projected/afdb629d-0850-4d80-a6b9-3556bd0d706e-kube-api-access-zngmr\") pod \"afdb629d-0850-4d80-a6b9-3556bd0d706e\" (UID: \"afdb629d-0850-4d80-a6b9-3556bd0d706e\") " Oct 30 21:06:10.745467 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:06:10.745416 5378 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/afdb629d-0850-4d80-a6b9-3556bd0d706e-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/afdb629d-0850-4d80-a6b9-3556bd0d706e-istio-data\") pod \"afdb629d-0850-4d80-a6b9-3556bd0d706e\" (UID: \"afdb629d-0850-4d80-a6b9-3556bd0d706e\") " Oct 30 21:06:10.745593 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:06:10.745477 5378 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/afdb629d-0850-4d80-a6b9-3556bd0d706e-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/afdb629d-0850-4d80-a6b9-3556bd0d706e-credential-socket\") pod \"afdb629d-0850-4d80-a6b9-3556bd0d706e\" (UID: \"afdb629d-0850-4d80-a6b9-3556bd0d706e\") " Oct 30 21:06:10.745593 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:06:10.745516 5378 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/afdb629d-0850-4d80-a6b9-3556bd0d706e-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/afdb629d-0850-4d80-a6b9-3556bd0d706e-istio-podinfo\") pod \"afdb629d-0850-4d80-a6b9-3556bd0d706e\" (UID: \"afdb629d-0850-4d80-a6b9-3556bd0d706e\") " Oct 30 21:06:10.745593 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:06:10.745535 5378 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/afdb629d-0850-4d80-a6b9-3556bd0d706e-workload-certs" pod "afdb629d-0850-4d80-a6b9-3556bd0d706e" (UID: "afdb629d-0850-4d80-a6b9-3556bd0d706e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:06:10.745801 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:06:10.745606 5378 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/afdb629d-0850-4d80-a6b9-3556bd0d706e-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/afdb629d-0850-4d80-a6b9-3556bd0d706e-istio-envoy\") pod \"afdb629d-0850-4d80-a6b9-3556bd0d706e\" (UID: \"afdb629d-0850-4d80-a6b9-3556bd0d706e\") " Oct 30 21:06:10.745861 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:06:10.745798 5378 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/afdb629d-0850-4d80-a6b9-3556bd0d706e-istiod-ca-cert" pod "afdb629d-0850-4d80-a6b9-3556bd0d706e" (UID: "afdb629d-0850-4d80-a6b9-3556bd0d706e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:06:10.745983 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:06:10.745963 5378 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/afdb629d-0850-4d80-a6b9-3556bd0d706e-istio-data" pod "afdb629d-0850-4d80-a6b9-3556bd0d706e" (UID: "afdb629d-0850-4d80-a6b9-3556bd0d706e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:06:10.746198 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:06:10.746178 5378 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/afdb629d-0850-4d80-a6b9-3556bd0d706e-workload-socket" pod "afdb629d-0850-4d80-a6b9-3556bd0d706e" (UID: "afdb629d-0850-4d80-a6b9-3556bd0d706e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:06:10.746280 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:06:10.746197 5378 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/afdb629d-0850-4d80-a6b9-3556bd0d706e-credential-socket" pod "afdb629d-0850-4d80-a6b9-3556bd0d706e" (UID: "afdb629d-0850-4d80-a6b9-3556bd0d706e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:06:10.746483 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:06:10.746466 5378 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/afdb629d-0850-4d80-a6b9-3556bd0d706e-istiod-ca-cert\") on node \"ip-10-0-138-149.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:06:10.746560 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:06:10.746513 5378 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/afdb629d-0850-4d80-a6b9-3556bd0d706e-workload-certs\") on node \"ip-10-0-138-149.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:06:10.746693 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:06:10.746673 5378 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/afdb629d-0850-4d80-a6b9-3556bd0d706e-istio-data\") on node \"ip-10-0-138-149.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:06:10.747971 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:06:10.747950 5378 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/afdb629d-0850-4d80-a6b9-3556bd0d706e-istio-token" pod "afdb629d-0850-4d80-a6b9-3556bd0d706e" (UID: "afdb629d-0850-4d80-a6b9-3556bd0d706e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:06:10.748244 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:06:10.748226 5378 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/afdb629d-0850-4d80-a6b9-3556bd0d706e-istio-envoy" pod "afdb629d-0850-4d80-a6b9-3556bd0d706e" (UID: "afdb629d-0850-4d80-a6b9-3556bd0d706e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:06:10.748825 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:06:10.748800 5378 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/afdb629d-0850-4d80-a6b9-3556bd0d706e-kube-api-access-zngmr" pod "afdb629d-0850-4d80-a6b9-3556bd0d706e" (UID: "afdb629d-0850-4d80-a6b9-3556bd0d706e"). InnerVolumeSpecName "kube-api-access-zngmr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:06:10.749727 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:06:10.749705 5378 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/afdb629d-0850-4d80-a6b9-3556bd0d706e-istio-podinfo" pod "afdb629d-0850-4d80-a6b9-3556bd0d706e" (UID: "afdb629d-0850-4d80-a6b9-3556bd0d706e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 21:06:10.849080 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:06:10.849047 5378 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/95e52f08-4390-4924-89e1-c520ced3743f-istio-token\" (UniqueName: \"kubernetes.io/projected/95e52f08-4390-4924-89e1-c520ced3743f-istio-token\") pod \"95e52f08-4390-4924-89e1-c520ced3743f\" (UID: \"95e52f08-4390-4924-89e1-c520ced3743f\") " Oct 30 21:06:10.849290 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:06:10.849116 5378 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ad099f6e-3035-4a6d-8fef-aea2e4bd1f3b-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ad099f6e-3035-4a6d-8fef-aea2e4bd1f3b-istio-envoy\") pod \"ad099f6e-3035-4a6d-8fef-aea2e4bd1f3b\" (UID: \"ad099f6e-3035-4a6d-8fef-aea2e4bd1f3b\") " Oct 30 21:06:10.849290 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:06:10.849171 5378 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/95e52f08-4390-4924-89e1-c520ced3743f-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/95e52f08-4390-4924-89e1-c520ced3743f-istio-podinfo\") pod \"95e52f08-4390-4924-89e1-c520ced3743f\" (UID: \"95e52f08-4390-4924-89e1-c520ced3743f\") " Oct 30 21:06:10.849290 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:06:10.849232 5378 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ad099f6e-3035-4a6d-8fef-aea2e4bd1f3b-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ad099f6e-3035-4a6d-8fef-aea2e4bd1f3b-istio-data\") pod \"ad099f6e-3035-4a6d-8fef-aea2e4bd1f3b\" (UID: \"ad099f6e-3035-4a6d-8fef-aea2e4bd1f3b\") " Oct 30 21:06:10.849540 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:06:10.849322 5378 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/95e52f08-4390-4924-89e1-c520ced3743f-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/95e52f08-4390-4924-89e1-c520ced3743f-workload-socket\") pod \"95e52f08-4390-4924-89e1-c520ced3743f\" (UID: \"95e52f08-4390-4924-89e1-c520ced3743f\") " Oct 30 21:06:10.849540 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:06:10.849356 5378 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ad099f6e-3035-4a6d-8fef-aea2e4bd1f3b-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ad099f6e-3035-4a6d-8fef-aea2e4bd1f3b-workload-socket\") pod \"ad099f6e-3035-4a6d-8fef-aea2e4bd1f3b\" (UID: \"ad099f6e-3035-4a6d-8fef-aea2e4bd1f3b\") " Oct 30 21:06:10.849540 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:06:10.849381 5378 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ad099f6e-3035-4a6d-8fef-aea2e4bd1f3b-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ad099f6e-3035-4a6d-8fef-aea2e4bd1f3b-credential-socket\") pod \"ad099f6e-3035-4a6d-8fef-aea2e4bd1f3b\" (UID: \"ad099f6e-3035-4a6d-8fef-aea2e4bd1f3b\") " Oct 30 21:06:10.849540 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:06:10.849412 5378 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/95e52f08-4390-4924-89e1-c520ced3743f-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/95e52f08-4390-4924-89e1-c520ced3743f-istio-envoy\") pod \"95e52f08-4390-4924-89e1-c520ced3743f\" (UID: \"95e52f08-4390-4924-89e1-c520ced3743f\") " Oct 30 21:06:10.849540 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:06:10.849471 5378 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ad099f6e-3035-4a6d-8fef-aea2e4bd1f3b-istio-token\" (UniqueName: \"kubernetes.io/projected/ad099f6e-3035-4a6d-8fef-aea2e4bd1f3b-istio-token\") pod \"ad099f6e-3035-4a6d-8fef-aea2e4bd1f3b\" (UID: \"ad099f6e-3035-4a6d-8fef-aea2e4bd1f3b\") " Oct 30 21:06:10.849540 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:06:10.849506 5378 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/95e52f08-4390-4924-89e1-c520ced3743f-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/95e52f08-4390-4924-89e1-c520ced3743f-credential-socket\") pod \"95e52f08-4390-4924-89e1-c520ced3743f\" (UID: \"95e52f08-4390-4924-89e1-c520ced3743f\") " Oct 30 21:06:10.849540 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:06:10.849535 5378 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ad099f6e-3035-4a6d-8fef-aea2e4bd1f3b-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ad099f6e-3035-4a6d-8fef-aea2e4bd1f3b-istiod-ca-cert\") pod \"ad099f6e-3035-4a6d-8fef-aea2e4bd1f3b\" (UID: \"ad099f6e-3035-4a6d-8fef-aea2e4bd1f3b\") " Oct 30 21:06:10.849903 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:06:10.849616 5378 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ad099f6e-3035-4a6d-8fef-aea2e4bd1f3b-kube-api-access-rkgqv\" (UniqueName: \"kubernetes.io/projected/ad099f6e-3035-4a6d-8fef-aea2e4bd1f3b-kube-api-access-rkgqv\") pod \"ad099f6e-3035-4a6d-8fef-aea2e4bd1f3b\" (UID: \"ad099f6e-3035-4a6d-8fef-aea2e4bd1f3b\") " Oct 30 21:06:10.849903 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:06:10.849659 5378 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/ad099f6e-3035-4a6d-8fef-aea2e4bd1f3b-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ad099f6e-3035-4a6d-8fef-aea2e4bd1f3b-istio-podinfo\") pod \"ad099f6e-3035-4a6d-8fef-aea2e4bd1f3b\" (UID: \"ad099f6e-3035-4a6d-8fef-aea2e4bd1f3b\") " Oct 30 21:06:10.849903 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:06:10.849706 5378 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ad099f6e-3035-4a6d-8fef-aea2e4bd1f3b-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ad099f6e-3035-4a6d-8fef-aea2e4bd1f3b-workload-certs\") pod \"ad099f6e-3035-4a6d-8fef-aea2e4bd1f3b\" (UID: \"ad099f6e-3035-4a6d-8fef-aea2e4bd1f3b\") " Oct 30 21:06:10.849903 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:06:10.849750 5378 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/95e52f08-4390-4924-89e1-c520ced3743f-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/95e52f08-4390-4924-89e1-c520ced3743f-istiod-ca-cert\") pod \"95e52f08-4390-4924-89e1-c520ced3743f\" (UID: \"95e52f08-4390-4924-89e1-c520ced3743f\") " Oct 30 21:06:10.849903 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:06:10.849801 5378 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/95e52f08-4390-4924-89e1-c520ced3743f-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/95e52f08-4390-4924-89e1-c520ced3743f-workload-certs\") pod \"95e52f08-4390-4924-89e1-c520ced3743f\" (UID: \"95e52f08-4390-4924-89e1-c520ced3743f\") " Oct 30 21:06:10.849903 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:06:10.849808 5378 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ad099f6e-3035-4a6d-8fef-aea2e4bd1f3b-workload-socket" pod "ad099f6e-3035-4a6d-8fef-aea2e4bd1f3b" (UID: "ad099f6e-3035-4a6d-8fef-aea2e4bd1f3b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:06:10.849903 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:06:10.849846 5378 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/95e52f08-4390-4924-89e1-c520ced3743f-kube-api-access-dkfsp\" (UniqueName: \"kubernetes.io/projected/95e52f08-4390-4924-89e1-c520ced3743f-kube-api-access-dkfsp\") pod \"95e52f08-4390-4924-89e1-c520ced3743f\" (UID: \"95e52f08-4390-4924-89e1-c520ced3743f\") " Oct 30 21:06:10.849903 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:06:10.849878 5378 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/95e52f08-4390-4924-89e1-c520ced3743f-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/95e52f08-4390-4924-89e1-c520ced3743f-istio-data\") pod \"95e52f08-4390-4924-89e1-c520ced3743f\" (UID: \"95e52f08-4390-4924-89e1-c520ced3743f\") " Oct 30 21:06:10.851815 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:06:10.850830 5378 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/afdb629d-0850-4d80-a6b9-3556bd0d706e-istio-podinfo\") on node \"ip-10-0-138-149.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:06:10.851815 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:06:10.850859 5378 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/afdb629d-0850-4d80-a6b9-3556bd0d706e-istio-envoy\") on node \"ip-10-0-138-149.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:06:10.851815 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:06:10.850874 5378 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/afdb629d-0850-4d80-a6b9-3556bd0d706e-workload-socket\") on node \"ip-10-0-138-149.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:06:10.851815 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:06:10.850890 5378 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ad099f6e-3035-4a6d-8fef-aea2e4bd1f3b-workload-socket\") on node \"ip-10-0-138-149.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:06:10.851815 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:06:10.850906 5378 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/afdb629d-0850-4d80-a6b9-3556bd0d706e-istio-token\") on node \"ip-10-0-138-149.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:06:10.851815 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:06:10.850920 5378 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zngmr\" (UniqueName: \"kubernetes.io/projected/afdb629d-0850-4d80-a6b9-3556bd0d706e-kube-api-access-zngmr\") on node \"ip-10-0-138-149.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:06:10.851815 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:06:10.850936 5378 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/afdb629d-0850-4d80-a6b9-3556bd0d706e-credential-socket\") on node \"ip-10-0-138-149.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:06:10.851815 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:06:10.851134 5378 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ad099f6e-3035-4a6d-8fef-aea2e4bd1f3b-istiod-ca-cert" pod "ad099f6e-3035-4a6d-8fef-aea2e4bd1f3b" (UID: "ad099f6e-3035-4a6d-8fef-aea2e4bd1f3b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:06:10.851815 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:06:10.851606 5378 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/95e52f08-4390-4924-89e1-c520ced3743f-workload-certs" pod "95e52f08-4390-4924-89e1-c520ced3743f" (UID: "95e52f08-4390-4924-89e1-c520ced3743f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:06:10.852506 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:06:10.851922 5378 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/95e52f08-4390-4924-89e1-c520ced3743f-istio-data" pod "95e52f08-4390-4924-89e1-c520ced3743f" (UID: "95e52f08-4390-4924-89e1-c520ced3743f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:06:10.897531 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:06:10.869824 5378 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ad099f6e-3035-4a6d-8fef-aea2e4bd1f3b-workload-certs" pod "ad099f6e-3035-4a6d-8fef-aea2e4bd1f3b" (UID: "ad099f6e-3035-4a6d-8fef-aea2e4bd1f3b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:06:10.897531 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:06:10.871045 5378 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ad099f6e-3035-4a6d-8fef-aea2e4bd1f3b-credential-socket" pod "ad099f6e-3035-4a6d-8fef-aea2e4bd1f3b" (UID: "ad099f6e-3035-4a6d-8fef-aea2e4bd1f3b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:06:10.897531 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:06:10.886470 5378 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/95e52f08-4390-4924-89e1-c520ced3743f-credential-socket" pod "95e52f08-4390-4924-89e1-c520ced3743f" (UID: "95e52f08-4390-4924-89e1-c520ced3743f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:06:10.897531 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:06:10.889113 5378 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ad099f6e-3035-4a6d-8fef-aea2e4bd1f3b-istio-envoy" pod "ad099f6e-3035-4a6d-8fef-aea2e4bd1f3b" (UID: "ad099f6e-3035-4a6d-8fef-aea2e4bd1f3b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:06:10.897531 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:06:10.893686 5378 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ad099f6e-3035-4a6d-8fef-aea2e4bd1f3b-istio-data" pod "ad099f6e-3035-4a6d-8fef-aea2e4bd1f3b" (UID: "ad099f6e-3035-4a6d-8fef-aea2e4bd1f3b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:06:10.897960 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:06:10.897637 5378 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/95e52f08-4390-4924-89e1-c520ced3743f-workload-socket" pod "95e52f08-4390-4924-89e1-c520ced3743f" (UID: "95e52f08-4390-4924-89e1-c520ced3743f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:06:10.897960 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:06:10.897799 5378 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/95e52f08-4390-4924-89e1-c520ced3743f-istiod-ca-cert" pod "95e52f08-4390-4924-89e1-c520ced3743f" (UID: "95e52f08-4390-4924-89e1-c520ced3743f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:06:10.920677 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:06:10.920629 5378 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ad099f6e-3035-4a6d-8fef-aea2e4bd1f3b-kube-api-access-rkgqv" pod "ad099f6e-3035-4a6d-8fef-aea2e4bd1f3b" (UID: "ad099f6e-3035-4a6d-8fef-aea2e4bd1f3b"). InnerVolumeSpecName "kube-api-access-rkgqv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:06:10.935675 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:06:10.935104 5378 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/95e52f08-4390-4924-89e1-c520ced3743f-istio-podinfo" pod "95e52f08-4390-4924-89e1-c520ced3743f" (UID: "95e52f08-4390-4924-89e1-c520ced3743f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 21:06:10.935675 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:06:10.935396 5378 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ad099f6e-3035-4a6d-8fef-aea2e4bd1f3b-istio-podinfo" pod "ad099f6e-3035-4a6d-8fef-aea2e4bd1f3b" (UID: "ad099f6e-3035-4a6d-8fef-aea2e4bd1f3b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 21:06:10.935675 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:06:10.935430 5378 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/95e52f08-4390-4924-89e1-c520ced3743f-istio-envoy" pod "95e52f08-4390-4924-89e1-c520ced3743f" (UID: "95e52f08-4390-4924-89e1-c520ced3743f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:06:10.935675 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:06:10.935521 5378 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/95e52f08-4390-4924-89e1-c520ced3743f-kube-api-access-dkfsp" pod "95e52f08-4390-4924-89e1-c520ced3743f" (UID: "95e52f08-4390-4924-89e1-c520ced3743f"). InnerVolumeSpecName "kube-api-access-dkfsp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:06:10.936035 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:06:10.935817 5378 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/95e52f08-4390-4924-89e1-c520ced3743f-istio-token" pod "95e52f08-4390-4924-89e1-c520ced3743f" (UID: "95e52f08-4390-4924-89e1-c520ced3743f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:06:10.937732 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:06:10.937702 5378 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ad099f6e-3035-4a6d-8fef-aea2e4bd1f3b-istio-token" pod "ad099f6e-3035-4a6d-8fef-aea2e4bd1f3b" (UID: "ad099f6e-3035-4a6d-8fef-aea2e4bd1f3b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:06:10.952480 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:06:10.952455 5378 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rkgqv\" (UniqueName: \"kubernetes.io/projected/ad099f6e-3035-4a6d-8fef-aea2e4bd1f3b-kube-api-access-rkgqv\") on node \"ip-10-0-138-149.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:06:10.952480 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:06:10.952480 5378 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ad099f6e-3035-4a6d-8fef-aea2e4bd1f3b-istio-podinfo\") on node \"ip-10-0-138-149.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:06:10.952698 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:06:10.952490 5378 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ad099f6e-3035-4a6d-8fef-aea2e4bd1f3b-workload-certs\") on node \"ip-10-0-138-149.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:06:10.952698 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:06:10.952499 5378 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/95e52f08-4390-4924-89e1-c520ced3743f-istiod-ca-cert\") on node \"ip-10-0-138-149.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:06:10.952698 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:06:10.952509 5378 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/95e52f08-4390-4924-89e1-c520ced3743f-workload-certs\") on node \"ip-10-0-138-149.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:06:10.952698 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:06:10.952521 5378 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dkfsp\" (UniqueName: \"kubernetes.io/projected/95e52f08-4390-4924-89e1-c520ced3743f-kube-api-access-dkfsp\") on node \"ip-10-0-138-149.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:06:10.952698 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:06:10.952534 5378 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/95e52f08-4390-4924-89e1-c520ced3743f-istio-data\") on node \"ip-10-0-138-149.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:06:10.952698 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:06:10.952546 5378 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/95e52f08-4390-4924-89e1-c520ced3743f-istio-token\") on node \"ip-10-0-138-149.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:06:10.952698 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:06:10.952559 5378 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ad099f6e-3035-4a6d-8fef-aea2e4bd1f3b-istio-envoy\") on node \"ip-10-0-138-149.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:06:10.952698 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:06:10.952613 5378 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/95e52f08-4390-4924-89e1-c520ced3743f-istio-podinfo\") on node \"ip-10-0-138-149.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:06:10.952698 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:06:10.952626 5378 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ad099f6e-3035-4a6d-8fef-aea2e4bd1f3b-istio-data\") on node \"ip-10-0-138-149.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:06:10.952698 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:06:10.952641 5378 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/95e52f08-4390-4924-89e1-c520ced3743f-workload-socket\") on node \"ip-10-0-138-149.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:06:10.952698 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:06:10.952654 5378 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ad099f6e-3035-4a6d-8fef-aea2e4bd1f3b-credential-socket\") on node \"ip-10-0-138-149.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:06:10.952698 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:06:10.952667 5378 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/95e52f08-4390-4924-89e1-c520ced3743f-istio-envoy\") on node \"ip-10-0-138-149.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:06:10.952698 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:06:10.952680 5378 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ad099f6e-3035-4a6d-8fef-aea2e4bd1f3b-istio-token\") on node \"ip-10-0-138-149.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:06:10.952698 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:06:10.952693 5378 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/95e52f08-4390-4924-89e1-c520ced3743f-credential-socket\") on node \"ip-10-0-138-149.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:06:10.952698 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:06:10.952706 5378 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ad099f6e-3035-4a6d-8fef-aea2e4bd1f3b-istiod-ca-cert\") on node \"ip-10-0-138-149.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:06:11.143705 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:06:11.143615 5378 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-2b425d-openshift-default-f547dbcbf-j99m9" event={"ID":"ad099f6e-3035-4a6d-8fef-aea2e4bd1f3b","Type":"ContainerDied","Data":"2ede2722fc5b8a46f879817b490ad7a6f2de00f3ae78584919d3579fd0b07bf8"} Oct 30 21:06:11.143705 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:06:11.143638 5378 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-2b425d-openshift-default-f547dbcbf-j99m9" Oct 30 21:06:11.143705 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:06:11.143690 5378 scope.go:122] "RemoveContainer" containerID="3119e4b6f8ac9712c0e43d657660719bcf26b0d85397424196cf757a1ad2f077" Oct 30 21:06:11.145748 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:06:11.145726 5378 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-682999-openshift-default-54dcb797b5-r27vc" event={"ID":"feae394a-bbbf-4624-8057-cf046f9ffb93","Type":"ContainerDied","Data":"919530f442fabc9c6940fba40a1b00fbe28153e954de4f432e3356e9568d065d"} Oct 30 21:06:11.145839 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:06:11.145799 5378 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-682999-openshift-default-54dcb797b5-r27vc" Oct 30 21:06:11.148907 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:06:11.148882 5378 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-682999-openshift-default-54dcb797b5-xhh97" Oct 30 21:06:11.149660 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:06:11.149634 5378 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-682999-openshift-default-54dcb797b5-xhh97" event={"ID":"95e52f08-4390-4924-89e1-c520ced3743f","Type":"ContainerDied","Data":"44d7c576d9e5b823fe45952d3aef0333805f0a52f72dd7f8931bec2a5dd48500"} Oct 30 21:06:11.214887 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:06:11.214858 5378 scope.go:122] "RemoveContainer" containerID="92f037c57c81aec3aac423afef2a5de0ad66cac77aad6a892e410e96d93b5543" Oct 30 21:06:11.216491 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:06:11.216462 5378 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-682999-openshift-default-54dcb797b5-zhslb" event={"ID":"afdb629d-0850-4d80-a6b9-3556bd0d706e","Type":"ContainerDied","Data":"7894447d60effb1a8aed8bdef86167c1fdb9ef27d8619a9bf5699c67e59ceb3c"} Oct 30 21:06:11.223794 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:06:11.223770 5378 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-682999-openshift-default-54dcb797b5-zhslb" Oct 30 21:06:11.234247 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:06:11.232878 5378 scope.go:122] "RemoveContainer" containerID="4820660cffc38c84c6a5bb8f793ea741daa1d57b1c65c2e8ae8dcc6211617a48" Oct 30 21:06:11.246757 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:06:11.246738 5378 scope.go:122] "RemoveContainer" containerID="1b8acf4813b04532a32225114288d63035b23d2beca277eb36e69cec3a33c818" Oct 30 21:06:11.435618 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:06:11.435587 5378 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-682999-openshift-default-54dcb797b5-xhh97"] Oct 30 21:06:11.438557 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:06:11.438537 5378 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-682999-openshift-default-54dcb797b5-xhh97"] Oct 30 21:06:11.584665 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:06:11.584625 5378 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-682999-openshift-default-54dcb797b5-zhslb"] Oct 30 21:06:11.588833 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:06:11.588797 5378 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-682999-openshift-default-54dcb797b5-zhslb"] Oct 30 21:06:11.701623 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:06:11.701517 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:06:11.706612 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:06:11.703677 5378 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-682999-openshift-default-54dcb797b5-r27vc"] Oct 30 21:06:11.709853 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:06:11.709834 5378 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-682999-openshift-default-54dcb797b5-r27vc"] Oct 30 21:06:11.744555 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:06:11.744521 5378 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="95e52f08-4390-4924-89e1-c520ced3743f" path="/var/lib/kubelet/pods/95e52f08-4390-4924-89e1-c520ced3743f/volumes" Oct 30 21:06:11.745113 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:06:11.745093 5378 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="afdb629d-0850-4d80-a6b9-3556bd0d706e" path="/var/lib/kubelet/pods/afdb629d-0850-4d80-a6b9-3556bd0d706e/volumes" Oct 30 21:06:11.745621 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:06:11.745603 5378 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ce8609a7-1869-4d3e-b1d7-adab0908fbb9" path="/var/lib/kubelet/pods/ce8609a7-1869-4d3e-b1d7-adab0908fbb9/volumes" Oct 30 21:06:11.746435 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:06:11.746414 5378 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="feae394a-bbbf-4624-8057-cf046f9ffb93" path="/var/lib/kubelet/pods/feae394a-bbbf-4624-8057-cf046f9ffb93/volumes" Oct 30 21:06:11.814930 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:06:11.814903 5378 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-2b425d-openshift-default-f547dbcbf-j99m9"] Oct 30 21:06:11.818159 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:06:11.818130 5378 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w5-2b425d-openshift-default-f547dbcbf-j99m9"] Oct 30 21:06:11.991258 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:06:11.991175 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:06:13.744872 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:06:13.744830 5378 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ad099f6e-3035-4a6d-8fef-aea2e4bd1f3b" path="/var/lib/kubelet/pods/ad099f6e-3035-4a6d-8fef-aea2e4bd1f3b/volumes" Oct 30 21:06:15.668797 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:06:15.668766 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:06:16.800511 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:06:16.800424 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:06:17.277720 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:06:17.277687 5378 generic.go:356] "Generic (PLEG): container finished" podID="0406854a-c204-4495-bdc8-23afaad6457f" containerID="8793bedb56a074407b42ba1d90aedb3e2ea15f4e71c7cdabf051ca723976299e" exitCode=0 Oct 30 21:06:17.278287 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:06:17.277757 5378 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-19db3d5f-openshift-default-7f4c69b8fc-vjtzt" event={"ID":"0406854a-c204-4495-bdc8-23afaad6457f","Type":"ContainerDied","Data":"8793bedb56a074407b42ba1d90aedb3e2ea15f4e71c7cdabf051ca723976299e"} Oct 30 21:06:17.278287 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:06:17.278225 5378 scope.go:122] "RemoveContainer" containerID="8793bedb56a074407b42ba1d90aedb3e2ea15f4e71c7cdabf051ca723976299e" Oct 30 21:06:17.279499 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:06:17.279479 5378 generic.go:356] "Generic (PLEG): container finished" podID="a5fd3bfb-32af-4284-9bed-e0e55a9dacd3" containerID="36d9a82525d731c9a76b2b3ef95df6120d9c572ec2ea1468eb7a27de25648b71" exitCode=0 Oct 30 21:06:17.279634 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:06:17.279549 5378 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-de53c2-openshift-default-58f78dcbb7-lxxs8" event={"ID":"a5fd3bfb-32af-4284-9bed-e0e55a9dacd3","Type":"ContainerDied","Data":"36d9a82525d731c9a76b2b3ef95df6120d9c572ec2ea1468eb7a27de25648b71"} Oct 30 21:06:17.279907 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:06:17.279878 5378 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 21:06:17.279997 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:06:17.279915 5378 scope.go:122] "RemoveContainer" containerID="36d9a82525d731c9a76b2b3ef95df6120d9c572ec2ea1468eb7a27de25648b71" Oct 30 21:06:17.279997 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:06:17.279968 5378 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 21:06:17.280123 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:06:17.280020 5378 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 21:06:17.281299 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:06:17.281273 5378 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 21:06:17.281419 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:06:17.281335 5378 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 21:06:17.281419 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:06:17.281366 5378 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 21:06:18.309177 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:06:18.309134 5378 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-19db3d5f-openshift-default-7f4c69b8fc-vjtzt" event={"ID":"0406854a-c204-4495-bdc8-23afaad6457f","Type":"ContainerStarted","Data":"43dad921d5d07df6927ef56819f534c8b66f7abac3bd91b97c2593ac511fc94c"} Oct 30 21:06:18.311252 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:06:18.311228 5378 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-de53c2-openshift-default-58f78dcbb7-lxxs8" event={"ID":"a5fd3bfb-32af-4284-9bed-e0e55a9dacd3","Type":"ContainerStarted","Data":"8bb6aa694c2f2f98db299ea198a6a5ac67e48d239e865c892c29e989901a4070"} Oct 30 21:06:18.840747 ip-10-0-138-149 kubenswrapper[5378]: E1030 21:06:18.840714 5378 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q6mx7" containerName="extractor" Oct 30 21:06:18.887589 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:06:18.883697 5378 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w2-de53c2-openshift-default-58f78dcbb7-lxxs8" Oct 30 21:06:19.115456 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:06:19.115365 5378 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w2-de53c2-openshift-default-58f78dcbb7-lxxs8" Oct 30 21:06:19.116210 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:06:19.116186 5378 patch_prober.go:29] interesting pod/e2e-shared-a-w2-de53c2-openshift-default-58f78dcbb7-lxxs8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.113:15021/healthz/ready\": dial tcp 10.133.0.113:15021: connect: connection refused" start-of-body= Oct 30 21:06:19.116270 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:06:19.116242 5378 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-de53c2-openshift-default-58f78dcbb7-lxxs8" podUID="a5fd3bfb-32af-4284-9bed-e0e55a9dacd3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.113:15021/healthz/ready\": dial tcp 10.133.0.113:15021: connect: connection refused" Oct 30 21:06:19.132032 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:06:19.132010 5378 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-19db3d5f-openshift-default-7f4c69b8fc-vjtzt" Oct 30 21:06:19.133717 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:06:19.132803 5378 patch_prober.go:29] interesting pod/e2e-mt-19db3d5f-openshift-default-7f4c69b8fc-vjtzt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.114:15021/healthz/ready\": dial tcp 10.133.0.114:15021: connect: connection refused" start-of-body= Oct 30 21:06:19.133717 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:06:19.132860 5378 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-19db3d5f-openshift-default-7f4c69b8fc-vjtzt" podUID="0406854a-c204-4495-bdc8-23afaad6457f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.114:15021/healthz/ready\": dial tcp 10.133.0.114:15021: connect: connection refused" Oct 30 21:06:20.115777 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:06:20.115743 5378 patch_prober.go:29] interesting pod/e2e-shared-a-w2-de53c2-openshift-default-58f78dcbb7-lxxs8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.113:15021/healthz/ready\": dial tcp 10.133.0.113:15021: connect: connection refused" start-of-body= Oct 30 21:06:20.116172 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:06:20.115804 5378 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-de53c2-openshift-default-58f78dcbb7-lxxs8" podUID="a5fd3bfb-32af-4284-9bed-e0e55a9dacd3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.113:15021/healthz/ready\": dial tcp 10.133.0.113:15021: connect: connection refused" Oct 30 21:06:20.132022 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:06:20.131996 5378 patch_prober.go:29] interesting pod/e2e-mt-19db3d5f-openshift-default-7f4c69b8fc-vjtzt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.114:15021/healthz/ready\": dial tcp 10.133.0.114:15021: connect: connection refused" start-of-body= Oct 30 21:06:20.132105 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:06:20.132036 5378 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-19db3d5f-openshift-default-7f4c69b8fc-vjtzt" podUID="0406854a-c204-4495-bdc8-23afaad6457f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.114:15021/healthz/ready\": dial tcp 10.133.0.114:15021: connect: connection refused" Oct 30 21:06:21.127209 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:06:21.127167 5378 patch_prober.go:29] interesting pod/e2e-shared-a-w2-de53c2-openshift-default-58f78dcbb7-lxxs8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.113:15021/healthz/ready\": dial tcp 10.133.0.113:15021: connect: connection refused" start-of-body= Oct 30 21:06:21.128326 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:06:21.128282 5378 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-de53c2-openshift-default-58f78dcbb7-lxxs8" podUID="a5fd3bfb-32af-4284-9bed-e0e55a9dacd3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.113:15021/healthz/ready\": dial tcp 10.133.0.113:15021: connect: connection refused" Oct 30 21:06:21.131965 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:06:21.131923 5378 patch_prober.go:29] interesting pod/e2e-mt-19db3d5f-openshift-default-7f4c69b8fc-vjtzt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.114:15021/healthz/ready\": dial tcp 10.133.0.114:15021: connect: connection refused" start-of-body= Oct 30 21:06:21.132165 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:06:21.132141 5378 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-19db3d5f-openshift-default-7f4c69b8fc-vjtzt" podUID="0406854a-c204-4495-bdc8-23afaad6457f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.114:15021/healthz/ready\": dial tcp 10.133.0.114:15021: connect: connection refused" Oct 30 21:06:21.899589 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:06:21.895701 5378 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-19db3d5f-openshift-default-7f4c69b8fc-vjtzt" Oct 30 21:06:22.116591 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:06:22.116250 5378 patch_prober.go:29] interesting pod/e2e-shared-a-w2-de53c2-openshift-default-58f78dcbb7-lxxs8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.113:15021/healthz/ready\": dial tcp 10.133.0.113:15021: connect: connection refused" start-of-body= Oct 30 21:06:22.116591 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:06:22.116360 5378 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-de53c2-openshift-default-58f78dcbb7-lxxs8" podUID="a5fd3bfb-32af-4284-9bed-e0e55a9dacd3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.113:15021/healthz/ready\": dial tcp 10.133.0.113:15021: connect: connection refused" Oct 30 21:06:22.132151 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:06:22.132101 5378 patch_prober.go:29] interesting pod/e2e-mt-19db3d5f-openshift-default-7f4c69b8fc-vjtzt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.114:15021/healthz/ready\": dial tcp 10.133.0.114:15021: connect: connection refused" start-of-body= Oct 30 21:06:22.132538 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:06:22.132179 5378 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-19db3d5f-openshift-default-7f4c69b8fc-vjtzt" podUID="0406854a-c204-4495-bdc8-23afaad6457f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.114:15021/healthz/ready\": dial tcp 10.133.0.114:15021: connect: connection refused" Oct 30 21:06:22.304186 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:06:22.304047 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:06:23.117589 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:06:23.116933 5378 patch_prober.go:29] interesting pod/e2e-shared-a-w2-de53c2-openshift-default-58f78dcbb7-lxxs8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.113:15021/healthz/ready\": dial tcp 10.133.0.113:15021: connect: connection refused" start-of-body= Oct 30 21:06:23.117589 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:06:23.116997 5378 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-de53c2-openshift-default-58f78dcbb7-lxxs8" podUID="a5fd3bfb-32af-4284-9bed-e0e55a9dacd3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.113:15021/healthz/ready\": dial tcp 10.133.0.113:15021: connect: connection refused" Oct 30 21:06:23.132590 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:06:23.131885 5378 patch_prober.go:29] interesting pod/e2e-mt-19db3d5f-openshift-default-7f4c69b8fc-vjtzt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.114:15021/healthz/ready\": dial tcp 10.133.0.114:15021: connect: connection refused" start-of-body= Oct 30 21:06:23.132590 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:06:23.131977 5378 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-19db3d5f-openshift-default-7f4c69b8fc-vjtzt" podUID="0406854a-c204-4495-bdc8-23afaad6457f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.114:15021/healthz/ready\": dial tcp 10.133.0.114:15021: connect: connection refused" Oct 30 21:06:24.119954 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:06:24.119921 5378 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w2-de53c2-openshift-default-58f78dcbb7-lxxs8" Oct 30 21:06:24.121008 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:06:24.120985 5378 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w2-de53c2-openshift-default-58f78dcbb7-lxxs8" Oct 30 21:06:24.137045 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:06:24.137025 5378 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-19db3d5f-openshift-default-7f4c69b8fc-vjtzt" Oct 30 21:06:24.138147 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:06:24.138123 5378 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-19db3d5f-openshift-default-7f4c69b8fc-vjtzt" Oct 30 21:06:27.157052 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:06:27.157002 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:06:27.603444 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:06:27.603411 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:06:27.737437 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:06:27.737357 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:06:28.199377 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:06:28.199303 5378 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w4-88da25-openshift-default-6c4ccc4cd9-tcsdd"] Oct 30 21:06:28.200198 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:06:28.200178 5378 cpu_manager.go:414] "containerMap: removing container" podUID="ad099f6e-3035-4a6d-8fef-aea2e4bd1f3b" containerName="istio-proxy" Oct 30 21:06:28.200251 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:06:28.200201 5378 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ad099f6e-3035-4a6d-8fef-aea2e4bd1f3b" containerName="istio-proxy" Oct 30 21:06:28.200251 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:06:28.200212 5378 cpu_manager.go:414] "containerMap: removing container" podUID="afdb629d-0850-4d80-a6b9-3556bd0d706e" containerName="istio-proxy" Oct 30 21:06:28.200251 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:06:28.200217 5378 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="afdb629d-0850-4d80-a6b9-3556bd0d706e" containerName="istio-proxy" Oct 30 21:06:28.200251 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:06:28.200234 5378 cpu_manager.go:414] "containerMap: removing container" podUID="feae394a-bbbf-4624-8057-cf046f9ffb93" containerName="istio-proxy" Oct 30 21:06:28.200251 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:06:28.200239 5378 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="feae394a-bbbf-4624-8057-cf046f9ffb93" containerName="istio-proxy" Oct 30 21:06:28.200251 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:06:28.200250 5378 cpu_manager.go:414] "containerMap: removing container" podUID="9723840f-d122-4d53-8600-346cd5f2db0d" containerName="payload-pre-processing" Oct 30 21:06:28.200441 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:06:28.200255 5378 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9723840f-d122-4d53-8600-346cd5f2db0d" containerName="payload-pre-processing" Oct 30 21:06:28.200441 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:06:28.200271 5378 cpu_manager.go:414] "containerMap: removing container" podUID="ce8609a7-1869-4d3e-b1d7-adab0908fbb9" containerName="istio-proxy" Oct 30 21:06:28.200441 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:06:28.200276 5378 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ce8609a7-1869-4d3e-b1d7-adab0908fbb9" containerName="istio-proxy" Oct 30 21:06:28.200441 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:06:28.200291 5378 cpu_manager.go:414] "containerMap: removing container" podUID="95e52f08-4390-4924-89e1-c520ced3743f" containerName="istio-proxy" Oct 30 21:06:28.200441 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:06:28.200295 5378 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="95e52f08-4390-4924-89e1-c520ced3743f" containerName="istio-proxy" Oct 30 21:06:28.200441 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:06:28.200394 5378 memory_manager.go:365] "RemoveStaleState removing state" podUID="ad099f6e-3035-4a6d-8fef-aea2e4bd1f3b" containerName="istio-proxy" Oct 30 21:06:28.200441 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:06:28.200404 5378 memory_manager.go:365] "RemoveStaleState removing state" podUID="afdb629d-0850-4d80-a6b9-3556bd0d706e" containerName="istio-proxy" Oct 30 21:06:28.200441 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:06:28.200410 5378 memory_manager.go:365] "RemoveStaleState removing state" podUID="feae394a-bbbf-4624-8057-cf046f9ffb93" containerName="istio-proxy" Oct 30 21:06:28.200441 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:06:28.200418 5378 memory_manager.go:365] "RemoveStaleState removing state" podUID="9723840f-d122-4d53-8600-346cd5f2db0d" containerName="payload-pre-processing" Oct 30 21:06:28.200441 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:06:28.200425 5378 memory_manager.go:365] "RemoveStaleState removing state" podUID="ce8609a7-1869-4d3e-b1d7-adab0908fbb9" containerName="istio-proxy" Oct 30 21:06:28.200441 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:06:28.200432 5378 memory_manager.go:365] "RemoveStaleState removing state" podUID="95e52f08-4390-4924-89e1-c520ced3743f" containerName="istio-proxy" Oct 30 21:06:28.203191 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:06:28.203167 5378 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w4-88da25-openshift-default-6c4ccc4cd9-tcsdd" Oct 30 21:06:28.265057 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:06:28.265014 5378 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w4-88da25-openshift-default-dockercfg-7fvlj\"" Oct 30 21:06:28.293703 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:06:28.293496 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:06:28.312603 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:06:28.305726 5378 patch_prober.go:29] interesting pod/e2e-shared-a-w4-5c45e5-openshift-default-9dcdd54d9-s697g container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.109:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 21:06:28.312603 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:06:28.305809 5378 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-w4-5c45e5-openshift-default-9dcdd54d9-s697g" podUID="23f08320-e0a1-42bc-8fd1-de3492b59bca" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.109:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 21:06:28.344152 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:06:28.344117 5378 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/191dc733-cb36-4efc-9572-e645e2eb59d8-istio-envoy\") pod \"e2e-shared-b-w4-88da25-openshift-default-6c4ccc4cd9-tcsdd\" (UID: \"191dc733-cb36-4efc-9572-e645e2eb59d8\") " pod="openshift-ingress/e2e-shared-b-w4-88da25-openshift-default-6c4ccc4cd9-tcsdd" Oct 30 21:06:28.365661 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:06:28.344402 5378 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/191dc733-cb36-4efc-9572-e645e2eb59d8-workload-socket\") pod \"e2e-shared-b-w4-88da25-openshift-default-6c4ccc4cd9-tcsdd\" (UID: \"191dc733-cb36-4efc-9572-e645e2eb59d8\") " pod="openshift-ingress/e2e-shared-b-w4-88da25-openshift-default-6c4ccc4cd9-tcsdd" Oct 30 21:06:28.366114 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:06:28.366086 5378 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/191dc733-cb36-4efc-9572-e645e2eb59d8-workload-certs\") pod \"e2e-shared-b-w4-88da25-openshift-default-6c4ccc4cd9-tcsdd\" (UID: \"191dc733-cb36-4efc-9572-e645e2eb59d8\") " pod="openshift-ingress/e2e-shared-b-w4-88da25-openshift-default-6c4ccc4cd9-tcsdd" Oct 30 21:06:28.372946 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:06:28.372822 5378 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/191dc733-cb36-4efc-9572-e645e2eb59d8-istiod-ca-cert\") pod \"e2e-shared-b-w4-88da25-openshift-default-6c4ccc4cd9-tcsdd\" (UID: \"191dc733-cb36-4efc-9572-e645e2eb59d8\") " pod="openshift-ingress/e2e-shared-b-w4-88da25-openshift-default-6c4ccc4cd9-tcsdd" Oct 30 21:06:28.406939 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:06:28.406905 5378 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/191dc733-cb36-4efc-9572-e645e2eb59d8-istio-podinfo\") pod \"e2e-shared-b-w4-88da25-openshift-default-6c4ccc4cd9-tcsdd\" (UID: \"191dc733-cb36-4efc-9572-e645e2eb59d8\") " pod="openshift-ingress/e2e-shared-b-w4-88da25-openshift-default-6c4ccc4cd9-tcsdd" Oct 30 21:06:28.409796 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:06:28.409764 5378 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/191dc733-cb36-4efc-9572-e645e2eb59d8-istio-token\") pod \"e2e-shared-b-w4-88da25-openshift-default-6c4ccc4cd9-tcsdd\" (UID: \"191dc733-cb36-4efc-9572-e645e2eb59d8\") " pod="openshift-ingress/e2e-shared-b-w4-88da25-openshift-default-6c4ccc4cd9-tcsdd" Oct 30 21:06:28.410152 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:06:28.410129 5378 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6n2gm\" (UniqueName: \"kubernetes.io/projected/191dc733-cb36-4efc-9572-e645e2eb59d8-kube-api-access-6n2gm\") pod \"e2e-shared-b-w4-88da25-openshift-default-6c4ccc4cd9-tcsdd\" (UID: \"191dc733-cb36-4efc-9572-e645e2eb59d8\") " pod="openshift-ingress/e2e-shared-b-w4-88da25-openshift-default-6c4ccc4cd9-tcsdd" Oct 30 21:06:28.419713 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:06:28.419673 5378 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/191dc733-cb36-4efc-9572-e645e2eb59d8-istio-data\") pod \"e2e-shared-b-w4-88da25-openshift-default-6c4ccc4cd9-tcsdd\" (UID: \"191dc733-cb36-4efc-9572-e645e2eb59d8\") " pod="openshift-ingress/e2e-shared-b-w4-88da25-openshift-default-6c4ccc4cd9-tcsdd" Oct 30 21:06:28.420089 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:06:28.420059 5378 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-88da25-openshift-default-6c4ccc4cd9-tcsdd"] Oct 30 21:06:28.420352 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:06:28.420327 5378 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/191dc733-cb36-4efc-9572-e645e2eb59d8-credential-socket\") pod \"e2e-shared-b-w4-88da25-openshift-default-6c4ccc4cd9-tcsdd\" (UID: \"191dc733-cb36-4efc-9572-e645e2eb59d8\") " pod="openshift-ingress/e2e-shared-b-w4-88da25-openshift-default-6c4ccc4cd9-tcsdd" Oct 30 21:06:28.561669 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:06:28.533714 5378 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/191dc733-cb36-4efc-9572-e645e2eb59d8-istio-podinfo\") pod \"e2e-shared-b-w4-88da25-openshift-default-6c4ccc4cd9-tcsdd\" (UID: \"191dc733-cb36-4efc-9572-e645e2eb59d8\") " pod="openshift-ingress/e2e-shared-b-w4-88da25-openshift-default-6c4ccc4cd9-tcsdd" Oct 30 21:06:28.571703 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:06:28.571668 5378 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/191dc733-cb36-4efc-9572-e645e2eb59d8-istio-token\") pod \"e2e-shared-b-w4-88da25-openshift-default-6c4ccc4cd9-tcsdd\" (UID: \"191dc733-cb36-4efc-9572-e645e2eb59d8\") " pod="openshift-ingress/e2e-shared-b-w4-88da25-openshift-default-6c4ccc4cd9-tcsdd" Oct 30 21:06:28.575172 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:06:28.575005 5378 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6n2gm\" (UniqueName: \"kubernetes.io/projected/191dc733-cb36-4efc-9572-e645e2eb59d8-kube-api-access-6n2gm\") pod \"e2e-shared-b-w4-88da25-openshift-default-6c4ccc4cd9-tcsdd\" (UID: \"191dc733-cb36-4efc-9572-e645e2eb59d8\") " pod="openshift-ingress/e2e-shared-b-w4-88da25-openshift-default-6c4ccc4cd9-tcsdd" Oct 30 21:06:28.576602 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:06:28.576558 5378 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/191dc733-cb36-4efc-9572-e645e2eb59d8-istio-data\") pod \"e2e-shared-b-w4-88da25-openshift-default-6c4ccc4cd9-tcsdd\" (UID: \"191dc733-cb36-4efc-9572-e645e2eb59d8\") " pod="openshift-ingress/e2e-shared-b-w4-88da25-openshift-default-6c4ccc4cd9-tcsdd" Oct 30 21:06:28.577213 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:06:28.577194 5378 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/191dc733-cb36-4efc-9572-e645e2eb59d8-credential-socket\") pod \"e2e-shared-b-w4-88da25-openshift-default-6c4ccc4cd9-tcsdd\" (UID: \"191dc733-cb36-4efc-9572-e645e2eb59d8\") " pod="openshift-ingress/e2e-shared-b-w4-88da25-openshift-default-6c4ccc4cd9-tcsdd" Oct 30 21:06:28.577669 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:06:28.577653 5378 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/191dc733-cb36-4efc-9572-e645e2eb59d8-istio-envoy\") pod \"e2e-shared-b-w4-88da25-openshift-default-6c4ccc4cd9-tcsdd\" (UID: \"191dc733-cb36-4efc-9572-e645e2eb59d8\") " pod="openshift-ingress/e2e-shared-b-w4-88da25-openshift-default-6c4ccc4cd9-tcsdd" Oct 30 21:06:28.578281 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:06:28.578261 5378 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/191dc733-cb36-4efc-9572-e645e2eb59d8-workload-socket\") pod \"e2e-shared-b-w4-88da25-openshift-default-6c4ccc4cd9-tcsdd\" (UID: \"191dc733-cb36-4efc-9572-e645e2eb59d8\") " pod="openshift-ingress/e2e-shared-b-w4-88da25-openshift-default-6c4ccc4cd9-tcsdd" Oct 30 21:06:28.578780 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:06:28.578764 5378 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/191dc733-cb36-4efc-9572-e645e2eb59d8-workload-certs\") pod \"e2e-shared-b-w4-88da25-openshift-default-6c4ccc4cd9-tcsdd\" (UID: \"191dc733-cb36-4efc-9572-e645e2eb59d8\") " pod="openshift-ingress/e2e-shared-b-w4-88da25-openshift-default-6c4ccc4cd9-tcsdd" Oct 30 21:06:28.579677 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:06:28.579659 5378 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/191dc733-cb36-4efc-9572-e645e2eb59d8-istiod-ca-cert\") pod \"e2e-shared-b-w4-88da25-openshift-default-6c4ccc4cd9-tcsdd\" (UID: \"191dc733-cb36-4efc-9572-e645e2eb59d8\") " pod="openshift-ingress/e2e-shared-b-w4-88da25-openshift-default-6c4ccc4cd9-tcsdd" Oct 30 21:06:28.587268 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:06:28.577106 5378 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/191dc733-cb36-4efc-9572-e645e2eb59d8-istio-data\") pod \"e2e-shared-b-w4-88da25-openshift-default-6c4ccc4cd9-tcsdd\" (UID: \"191dc733-cb36-4efc-9572-e645e2eb59d8\") " pod="openshift-ingress/e2e-shared-b-w4-88da25-openshift-default-6c4ccc4cd9-tcsdd" Oct 30 21:06:28.587553 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:06:28.579102 5378 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/191dc733-cb36-4efc-9572-e645e2eb59d8-workload-certs\") pod \"e2e-shared-b-w4-88da25-openshift-default-6c4ccc4cd9-tcsdd\" (UID: \"191dc733-cb36-4efc-9572-e645e2eb59d8\") " pod="openshift-ingress/e2e-shared-b-w4-88da25-openshift-default-6c4ccc4cd9-tcsdd" Oct 30 21:06:28.587749 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:06:28.577558 5378 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/191dc733-cb36-4efc-9572-e645e2eb59d8-credential-socket\") pod \"e2e-shared-b-w4-88da25-openshift-default-6c4ccc4cd9-tcsdd\" (UID: \"191dc733-cb36-4efc-9572-e645e2eb59d8\") " pod="openshift-ingress/e2e-shared-b-w4-88da25-openshift-default-6c4ccc4cd9-tcsdd" Oct 30 21:06:28.587865 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:06:28.580350 5378 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/191dc733-cb36-4efc-9572-e645e2eb59d8-istiod-ca-cert\") pod \"e2e-shared-b-w4-88da25-openshift-default-6c4ccc4cd9-tcsdd\" (UID: \"191dc733-cb36-4efc-9572-e645e2eb59d8\") " pod="openshift-ingress/e2e-shared-b-w4-88da25-openshift-default-6c4ccc4cd9-tcsdd" Oct 30 21:06:28.587960 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:06:28.578643 5378 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/191dc733-cb36-4efc-9572-e645e2eb59d8-workload-socket\") pod \"e2e-shared-b-w4-88da25-openshift-default-6c4ccc4cd9-tcsdd\" (UID: \"191dc733-cb36-4efc-9572-e645e2eb59d8\") " pod="openshift-ingress/e2e-shared-b-w4-88da25-openshift-default-6c4ccc4cd9-tcsdd" Oct 30 21:06:28.601619 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:06:28.591402 5378 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/191dc733-cb36-4efc-9572-e645e2eb59d8-istio-podinfo\") pod \"e2e-shared-b-w4-88da25-openshift-default-6c4ccc4cd9-tcsdd\" (UID: \"191dc733-cb36-4efc-9572-e645e2eb59d8\") " pod="openshift-ingress/e2e-shared-b-w4-88da25-openshift-default-6c4ccc4cd9-tcsdd" Oct 30 21:06:28.604296 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:06:28.602263 5378 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/191dc733-cb36-4efc-9572-e645e2eb59d8-istio-envoy\") pod \"e2e-shared-b-w4-88da25-openshift-default-6c4ccc4cd9-tcsdd\" (UID: \"191dc733-cb36-4efc-9572-e645e2eb59d8\") " pod="openshift-ingress/e2e-shared-b-w4-88da25-openshift-default-6c4ccc4cd9-tcsdd" Oct 30 21:06:28.634158 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:06:28.634124 5378 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/191dc733-cb36-4efc-9572-e645e2eb59d8-istio-token\") pod \"e2e-shared-b-w4-88da25-openshift-default-6c4ccc4cd9-tcsdd\" (UID: \"191dc733-cb36-4efc-9572-e645e2eb59d8\") " pod="openshift-ingress/e2e-shared-b-w4-88da25-openshift-default-6c4ccc4cd9-tcsdd" Oct 30 21:06:28.635378 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:06:28.635346 5378 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6n2gm\" (UniqueName: \"kubernetes.io/projected/191dc733-cb36-4efc-9572-e645e2eb59d8-kube-api-access-6n2gm\") pod \"e2e-shared-b-w4-88da25-openshift-default-6c4ccc4cd9-tcsdd\" (UID: \"191dc733-cb36-4efc-9572-e645e2eb59d8\") " pod="openshift-ingress/e2e-shared-b-w4-88da25-openshift-default-6c4ccc4cd9-tcsdd" Oct 30 21:06:28.817148 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:06:28.817064 5378 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w4-88da25-openshift-default-6c4ccc4cd9-tcsdd" Oct 30 21:06:29.118272 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:06:29.118138 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:06:29.290961 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:06:29.290930 5378 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-88da25-openshift-default-6c4ccc4cd9-tcsdd"] Oct 30 21:06:29.291691 ip-10-0-138-149 kubenswrapper[5378]: W1030 21:06:29.291658 5378 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod191dc733_cb36_4efc_9572_e645e2eb59d8.slice/crio-965a8bf649a8e1af9ad3a6bcb746a8efed57d36598d839e817767c74c016f274 WatchSource:0}: Error finding container 965a8bf649a8e1af9ad3a6bcb746a8efed57d36598d839e817767c74c016f274: Status 404 returned error can't find the container with id 965a8bf649a8e1af9ad3a6bcb746a8efed57d36598d839e817767c74c016f274 Oct 30 21:06:29.294852 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:06:29.294812 5378 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 21:06:29.294972 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:06:29.294899 5378 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 21:06:29.294972 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:06:29.294936 5378 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 21:06:29.523971 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:06:29.523936 5378 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-88da25-openshift-default-6c4ccc4cd9-tcsdd" event={"ID":"191dc733-cb36-4efc-9572-e645e2eb59d8","Type":"ContainerStarted","Data":"345ba2ed9062b536d084e4b4475ec52a5838cdb19de04968cd6e6b8f301aa257"} Oct 30 21:06:29.524156 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:06:29.523994 5378 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-88da25-openshift-default-6c4ccc4cd9-tcsdd" event={"ID":"191dc733-cb36-4efc-9572-e645e2eb59d8","Type":"ContainerStarted","Data":"965a8bf649a8e1af9ad3a6bcb746a8efed57d36598d839e817767c74c016f274"} Oct 30 21:06:29.651485 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:06:29.651379 5378 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w4-88da25-openshift-default-6c4ccc4cd9-tcsdd" podStartSLOduration=1.651360166 podStartE2EDuration="1.651360166s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:06:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:06:29.645624994 +0000 UTC m=+1680.073738913" watchObservedRunningTime="2026-10-30 21:06:29.651360166 +0000 UTC m=+1680.079474085" Oct 30 21:06:29.825423 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:06:29.818855 5378 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w4-88da25-openshift-default-6c4ccc4cd9-tcsdd" Oct 30 21:06:30.458057 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:06:30.458026 5378 scope.go:122] "RemoveContainer" containerID="4d310153550a3bb25aca95f21baacaec9e19fcd98b0b127345a928c8ff837039" Oct 30 21:06:30.468346 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:06:30.468324 5378 scope.go:122] "RemoveContainer" containerID="4eedd9f39fe7328f3b420ba428228ea8000bf66d9c46f94cee388b733ae7a7bf" Oct 30 21:06:30.477150 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:06:30.477133 5378 scope.go:122] "RemoveContainer" containerID="e636fc8115299f0916b8822ff29832fec0ea4a10f340562fe04ee4014ab985c5" Oct 30 21:06:30.486735 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:06:30.486717 5378 scope.go:122] "RemoveContainer" containerID="35d3fc32d46868470080fb357ea2e31ee7e46852a14a162472bd4d9cf2ac8cd9" Oct 30 21:06:30.497503 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:06:30.497486 5378 scope.go:122] "RemoveContainer" containerID="323ffdb133d1b6c43258e5efb100df0729756f618bb2c4b41c5468333b6ea6b7" Oct 30 21:06:30.509069 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:06:30.509049 5378 scope.go:122] "RemoveContainer" containerID="56a7dee9064f0f42b11e3587f69a27cdf9b83c617ffd30f92e0cd9ae9e312570" Oct 30 21:06:30.519635 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:06:30.519614 5378 scope.go:122] "RemoveContainer" containerID="d89f3cef996a81cab6f492854996217d11de5c7d1fe066f9f3e49f565bcfe1ab" Oct 30 21:06:30.823192 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:06:30.823124 5378 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w4-88da25-openshift-default-6c4ccc4cd9-tcsdd" Oct 30 21:06:31.093719 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:06:31.093642 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:06:31.576727 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:06:31.576265 5378 generic.go:356] "Generic (PLEG): container finished" podID="23f08320-e0a1-42bc-8fd1-de3492b59bca" containerID="565200d98f097d3eda2ed71305f48ca3f2478d53f968badf2cb34cdaa05da40a" exitCode=0 Oct 30 21:06:31.576727 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:06:31.576344 5378 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-5c45e5-openshift-default-9dcdd54d9-s697g" event={"ID":"23f08320-e0a1-42bc-8fd1-de3492b59bca","Type":"ContainerDied","Data":"565200d98f097d3eda2ed71305f48ca3f2478d53f968badf2cb34cdaa05da40a"} Oct 30 21:06:31.577312 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:06:31.577019 5378 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w4-88da25-openshift-default-6c4ccc4cd9-tcsdd" Oct 30 21:06:31.577379 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:06:31.577345 5378 scope.go:122] "RemoveContainer" containerID="565200d98f097d3eda2ed71305f48ca3f2478d53f968badf2cb34cdaa05da40a" Oct 30 21:06:31.578401 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:06:31.578021 5378 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w4-88da25-openshift-default-6c4ccc4cd9-tcsdd" Oct 30 21:06:31.579556 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:06:31.579153 5378 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 21:06:31.579556 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:06:31.579225 5378 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 21:06:31.579556 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:06:31.579255 5378 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 21:06:31.695588 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:06:31.694297 5378 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w6-dc6125-openshift-default-659bcc49bf-ms7zg"] Oct 30 21:06:31.802240 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:06:31.785963 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:06:31.947616 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:06:31.946590 5378 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w6-dc6125-openshift-default-659bcc49bf-ms7zg" Oct 30 21:06:32.041791 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:06:32.039158 5378 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w6-dc6125-openshift-default-dockercfg-8wlkx\"" Oct 30 21:06:32.041791 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:06:32.041368 5378 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d3278471-08a8-4d1e-8030-c6256d75265d-workload-certs\") pod \"e2e-shared-a-w6-dc6125-openshift-default-659bcc49bf-ms7zg\" (UID: \"d3278471-08a8-4d1e-8030-c6256d75265d\") " pod="openshift-ingress/e2e-shared-a-w6-dc6125-openshift-default-659bcc49bf-ms7zg" Oct 30 21:06:32.041791 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:06:32.041424 5378 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d3278471-08a8-4d1e-8030-c6256d75265d-workload-socket\") pod \"e2e-shared-a-w6-dc6125-openshift-default-659bcc49bf-ms7zg\" (UID: \"d3278471-08a8-4d1e-8030-c6256d75265d\") " pod="openshift-ingress/e2e-shared-a-w6-dc6125-openshift-default-659bcc49bf-ms7zg" Oct 30 21:06:32.041791 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:06:32.041513 5378 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d3278471-08a8-4d1e-8030-c6256d75265d-istio-envoy\") pod \"e2e-shared-a-w6-dc6125-openshift-default-659bcc49bf-ms7zg\" (UID: \"d3278471-08a8-4d1e-8030-c6256d75265d\") " pod="openshift-ingress/e2e-shared-a-w6-dc6125-openshift-default-659bcc49bf-ms7zg" Oct 30 21:06:32.082577 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:06:32.082477 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:06:32.090197 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:06:32.087528 5378 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d3278471-08a8-4d1e-8030-c6256d75265d-istiod-ca-cert\") pod \"e2e-shared-a-w6-dc6125-openshift-default-659bcc49bf-ms7zg\" (UID: \"d3278471-08a8-4d1e-8030-c6256d75265d\") " pod="openshift-ingress/e2e-shared-a-w6-dc6125-openshift-default-659bcc49bf-ms7zg" Oct 30 21:06:32.090197 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:06:32.087621 5378 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d3278471-08a8-4d1e-8030-c6256d75265d-istio-podinfo\") pod \"e2e-shared-a-w6-dc6125-openshift-default-659bcc49bf-ms7zg\" (UID: \"d3278471-08a8-4d1e-8030-c6256d75265d\") " pod="openshift-ingress/e2e-shared-a-w6-dc6125-openshift-default-659bcc49bf-ms7zg" Oct 30 21:06:32.090197 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:06:32.087692 5378 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d3278471-08a8-4d1e-8030-c6256d75265d-istio-token\") pod \"e2e-shared-a-w6-dc6125-openshift-default-659bcc49bf-ms7zg\" (UID: \"d3278471-08a8-4d1e-8030-c6256d75265d\") " pod="openshift-ingress/e2e-shared-a-w6-dc6125-openshift-default-659bcc49bf-ms7zg" Oct 30 21:06:32.090197 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:06:32.087831 5378 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d3278471-08a8-4d1e-8030-c6256d75265d-credential-socket\") pod \"e2e-shared-a-w6-dc6125-openshift-default-659bcc49bf-ms7zg\" (UID: \"d3278471-08a8-4d1e-8030-c6256d75265d\") " pod="openshift-ingress/e2e-shared-a-w6-dc6125-openshift-default-659bcc49bf-ms7zg" Oct 30 21:06:32.090197 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:06:32.087858 5378 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nm92g\" (UniqueName: \"kubernetes.io/projected/d3278471-08a8-4d1e-8030-c6256d75265d-kube-api-access-nm92g\") pod \"e2e-shared-a-w6-dc6125-openshift-default-659bcc49bf-ms7zg\" (UID: \"d3278471-08a8-4d1e-8030-c6256d75265d\") " pod="openshift-ingress/e2e-shared-a-w6-dc6125-openshift-default-659bcc49bf-ms7zg" Oct 30 21:06:32.090197 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:06:32.087908 5378 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d3278471-08a8-4d1e-8030-c6256d75265d-istio-data\") pod \"e2e-shared-a-w6-dc6125-openshift-default-659bcc49bf-ms7zg\" (UID: \"d3278471-08a8-4d1e-8030-c6256d75265d\") " pod="openshift-ingress/e2e-shared-a-w6-dc6125-openshift-default-659bcc49bf-ms7zg" Oct 30 21:06:32.248654 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:06:32.245385 5378 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d3278471-08a8-4d1e-8030-c6256d75265d-istio-envoy\") pod \"e2e-shared-a-w6-dc6125-openshift-default-659bcc49bf-ms7zg\" (UID: \"d3278471-08a8-4d1e-8030-c6256d75265d\") " pod="openshift-ingress/e2e-shared-a-w6-dc6125-openshift-default-659bcc49bf-ms7zg" Oct 30 21:06:32.248968 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:06:32.248941 5378 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d3278471-08a8-4d1e-8030-c6256d75265d-istiod-ca-cert\") pod \"e2e-shared-a-w6-dc6125-openshift-default-659bcc49bf-ms7zg\" (UID: \"d3278471-08a8-4d1e-8030-c6256d75265d\") " pod="openshift-ingress/e2e-shared-a-w6-dc6125-openshift-default-659bcc49bf-ms7zg" Oct 30 21:06:32.271826 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:06:32.271771 5378 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d3278471-08a8-4d1e-8030-c6256d75265d-istio-podinfo\") pod \"e2e-shared-a-w6-dc6125-openshift-default-659bcc49bf-ms7zg\" (UID: \"d3278471-08a8-4d1e-8030-c6256d75265d\") " pod="openshift-ingress/e2e-shared-a-w6-dc6125-openshift-default-659bcc49bf-ms7zg" Oct 30 21:06:32.272190 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:06:32.272167 5378 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d3278471-08a8-4d1e-8030-c6256d75265d-istio-token\") pod \"e2e-shared-a-w6-dc6125-openshift-default-659bcc49bf-ms7zg\" (UID: \"d3278471-08a8-4d1e-8030-c6256d75265d\") " pod="openshift-ingress/e2e-shared-a-w6-dc6125-openshift-default-659bcc49bf-ms7zg" Oct 30 21:06:32.272489 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:06:32.272450 5378 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d3278471-08a8-4d1e-8030-c6256d75265d-credential-socket\") pod \"e2e-shared-a-w6-dc6125-openshift-default-659bcc49bf-ms7zg\" (UID: \"d3278471-08a8-4d1e-8030-c6256d75265d\") " pod="openshift-ingress/e2e-shared-a-w6-dc6125-openshift-default-659bcc49bf-ms7zg" Oct 30 21:06:32.272710 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:06:32.272685 5378 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nm92g\" (UniqueName: \"kubernetes.io/projected/d3278471-08a8-4d1e-8030-c6256d75265d-kube-api-access-nm92g\") pod \"e2e-shared-a-w6-dc6125-openshift-default-659bcc49bf-ms7zg\" (UID: \"d3278471-08a8-4d1e-8030-c6256d75265d\") " pod="openshift-ingress/e2e-shared-a-w6-dc6125-openshift-default-659bcc49bf-ms7zg" Oct 30 21:06:32.274379 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:06:32.274357 5378 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d3278471-08a8-4d1e-8030-c6256d75265d-istio-data\") pod \"e2e-shared-a-w6-dc6125-openshift-default-659bcc49bf-ms7zg\" (UID: \"d3278471-08a8-4d1e-8030-c6256d75265d\") " pod="openshift-ingress/e2e-shared-a-w6-dc6125-openshift-default-659bcc49bf-ms7zg" Oct 30 21:06:32.274836 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:06:32.274812 5378 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d3278471-08a8-4d1e-8030-c6256d75265d-workload-certs\") pod \"e2e-shared-a-w6-dc6125-openshift-default-659bcc49bf-ms7zg\" (UID: \"d3278471-08a8-4d1e-8030-c6256d75265d\") " pod="openshift-ingress/e2e-shared-a-w6-dc6125-openshift-default-659bcc49bf-ms7zg" Oct 30 21:06:32.275164 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:06:32.275144 5378 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d3278471-08a8-4d1e-8030-c6256d75265d-workload-socket\") pod \"e2e-shared-a-w6-dc6125-openshift-default-659bcc49bf-ms7zg\" (UID: \"d3278471-08a8-4d1e-8030-c6256d75265d\") " pod="openshift-ingress/e2e-shared-a-w6-dc6125-openshift-default-659bcc49bf-ms7zg" Oct 30 21:06:32.275399 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:06:32.267679 5378 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w6-dc6125-openshift-default-659bcc49bf-ms7zg"] Oct 30 21:06:32.278134 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:06:32.261210 5378 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d3278471-08a8-4d1e-8030-c6256d75265d-istiod-ca-cert\") pod \"e2e-shared-a-w6-dc6125-openshift-default-659bcc49bf-ms7zg\" (UID: \"d3278471-08a8-4d1e-8030-c6256d75265d\") " pod="openshift-ingress/e2e-shared-a-w6-dc6125-openshift-default-659bcc49bf-ms7zg" Oct 30 21:06:32.278296 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:06:32.275895 5378 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d3278471-08a8-4d1e-8030-c6256d75265d-istio-data\") pod \"e2e-shared-a-w6-dc6125-openshift-default-659bcc49bf-ms7zg\" (UID: \"d3278471-08a8-4d1e-8030-c6256d75265d\") " pod="openshift-ingress/e2e-shared-a-w6-dc6125-openshift-default-659bcc49bf-ms7zg" Oct 30 21:06:32.278410 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:06:32.277169 5378 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d3278471-08a8-4d1e-8030-c6256d75265d-workload-certs\") pod \"e2e-shared-a-w6-dc6125-openshift-default-659bcc49bf-ms7zg\" (UID: \"d3278471-08a8-4d1e-8030-c6256d75265d\") " pod="openshift-ingress/e2e-shared-a-w6-dc6125-openshift-default-659bcc49bf-ms7zg" Oct 30 21:06:32.278509 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:06:32.277971 5378 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d3278471-08a8-4d1e-8030-c6256d75265d-credential-socket\") pod \"e2e-shared-a-w6-dc6125-openshift-default-659bcc49bf-ms7zg\" (UID: \"d3278471-08a8-4d1e-8030-c6256d75265d\") " pod="openshift-ingress/e2e-shared-a-w6-dc6125-openshift-default-659bcc49bf-ms7zg" Oct 30 21:06:32.279658 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:06:32.279636 5378 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d3278471-08a8-4d1e-8030-c6256d75265d-workload-socket\") pod \"e2e-shared-a-w6-dc6125-openshift-default-659bcc49bf-ms7zg\" (UID: \"d3278471-08a8-4d1e-8030-c6256d75265d\") " pod="openshift-ingress/e2e-shared-a-w6-dc6125-openshift-default-659bcc49bf-ms7zg" Oct 30 21:06:32.281547 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:06:32.281527 5378 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d3278471-08a8-4d1e-8030-c6256d75265d-istio-envoy\") pod \"e2e-shared-a-w6-dc6125-openshift-default-659bcc49bf-ms7zg\" (UID: \"d3278471-08a8-4d1e-8030-c6256d75265d\") " pod="openshift-ingress/e2e-shared-a-w6-dc6125-openshift-default-659bcc49bf-ms7zg" Oct 30 21:06:32.303467 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:06:32.303435 5378 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d3278471-08a8-4d1e-8030-c6256d75265d-istio-podinfo\") pod \"e2e-shared-a-w6-dc6125-openshift-default-659bcc49bf-ms7zg\" (UID: \"d3278471-08a8-4d1e-8030-c6256d75265d\") " pod="openshift-ingress/e2e-shared-a-w6-dc6125-openshift-default-659bcc49bf-ms7zg" Oct 30 21:06:32.334000 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:06:32.333931 5378 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d3278471-08a8-4d1e-8030-c6256d75265d-istio-token\") pod \"e2e-shared-a-w6-dc6125-openshift-default-659bcc49bf-ms7zg\" (UID: \"d3278471-08a8-4d1e-8030-c6256d75265d\") " pod="openshift-ingress/e2e-shared-a-w6-dc6125-openshift-default-659bcc49bf-ms7zg" Oct 30 21:06:32.334168 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:06:32.334145 5378 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nm92g\" (UniqueName: \"kubernetes.io/projected/d3278471-08a8-4d1e-8030-c6256d75265d-kube-api-access-nm92g\") pod \"e2e-shared-a-w6-dc6125-openshift-default-659bcc49bf-ms7zg\" (UID: \"d3278471-08a8-4d1e-8030-c6256d75265d\") " pod="openshift-ingress/e2e-shared-a-w6-dc6125-openshift-default-659bcc49bf-ms7zg" Oct 30 21:06:32.387306 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:06:32.387255 5378 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-2cf21005-7d898b7f7b-l7vts"] Oct 30 21:06:32.387543 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:06:32.387517 5378 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-2cf21005-7d898b7f7b-l7vts" podUID="e132414d-c8ea-4b13-a8d6-639bfe64effe" containerName="maas-api" containerID="cri-o://f67c442517d5b30da76988ff53f6b38fece6aea343fdf703ea62e9787f31e1ad" gracePeriod=30 Oct 30 21:06:32.430065 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:06:32.429813 5378 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w6-dc6125-openshift-default-659bcc49bf-ms7zg" Oct 30 21:06:32.478894 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:06:32.478838 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:06:32.683116 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:06:32.679407 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:06:32.715621 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:06:32.709306 5378 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-5c45e5-openshift-default-9dcdd54d9-s697g" event={"ID":"23f08320-e0a1-42bc-8fd1-de3492b59bca","Type":"ContainerStarted","Data":"3411ad24d40a29ed63e18fa10cf40588cbb83130efc1cf2232c5acd10a3a0303"} Oct 30 21:06:32.715621 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:06:32.712744 5378 generic.go:356] "Generic (PLEG): container finished" podID="e132414d-c8ea-4b13-a8d6-639bfe64effe" containerID="f67c442517d5b30da76988ff53f6b38fece6aea343fdf703ea62e9787f31e1ad" exitCode=0 Oct 30 21:06:32.715621 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:06:32.713704 5378 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-2cf21005-7d898b7f7b-l7vts" event={"ID":"e132414d-c8ea-4b13-a8d6-639bfe64effe","Type":"ContainerDied","Data":"f67c442517d5b30da76988ff53f6b38fece6aea343fdf703ea62e9787f31e1ad"} Oct 30 21:06:32.840256 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:06:32.840227 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:06:32.895219 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:06:32.895192 5378 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-2cf21005-7d898b7f7b-l7vts" Oct 30 21:06:32.896287 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:06:32.896263 5378 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w6-dc6125-openshift-default-659bcc49bf-ms7zg"] Oct 30 21:06:32.905112 ip-10-0-138-149 kubenswrapper[5378]: W1030 21:06:32.905086 5378 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd3278471_08a8_4d1e_8030_c6256d75265d.slice/crio-c1cc2cdbb15e6bbf444201e3f80e48d92e9e2577b02b79f902acfd29b1a7c9fc WatchSource:0}: Error finding container c1cc2cdbb15e6bbf444201e3f80e48d92e9e2577b02b79f902acfd29b1a7c9fc: Status 404 returned error can't find the container with id c1cc2cdbb15e6bbf444201e3f80e48d92e9e2577b02b79f902acfd29b1a7c9fc Oct 30 21:06:32.907862 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:06:32.907829 5378 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 21:06:32.907969 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:06:32.907911 5378 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 21:06:32.907969 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:06:32.907960 5378 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 21:06:32.986601 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:06:32.981907 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:06:33.022595 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:06:33.019042 5378 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e132414d-c8ea-4b13-a8d6-639bfe64effe-kube-api-access-w6w9w\" (UniqueName: \"kubernetes.io/projected/e132414d-c8ea-4b13-a8d6-639bfe64effe-kube-api-access-w6w9w\") pod \"e132414d-c8ea-4b13-a8d6-639bfe64effe\" (UID: \"e132414d-c8ea-4b13-a8d6-639bfe64effe\") " Oct 30 21:06:33.022595 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:06:33.019269 5378 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e132414d-c8ea-4b13-a8d6-639bfe64effe-metrics-certs\" (UniqueName: \"kubernetes.io/secret/e132414d-c8ea-4b13-a8d6-639bfe64effe-metrics-certs\") pod \"e132414d-c8ea-4b13-a8d6-639bfe64effe\" (UID: \"e132414d-c8ea-4b13-a8d6-639bfe64effe\") " Oct 30 21:06:33.022595 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:06:33.019311 5378 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e132414d-c8ea-4b13-a8d6-639bfe64effe-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e132414d-c8ea-4b13-a8d6-639bfe64effe-maas-api-tls\") pod \"e132414d-c8ea-4b13-a8d6-639bfe64effe\" (UID: \"e132414d-c8ea-4b13-a8d6-639bfe64effe\") " Oct 30 21:06:33.022595 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:06:33.019473 5378 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e132414d-c8ea-4b13-a8d6-639bfe64effe-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e132414d-c8ea-4b13-a8d6-639bfe64effe-openshift-service-ca\") pod \"e132414d-c8ea-4b13-a8d6-639bfe64effe\" (UID: \"e132414d-c8ea-4b13-a8d6-639bfe64effe\") " Oct 30 21:06:33.048828 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:06:33.043338 5378 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e132414d-c8ea-4b13-a8d6-639bfe64effe-openshift-service-ca" pod "e132414d-c8ea-4b13-a8d6-639bfe64effe" (UID: "e132414d-c8ea-4b13-a8d6-639bfe64effe"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:06:33.085794 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:06:33.085751 5378 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e132414d-c8ea-4b13-a8d6-639bfe64effe-kube-api-access-w6w9w" pod "e132414d-c8ea-4b13-a8d6-639bfe64effe" (UID: "e132414d-c8ea-4b13-a8d6-639bfe64effe"). InnerVolumeSpecName "kube-api-access-w6w9w". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:06:33.117069 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:06:33.116994 5378 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e132414d-c8ea-4b13-a8d6-639bfe64effe-metrics-certs" pod "e132414d-c8ea-4b13-a8d6-639bfe64effe" (UID: "e132414d-c8ea-4b13-a8d6-639bfe64effe"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:06:33.123671 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:06:33.123637 5378 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e132414d-c8ea-4b13-a8d6-639bfe64effe-metrics-certs\") on node \"ip-10-0-138-149.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:06:33.124085 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:06:33.124015 5378 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e132414d-c8ea-4b13-a8d6-639bfe64effe-openshift-service-ca\") on node \"ip-10-0-138-149.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:06:33.124227 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:06:33.124211 5378 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w6w9w\" (UniqueName: \"kubernetes.io/projected/e132414d-c8ea-4b13-a8d6-639bfe64effe-kube-api-access-w6w9w\") on node \"ip-10-0-138-149.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:06:33.141552 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:06:33.125280 5378 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e132414d-c8ea-4b13-a8d6-639bfe64effe-maas-api-tls" pod "e132414d-c8ea-4b13-a8d6-639bfe64effe" (UID: "e132414d-c8ea-4b13-a8d6-639bfe64effe"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:06:33.230680 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:06:33.229426 5378 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e132414d-c8ea-4b13-a8d6-639bfe64effe-maas-api-tls\") on node \"ip-10-0-138-149.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:06:33.472586 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:06:33.467311 5378 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w4-5c45e5-openshift-default-9dcdd54d9-s697g" Oct 30 21:06:33.472586 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:06:33.468391 5378 patch_prober.go:29] interesting pod/e2e-shared-a-w4-5c45e5-openshift-default-9dcdd54d9-s697g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.109:15021/healthz/ready\": dial tcp 10.133.0.109:15021: connect: connection refused" start-of-body= Oct 30 21:06:33.472586 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:06:33.468469 5378 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-5c45e5-openshift-default-9dcdd54d9-s697g" podUID="23f08320-e0a1-42bc-8fd1-de3492b59bca" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.109:15021/healthz/ready\": dial tcp 10.133.0.109:15021: connect: connection refused" Oct 30 21:06:33.724135 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:06:33.724094 5378 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w6-dc6125-openshift-default-659bcc49bf-ms7zg" event={"ID":"d3278471-08a8-4d1e-8030-c6256d75265d","Type":"ContainerStarted","Data":"b66cc3f1cfa154e91f53a673aaac48f9fcd49cde853157e43d7fc96cae4a8cf5"} Oct 30 21:06:33.724642 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:06:33.724176 5378 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w6-dc6125-openshift-default-659bcc49bf-ms7zg" event={"ID":"d3278471-08a8-4d1e-8030-c6256d75265d","Type":"ContainerStarted","Data":"c1cc2cdbb15e6bbf444201e3f80e48d92e9e2577b02b79f902acfd29b1a7c9fc"} Oct 30 21:06:33.725765 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:06:33.725742 5378 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-2cf21005-7d898b7f7b-l7vts" Oct 30 21:06:33.725857 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:06:33.725772 5378 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-2cf21005-7d898b7f7b-l7vts" event={"ID":"e132414d-c8ea-4b13-a8d6-639bfe64effe","Type":"ContainerDied","Data":"34478f3ae8c93947cd353445469cea698d0a9925576c1e23c41fb1596a72880f"} Oct 30 21:06:33.725857 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:06:33.725827 5378 scope.go:122] "RemoveContainer" containerID="f67c442517d5b30da76988ff53f6b38fece6aea343fdf703ea62e9787f31e1ad" Oct 30 21:06:33.828809 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:06:33.828717 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:06:33.905766 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:06:33.905697 5378 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w6-dc6125-openshift-default-659bcc49bf-ms7zg" podStartSLOduration=2.90567817 podStartE2EDuration="2.90567817s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:06:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:06:33.878751175 +0000 UTC m=+1684.306865095" watchObservedRunningTime="2026-10-30 21:06:33.90567817 +0000 UTC m=+1684.333792091" Oct 30 21:06:33.994843 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:06:33.994808 5378 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-2cf21005-7d898b7f7b-l7vts"] Oct 30 21:06:33.998786 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:06:33.998755 5378 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-2cf21005-7d898b7f7b-l7vts"] Oct 30 21:06:34.345588 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:06:34.345114 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:06:34.430489 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:06:34.430452 5378 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w6-dc6125-openshift-default-659bcc49bf-ms7zg" Oct 30 21:06:34.437752 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:06:34.436546 5378 patch_prober.go:29] interesting pod/e2e-shared-a-w4-5c45e5-openshift-default-9dcdd54d9-s697g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.109:15021/healthz/ready\": dial tcp 10.133.0.109:15021: connect: connection refused" start-of-body= Oct 30 21:06:34.437752 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:06:34.436642 5378 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-5c45e5-openshift-default-9dcdd54d9-s697g" podUID="23f08320-e0a1-42bc-8fd1-de3492b59bca" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.109:15021/healthz/ready\": dial tcp 10.133.0.109:15021: connect: connection refused" Oct 30 21:06:34.441333 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:06:34.441301 5378 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w6-dc6125-openshift-default-659bcc49bf-ms7zg" Oct 30 21:06:34.578766 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:06:34.578720 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:06:34.734706 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:06:34.734623 5378 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w6-dc6125-openshift-default-659bcc49bf-ms7zg" Oct 30 21:06:34.735841 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:06:34.735823 5378 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w6-dc6125-openshift-default-659bcc49bf-ms7zg" Oct 30 21:06:35.397916 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:06:35.397883 5378 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-2cf21005-openshift-default-5fdcf57455-c7jwb"] Oct 30 21:06:35.398392 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:06:35.398360 5378 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-2cf21005-openshift-default-5fdcf57455-c7jwb" podUID="c7035375-d666-498d-b9e7-f699e1e86c99" containerName="istio-proxy" containerID="cri-o://51c283842e2cefe34949229234561d3e96098541d98c1f3de02529a6df501442" gracePeriod=30 Oct 30 21:06:35.402360 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:06:35.402333 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:06:35.436491 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:06:35.436463 5378 patch_prober.go:29] interesting pod/e2e-shared-a-w4-5c45e5-openshift-default-9dcdd54d9-s697g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.109:15021/healthz/ready\": dial tcp 10.133.0.109:15021: connect: connection refused" start-of-body= Oct 30 21:06:35.436641 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:06:35.436518 5378 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-5c45e5-openshift-default-9dcdd54d9-s697g" podUID="23f08320-e0a1-42bc-8fd1-de3492b59bca" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.109:15021/healthz/ready\": dial tcp 10.133.0.109:15021: connect: connection refused" Oct 30 21:06:35.789677 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:06:35.789590 5378 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e132414d-c8ea-4b13-a8d6-639bfe64effe" path="/var/lib/kubelet/pods/e132414d-c8ea-4b13-a8d6-639bfe64effe/volumes" Oct 30 21:06:36.152663 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:06:36.152631 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:06:36.284601 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:06:36.283160 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:06:36.437107 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:06:36.437025 5378 patch_prober.go:29] interesting pod/e2e-shared-a-w4-5c45e5-openshift-default-9dcdd54d9-s697g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.109:15021/healthz/ready\": dial tcp 10.133.0.109:15021: connect: connection refused" start-of-body= Oct 30 21:06:36.437107 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:06:36.437074 5378 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-5c45e5-openshift-default-9dcdd54d9-s697g" podUID="23f08320-e0a1-42bc-8fd1-de3492b59bca" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.109:15021/healthz/ready\": dial tcp 10.133.0.109:15021: connect: connection refused" Oct 30 21:06:36.740435 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:06:36.740364 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:06:37.065633 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:06:37.044310 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:06:37.444593 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:06:37.441814 5378 patch_prober.go:29] interesting pod/e2e-shared-a-w4-5c45e5-openshift-default-9dcdd54d9-s697g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.109:15021/healthz/ready\": dial tcp 10.133.0.109:15021: connect: connection refused" start-of-body= Oct 30 21:06:37.444593 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:06:37.441892 5378 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-5c45e5-openshift-default-9dcdd54d9-s697g" podUID="23f08320-e0a1-42bc-8fd1-de3492b59bca" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.109:15021/healthz/ready\": dial tcp 10.133.0.109:15021: connect: connection refused" Oct 30 21:06:37.453596 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:06:37.452806 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:06:37.647819 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:06:37.647750 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:06:37.648650 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:06:37.648618 5378 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-derive-bf9c0727-openshift-default-7fb479dd89-jzzdk"] Oct 30 21:06:37.649652 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:06:37.649345 5378 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-derive-bf9c0727-openshift-default-7fb479dd89-jzzdk" podUID="eafa34dc-6d4b-4507-a2a3-989ad8cf7de2" containerName="istio-proxy" containerID="cri-o://36a0351f1605358413b4ed86dc714b151758b0961124deaed1360602b7804c2a" gracePeriod=30 Oct 30 21:06:37.656593 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:06:37.652598 5378 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-derive-bf9c0727-openshift-default-7fb479dd89-lrsjm"] Oct 30 21:06:37.656593 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:06:37.652810 5378 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-derive-bf9c0727-openshift-default-7fb479dd89-lrsjm" podUID="88bbd407-197c-4ba6-b31f-fc097ec5fbf3" containerName="istio-proxy" containerID="cri-o://434ca2e3b8666ac602b32ac4097c0bb6d841dcdeac21e53e8c262b96fa49f606" gracePeriod=30 Oct 30 21:06:37.693225 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:06:37.693191 5378 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w1-4d1110-openshift-default-6576975b6c-gvzcm"] Oct 30 21:06:37.694122 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:06:37.694099 5378 cpu_manager.go:414] "containerMap: removing container" podUID="e132414d-c8ea-4b13-a8d6-639bfe64effe" containerName="maas-api" Oct 30 21:06:37.694122 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:06:37.694119 5378 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e132414d-c8ea-4b13-a8d6-639bfe64effe" containerName="maas-api" Oct 30 21:06:37.694284 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:06:37.694273 5378 memory_manager.go:365] "RemoveStaleState removing state" podUID="e132414d-c8ea-4b13-a8d6-639bfe64effe" containerName="maas-api" Oct 30 21:06:37.697687 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:06:37.697645 5378 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w1-4d1110-openshift-default-6576975b6c-gvzcm" Oct 30 21:06:37.745282 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:06:37.745252 5378 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w4-88da25-5ddcdff7f9-5p4dg"] Oct 30 21:06:37.746764 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:06:37.746721 5378 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vgfxz\" (UniqueName: \"kubernetes.io/projected/1b385f61-b9d9-4f5d-8e71-7d86cd0d867d-kube-api-access-vgfxz\") pod \"e2e-shared-a-w1-4d1110-openshift-default-6576975b6c-gvzcm\" (UID: \"1b385f61-b9d9-4f5d-8e71-7d86cd0d867d\") " pod="openshift-ingress/e2e-shared-a-w1-4d1110-openshift-default-6576975b6c-gvzcm" Oct 30 21:06:37.746918 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:06:37.746803 5378 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1b385f61-b9d9-4f5d-8e71-7d86cd0d867d-workload-certs\") pod \"e2e-shared-a-w1-4d1110-openshift-default-6576975b6c-gvzcm\" (UID: \"1b385f61-b9d9-4f5d-8e71-7d86cd0d867d\") " pod="openshift-ingress/e2e-shared-a-w1-4d1110-openshift-default-6576975b6c-gvzcm" Oct 30 21:06:37.746918 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:06:37.746832 5378 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1b385f61-b9d9-4f5d-8e71-7d86cd0d867d-istio-envoy\") pod \"e2e-shared-a-w1-4d1110-openshift-default-6576975b6c-gvzcm\" (UID: \"1b385f61-b9d9-4f5d-8e71-7d86cd0d867d\") " pod="openshift-ingress/e2e-shared-a-w1-4d1110-openshift-default-6576975b6c-gvzcm" Oct 30 21:06:37.746918 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:06:37.746885 5378 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1b385f61-b9d9-4f5d-8e71-7d86cd0d867d-istio-token\") pod \"e2e-shared-a-w1-4d1110-openshift-default-6576975b6c-gvzcm\" (UID: \"1b385f61-b9d9-4f5d-8e71-7d86cd0d867d\") " pod="openshift-ingress/e2e-shared-a-w1-4d1110-openshift-default-6576975b6c-gvzcm" Oct 30 21:06:37.747107 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:06:37.746939 5378 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1b385f61-b9d9-4f5d-8e71-7d86cd0d867d-credential-socket\") pod \"e2e-shared-a-w1-4d1110-openshift-default-6576975b6c-gvzcm\" (UID: \"1b385f61-b9d9-4f5d-8e71-7d86cd0d867d\") " pod="openshift-ingress/e2e-shared-a-w1-4d1110-openshift-default-6576975b6c-gvzcm" Oct 30 21:06:37.747107 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:06:37.746979 5378 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1b385f61-b9d9-4f5d-8e71-7d86cd0d867d-istio-podinfo\") pod \"e2e-shared-a-w1-4d1110-openshift-default-6576975b6c-gvzcm\" (UID: \"1b385f61-b9d9-4f5d-8e71-7d86cd0d867d\") " pod="openshift-ingress/e2e-shared-a-w1-4d1110-openshift-default-6576975b6c-gvzcm" Oct 30 21:06:37.747107 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:06:37.747045 5378 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1b385f61-b9d9-4f5d-8e71-7d86cd0d867d-istiod-ca-cert\") pod \"e2e-shared-a-w1-4d1110-openshift-default-6576975b6c-gvzcm\" (UID: \"1b385f61-b9d9-4f5d-8e71-7d86cd0d867d\") " pod="openshift-ingress/e2e-shared-a-w1-4d1110-openshift-default-6576975b6c-gvzcm" Oct 30 21:06:37.747107 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:06:37.747098 5378 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1b385f61-b9d9-4f5d-8e71-7d86cd0d867d-workload-socket\") pod \"e2e-shared-a-w1-4d1110-openshift-default-6576975b6c-gvzcm\" (UID: \"1b385f61-b9d9-4f5d-8e71-7d86cd0d867d\") " pod="openshift-ingress/e2e-shared-a-w1-4d1110-openshift-default-6576975b6c-gvzcm" Oct 30 21:06:37.747311 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:06:37.747193 5378 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1b385f61-b9d9-4f5d-8e71-7d86cd0d867d-istio-data\") pod \"e2e-shared-a-w1-4d1110-openshift-default-6576975b6c-gvzcm\" (UID: \"1b385f61-b9d9-4f5d-8e71-7d86cd0d867d\") " pod="openshift-ingress/e2e-shared-a-w1-4d1110-openshift-default-6576975b6c-gvzcm" Oct 30 21:06:37.751155 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:06:37.751128 5378 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-88da25-5ddcdff7f9-5p4dg" Oct 30 21:06:37.796728 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:06:37.795040 5378 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w1-4d1110-openshift-default-dockercfg-wcp4l\"" Oct 30 21:06:37.807772 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:06:37.807643 5378 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-4d1110-openshift-default-6576975b6c-gvzcm"] Oct 30 21:06:37.819623 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:06:37.814984 5378 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w4-88da25-5ddcdff7f9-5p4dg"] Oct 30 21:06:37.819623 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:06:37.816691 5378 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w4-88da25\"" Oct 30 21:06:37.819623 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:06:37.817885 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:06:37.848611 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:06:37.848177 5378 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vgfxz\" (UniqueName: \"kubernetes.io/projected/1b385f61-b9d9-4f5d-8e71-7d86cd0d867d-kube-api-access-vgfxz\") pod \"e2e-shared-a-w1-4d1110-openshift-default-6576975b6c-gvzcm\" (UID: \"1b385f61-b9d9-4f5d-8e71-7d86cd0d867d\") " pod="openshift-ingress/e2e-shared-a-w1-4d1110-openshift-default-6576975b6c-gvzcm" Oct 30 21:06:37.848611 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:06:37.848252 5378 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1b385f61-b9d9-4f5d-8e71-7d86cd0d867d-workload-certs\") pod \"e2e-shared-a-w1-4d1110-openshift-default-6576975b6c-gvzcm\" (UID: \"1b385f61-b9d9-4f5d-8e71-7d86cd0d867d\") " pod="openshift-ingress/e2e-shared-a-w1-4d1110-openshift-default-6576975b6c-gvzcm" Oct 30 21:06:37.848611 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:06:37.848278 5378 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1b385f61-b9d9-4f5d-8e71-7d86cd0d867d-istio-envoy\") pod \"e2e-shared-a-w1-4d1110-openshift-default-6576975b6c-gvzcm\" (UID: \"1b385f61-b9d9-4f5d-8e71-7d86cd0d867d\") " pod="openshift-ingress/e2e-shared-a-w1-4d1110-openshift-default-6576975b6c-gvzcm" Oct 30 21:06:37.848611 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:06:37.848332 5378 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c537a2c1-13ee-488d-98b8-e8195b60d2be-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w4-88da25-5ddcdff7f9-5p4dg\" (UID: \"c537a2c1-13ee-488d-98b8-e8195b60d2be\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-88da25-5ddcdff7f9-5p4dg" Oct 30 21:06:37.848611 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:06:37.848384 5378 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1b385f61-b9d9-4f5d-8e71-7d86cd0d867d-istio-token\") pod \"e2e-shared-a-w1-4d1110-openshift-default-6576975b6c-gvzcm\" (UID: \"1b385f61-b9d9-4f5d-8e71-7d86cd0d867d\") " pod="openshift-ingress/e2e-shared-a-w1-4d1110-openshift-default-6576975b6c-gvzcm" Oct 30 21:06:37.848611 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:06:37.848408 5378 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wdq75\" (UniqueName: \"kubernetes.io/projected/c537a2c1-13ee-488d-98b8-e8195b60d2be-kube-api-access-wdq75\") pod \"payload-pre-processing-e2e-shared-b-w4-88da25-5ddcdff7f9-5p4dg\" (UID: \"c537a2c1-13ee-488d-98b8-e8195b60d2be\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-88da25-5ddcdff7f9-5p4dg" Oct 30 21:06:37.848611 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:06:37.848451 5378 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1b385f61-b9d9-4f5d-8e71-7d86cd0d867d-credential-socket\") pod \"e2e-shared-a-w1-4d1110-openshift-default-6576975b6c-gvzcm\" (UID: \"1b385f61-b9d9-4f5d-8e71-7d86cd0d867d\") " pod="openshift-ingress/e2e-shared-a-w1-4d1110-openshift-default-6576975b6c-gvzcm" Oct 30 21:06:37.848611 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:06:37.848492 5378 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1b385f61-b9d9-4f5d-8e71-7d86cd0d867d-istio-podinfo\") pod \"e2e-shared-a-w1-4d1110-openshift-default-6576975b6c-gvzcm\" (UID: \"1b385f61-b9d9-4f5d-8e71-7d86cd0d867d\") " pod="openshift-ingress/e2e-shared-a-w1-4d1110-openshift-default-6576975b6c-gvzcm" Oct 30 21:06:37.850117 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:06:37.849150 5378 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1b385f61-b9d9-4f5d-8e71-7d86cd0d867d-istiod-ca-cert\") pod \"e2e-shared-a-w1-4d1110-openshift-default-6576975b6c-gvzcm\" (UID: \"1b385f61-b9d9-4f5d-8e71-7d86cd0d867d\") " pod="openshift-ingress/e2e-shared-a-w1-4d1110-openshift-default-6576975b6c-gvzcm" Oct 30 21:06:37.850117 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:06:37.849216 5378 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1b385f61-b9d9-4f5d-8e71-7d86cd0d867d-workload-socket\") pod \"e2e-shared-a-w1-4d1110-openshift-default-6576975b6c-gvzcm\" (UID: \"1b385f61-b9d9-4f5d-8e71-7d86cd0d867d\") " pod="openshift-ingress/e2e-shared-a-w1-4d1110-openshift-default-6576975b6c-gvzcm" Oct 30 21:06:37.850117 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:06:37.849306 5378 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1b385f61-b9d9-4f5d-8e71-7d86cd0d867d-istio-data\") pod \"e2e-shared-a-w1-4d1110-openshift-default-6576975b6c-gvzcm\" (UID: \"1b385f61-b9d9-4f5d-8e71-7d86cd0d867d\") " pod="openshift-ingress/e2e-shared-a-w1-4d1110-openshift-default-6576975b6c-gvzcm" Oct 30 21:06:37.850117 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:06:37.849799 5378 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1b385f61-b9d9-4f5d-8e71-7d86cd0d867d-istio-data\") pod \"e2e-shared-a-w1-4d1110-openshift-default-6576975b6c-gvzcm\" (UID: \"1b385f61-b9d9-4f5d-8e71-7d86cd0d867d\") " pod="openshift-ingress/e2e-shared-a-w1-4d1110-openshift-default-6576975b6c-gvzcm" Oct 30 21:06:37.851064 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:06:37.850768 5378 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1b385f61-b9d9-4f5d-8e71-7d86cd0d867d-workload-certs\") pod \"e2e-shared-a-w1-4d1110-openshift-default-6576975b6c-gvzcm\" (UID: \"1b385f61-b9d9-4f5d-8e71-7d86cd0d867d\") " pod="openshift-ingress/e2e-shared-a-w1-4d1110-openshift-default-6576975b6c-gvzcm" Oct 30 21:06:37.852453 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:06:37.852412 5378 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1b385f61-b9d9-4f5d-8e71-7d86cd0d867d-istiod-ca-cert\") pod \"e2e-shared-a-w1-4d1110-openshift-default-6576975b6c-gvzcm\" (UID: \"1b385f61-b9d9-4f5d-8e71-7d86cd0d867d\") " pod="openshift-ingress/e2e-shared-a-w1-4d1110-openshift-default-6576975b6c-gvzcm" Oct 30 21:06:37.856608 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:06:37.853076 5378 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1b385f61-b9d9-4f5d-8e71-7d86cd0d867d-workload-socket\") pod \"e2e-shared-a-w1-4d1110-openshift-default-6576975b6c-gvzcm\" (UID: \"1b385f61-b9d9-4f5d-8e71-7d86cd0d867d\") " pod="openshift-ingress/e2e-shared-a-w1-4d1110-openshift-default-6576975b6c-gvzcm" Oct 30 21:06:37.856608 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:06:37.853103 5378 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1b385f61-b9d9-4f5d-8e71-7d86cd0d867d-credential-socket\") pod \"e2e-shared-a-w1-4d1110-openshift-default-6576975b6c-gvzcm\" (UID: \"1b385f61-b9d9-4f5d-8e71-7d86cd0d867d\") " pod="openshift-ingress/e2e-shared-a-w1-4d1110-openshift-default-6576975b6c-gvzcm" Oct 30 21:06:37.856608 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:06:37.853323 5378 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w4-88da25-dockercfg-khgc8\"" Oct 30 21:06:37.878945 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:06:37.878868 5378 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1b385f61-b9d9-4f5d-8e71-7d86cd0d867d-istio-envoy\") pod \"e2e-shared-a-w1-4d1110-openshift-default-6576975b6c-gvzcm\" (UID: \"1b385f61-b9d9-4f5d-8e71-7d86cd0d867d\") " pod="openshift-ingress/e2e-shared-a-w1-4d1110-openshift-default-6576975b6c-gvzcm" Oct 30 21:06:37.933584 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:06:37.918112 5378 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1b385f61-b9d9-4f5d-8e71-7d86cd0d867d-istio-podinfo\") pod \"e2e-shared-a-w1-4d1110-openshift-default-6576975b6c-gvzcm\" (UID: \"1b385f61-b9d9-4f5d-8e71-7d86cd0d867d\") " pod="openshift-ingress/e2e-shared-a-w1-4d1110-openshift-default-6576975b6c-gvzcm" Oct 30 21:06:37.938485 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:06:37.938451 5378 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vgfxz\" (UniqueName: \"kubernetes.io/projected/1b385f61-b9d9-4f5d-8e71-7d86cd0d867d-kube-api-access-vgfxz\") pod \"e2e-shared-a-w1-4d1110-openshift-default-6576975b6c-gvzcm\" (UID: \"1b385f61-b9d9-4f5d-8e71-7d86cd0d867d\") " pod="openshift-ingress/e2e-shared-a-w1-4d1110-openshift-default-6576975b6c-gvzcm" Oct 30 21:06:37.942219 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:06:37.942157 5378 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1b385f61-b9d9-4f5d-8e71-7d86cd0d867d-istio-token\") pod \"e2e-shared-a-w1-4d1110-openshift-default-6576975b6c-gvzcm\" (UID: \"1b385f61-b9d9-4f5d-8e71-7d86cd0d867d\") " pod="openshift-ingress/e2e-shared-a-w1-4d1110-openshift-default-6576975b6c-gvzcm" Oct 30 21:06:37.955760 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:06:37.951721 5378 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c537a2c1-13ee-488d-98b8-e8195b60d2be-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w4-88da25-5ddcdff7f9-5p4dg\" (UID: \"c537a2c1-13ee-488d-98b8-e8195b60d2be\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-88da25-5ddcdff7f9-5p4dg" Oct 30 21:06:37.955760 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:06:37.951807 5378 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wdq75\" (UniqueName: \"kubernetes.io/projected/c537a2c1-13ee-488d-98b8-e8195b60d2be-kube-api-access-wdq75\") pod \"payload-pre-processing-e2e-shared-b-w4-88da25-5ddcdff7f9-5p4dg\" (UID: \"c537a2c1-13ee-488d-98b8-e8195b60d2be\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-88da25-5ddcdff7f9-5p4dg" Oct 30 21:06:37.968589 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:06:37.956641 5378 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c537a2c1-13ee-488d-98b8-e8195b60d2be-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w4-88da25-5ddcdff7f9-5p4dg\" (UID: \"c537a2c1-13ee-488d-98b8-e8195b60d2be\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-88da25-5ddcdff7f9-5p4dg" Oct 30 21:06:38.010877 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:06:38.010843 5378 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w1-4d1110-openshift-default-6576975b6c-gvzcm" Oct 30 21:06:38.047475 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:06:38.047442 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:06:38.177590 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:06:38.177262 5378 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wdq75\" (UniqueName: \"kubernetes.io/projected/c537a2c1-13ee-488d-98b8-e8195b60d2be-kube-api-access-wdq75\") pod \"payload-pre-processing-e2e-shared-b-w4-88da25-5ddcdff7f9-5p4dg\" (UID: \"c537a2c1-13ee-488d-98b8-e8195b60d2be\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-88da25-5ddcdff7f9-5p4dg" Oct 30 21:06:38.432914 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:06:38.432881 5378 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-88da25-5ddcdff7f9-5p4dg" Oct 30 21:06:38.437088 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:06:38.436697 5378 patch_prober.go:29] interesting pod/e2e-shared-a-w4-5c45e5-openshift-default-9dcdd54d9-s697g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.109:15021/healthz/ready\": dial tcp 10.133.0.109:15021: connect: connection refused" start-of-body= Oct 30 21:06:38.437088 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:06:38.436747 5378 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-5c45e5-openshift-default-9dcdd54d9-s697g" podUID="23f08320-e0a1-42bc-8fd1-de3492b59bca" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.109:15021/healthz/ready\": dial tcp 10.133.0.109:15021: connect: connection refused" Oct 30 21:06:38.680923 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:06:38.680883 5378 patch_prober.go:29] interesting pod/e2e-derive-bf9c0727-openshift-default-7fb479dd89-jzzdk container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 21:06:38.681188 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:06:38.681158 5378 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-derive-bf9c0727-openshift-default-7fb479dd89-jzzdk" podUID="eafa34dc-6d4b-4507-a2a3-989ad8cf7de2" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 21:06:38.693333 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:06:38.693305 5378 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-4d1110-openshift-default-6576975b6c-gvzcm"] Oct 30 21:06:38.694969 ip-10-0-138-149 kubenswrapper[5378]: W1030 21:06:38.694945 5378 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1b385f61_b9d9_4f5d_8e71_7d86cd0d867d.slice/crio-89b9de5fc01269735421ccbfcc29bec60d14af2f60651411e270d4c5a4eba5aa WatchSource:0}: Error finding container 89b9de5fc01269735421ccbfcc29bec60d14af2f60651411e270d4c5a4eba5aa: Status 404 returned error can't find the container with id 89b9de5fc01269735421ccbfcc29bec60d14af2f60651411e270d4c5a4eba5aa Oct 30 21:06:38.698475 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:06:38.698435 5378 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 21:06:38.698590 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:06:38.698524 5378 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 21:06:38.698644 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:06:38.698626 5378 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 21:06:38.831794 ip-10-0-138-149 kubenswrapper[5378]: W1030 21:06:38.831763 5378 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc537a2c1_13ee_488d_98b8_e8195b60d2be.slice/crio-58ec8fca92488331b2fb594fae4d1c4d6a489bee9a64174433d3d3f9009c14ff WatchSource:0}: Error finding container 58ec8fca92488331b2fb594fae4d1c4d6a489bee9a64174433d3d3f9009c14ff: Status 404 returned error can't find the container with id 58ec8fca92488331b2fb594fae4d1c4d6a489bee9a64174433d3d3f9009c14ff Oct 30 21:06:38.831960 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:06:38.831936 5378 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w4-88da25-5ddcdff7f9-5p4dg"] Oct 30 21:06:38.844014 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:06:38.843980 5378 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-4d1110-openshift-default-6576975b6c-gvzcm" event={"ID":"1b385f61-b9d9-4f5d-8e71-7d86cd0d867d","Type":"ContainerStarted","Data":"b75ef819ca3ba0cace281a1b6a7a8378147dfa4b114d39242abfe2a7d5c9bb44"} Oct 30 21:06:38.844130 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:06:38.844043 5378 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-4d1110-openshift-default-6576975b6c-gvzcm" event={"ID":"1b385f61-b9d9-4f5d-8e71-7d86cd0d867d","Type":"ContainerStarted","Data":"89b9de5fc01269735421ccbfcc29bec60d14af2f60651411e270d4c5a4eba5aa"} Oct 30 21:06:38.932823 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:06:38.932723 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:06:38.977427 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:06:38.977359 5378 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w1-4d1110-openshift-default-6576975b6c-gvzcm" podStartSLOduration=1.9773398229999999 podStartE2EDuration="1.977339823s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:06:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:06:38.970304643 +0000 UTC m=+1689.398418563" watchObservedRunningTime="2026-10-30 21:06:38.977339823 +0000 UTC m=+1689.405453744" Oct 30 21:06:39.046644 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:06:39.046604 5378 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w1-4d1110-openshift-default-6576975b6c-gvzcm" Oct 30 21:06:39.048343 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:06:39.048308 5378 patch_prober.go:29] interesting pod/e2e-shared-a-w1-4d1110-openshift-default-6576975b6c-gvzcm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.123:15021/healthz/ready\": dial tcp 10.133.0.123:15021: connect: connection refused" start-of-body= Oct 30 21:06:39.048495 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:06:39.048363 5378 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-4d1110-openshift-default-6576975b6c-gvzcm" podUID="1b385f61-b9d9-4f5d-8e71-7d86cd0d867d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.123:15021/healthz/ready\": dial tcp 10.133.0.123:15021: connect: connection refused" Oct 30 21:06:39.080960 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:06:39.080930 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:06:39.215639 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:06:39.214108 5378 patch_prober.go:29] interesting pod/e2e-ait-2cf21005-openshift-default-5fdcf57455-c7jwb container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 21:06:39.215639 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:06:39.214180 5378 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ait-2cf21005-openshift-default-5fdcf57455-c7jwb" podUID="c7035375-d666-498d-b9e7-f699e1e86c99" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 21:06:39.292527 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:06:39.292479 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:06:39.441279 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:06:39.441246 5378 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w4-5c45e5-openshift-default-9dcdd54d9-s697g" Oct 30 21:06:39.441776 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:06:39.441754 5378 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w4-5c45e5-openshift-default-9dcdd54d9-s697g" Oct 30 21:06:39.442559 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:06:39.442539 5378 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w4-5c45e5-openshift-default-9dcdd54d9-s697g" Oct 30 21:06:39.483896 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:06:39.483828 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:06:39.854656 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:06:39.854621 5378 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-88da25-5ddcdff7f9-5p4dg" event={"ID":"c537a2c1-13ee-488d-98b8-e8195b60d2be","Type":"ContainerStarted","Data":"016794a990a4b6581264a8720f413a29daa35f7c022e18e8ef04bf21897e8c92"} Oct 30 21:06:39.854866 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:06:39.854677 5378 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-88da25-5ddcdff7f9-5p4dg" event={"ID":"c537a2c1-13ee-488d-98b8-e8195b60d2be","Type":"ContainerStarted","Data":"58ec8fca92488331b2fb594fae4d1c4d6a489bee9a64174433d3d3f9009c14ff"} Oct 30 21:06:39.854866 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:06:39.854703 5378 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-88da25-5ddcdff7f9-5p4dg" Oct 30 21:06:39.990343 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:06:39.990288 5378 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-88da25-5ddcdff7f9-5p4dg" podStartSLOduration=2.990275512 podStartE2EDuration="2.990275512s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:06:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:06:39.987863681 +0000 UTC m=+1690.415977601" watchObservedRunningTime="2026-10-30 21:06:39.990275512 +0000 UTC m=+1690.418389456" Oct 30 21:06:40.050673 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:06:40.050642 5378 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w1-4d1110-openshift-default-6576975b6c-gvzcm" Oct 30 21:06:40.611210 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:06:40.611179 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:06:40.695718 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:06:40.695695 5378 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-2cf21005-openshift-default-5fdcf57455-c7jwb" Oct 30 21:06:40.726009 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:06:40.725972 5378 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c7035375-d666-498d-b9e7-f699e1e86c99-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c7035375-d666-498d-b9e7-f699e1e86c99-credential-socket\") pod \"c7035375-d666-498d-b9e7-f699e1e86c99\" (UID: \"c7035375-d666-498d-b9e7-f699e1e86c99\") " Oct 30 21:06:40.726161 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:06:40.726073 5378 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/c7035375-d666-498d-b9e7-f699e1e86c99-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c7035375-d666-498d-b9e7-f699e1e86c99-istio-podinfo\") pod \"c7035375-d666-498d-b9e7-f699e1e86c99\" (UID: \"c7035375-d666-498d-b9e7-f699e1e86c99\") " Oct 30 21:06:40.726241 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:06:40.726162 5378 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c7035375-d666-498d-b9e7-f699e1e86c99-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c7035375-d666-498d-b9e7-f699e1e86c99-istio-data\") pod \"c7035375-d666-498d-b9e7-f699e1e86c99\" (UID: \"c7035375-d666-498d-b9e7-f699e1e86c99\") " Oct 30 21:06:40.726241 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:06:40.726204 5378 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c7035375-d666-498d-b9e7-f699e1e86c99-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c7035375-d666-498d-b9e7-f699e1e86c99-workload-socket\") pod \"c7035375-d666-498d-b9e7-f699e1e86c99\" (UID: \"c7035375-d666-498d-b9e7-f699e1e86c99\") " Oct 30 21:06:40.726362 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:06:40.726256 5378 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c7035375-d666-498d-b9e7-f699e1e86c99-credential-socket" pod "c7035375-d666-498d-b9e7-f699e1e86c99" (UID: "c7035375-d666-498d-b9e7-f699e1e86c99"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:06:40.726362 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:06:40.726296 5378 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c7035375-d666-498d-b9e7-f699e1e86c99-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c7035375-d666-498d-b9e7-f699e1e86c99-istio-envoy\") pod \"c7035375-d666-498d-b9e7-f699e1e86c99\" (UID: \"c7035375-d666-498d-b9e7-f699e1e86c99\") " Oct 30 21:06:40.726362 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:06:40.726331 5378 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c7035375-d666-498d-b9e7-f699e1e86c99-istio-token\" (UniqueName: \"kubernetes.io/projected/c7035375-d666-498d-b9e7-f699e1e86c99-istio-token\") pod \"c7035375-d666-498d-b9e7-f699e1e86c99\" (UID: \"c7035375-d666-498d-b9e7-f699e1e86c99\") " Oct 30 21:06:40.726362 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:06:40.726360 5378 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c7035375-d666-498d-b9e7-f699e1e86c99-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c7035375-d666-498d-b9e7-f699e1e86c99-istiod-ca-cert\") pod \"c7035375-d666-498d-b9e7-f699e1e86c99\" (UID: \"c7035375-d666-498d-b9e7-f699e1e86c99\") " Oct 30 21:06:40.726611 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:06:40.726446 5378 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c7035375-d666-498d-b9e7-f699e1e86c99-kube-api-access-lcb5t\" (UniqueName: \"kubernetes.io/projected/c7035375-d666-498d-b9e7-f699e1e86c99-kube-api-access-lcb5t\") pod \"c7035375-d666-498d-b9e7-f699e1e86c99\" (UID: \"c7035375-d666-498d-b9e7-f699e1e86c99\") " Oct 30 21:06:40.726611 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:06:40.726462 5378 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c7035375-d666-498d-b9e7-f699e1e86c99-istio-data" pod "c7035375-d666-498d-b9e7-f699e1e86c99" (UID: "c7035375-d666-498d-b9e7-f699e1e86c99"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:06:40.726611 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:06:40.726470 5378 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c7035375-d666-498d-b9e7-f699e1e86c99-workload-socket" pod "c7035375-d666-498d-b9e7-f699e1e86c99" (UID: "c7035375-d666-498d-b9e7-f699e1e86c99"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:06:40.726611 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:06:40.726499 5378 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c7035375-d666-498d-b9e7-f699e1e86c99-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c7035375-d666-498d-b9e7-f699e1e86c99-workload-certs\") pod \"c7035375-d666-498d-b9e7-f699e1e86c99\" (UID: \"c7035375-d666-498d-b9e7-f699e1e86c99\") " Oct 30 21:06:40.726853 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:06:40.726823 5378 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c7035375-d666-498d-b9e7-f699e1e86c99-istiod-ca-cert" pod "c7035375-d666-498d-b9e7-f699e1e86c99" (UID: "c7035375-d666-498d-b9e7-f699e1e86c99"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:06:40.727055 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:06:40.727031 5378 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c7035375-d666-498d-b9e7-f699e1e86c99-workload-certs" pod "c7035375-d666-498d-b9e7-f699e1e86c99" (UID: "c7035375-d666-498d-b9e7-f699e1e86c99"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:06:40.727315 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:06:40.727292 5378 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c7035375-d666-498d-b9e7-f699e1e86c99-credential-socket\") on node \"ip-10-0-138-149.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:06:40.727315 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:06:40.727316 5378 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c7035375-d666-498d-b9e7-f699e1e86c99-istio-data\") on node \"ip-10-0-138-149.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:06:40.727501 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:06:40.727330 5378 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c7035375-d666-498d-b9e7-f699e1e86c99-workload-socket\") on node \"ip-10-0-138-149.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:06:40.727501 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:06:40.727346 5378 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c7035375-d666-498d-b9e7-f699e1e86c99-istiod-ca-cert\") on node \"ip-10-0-138-149.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:06:40.727501 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:06:40.727359 5378 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c7035375-d666-498d-b9e7-f699e1e86c99-workload-certs\") on node \"ip-10-0-138-149.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:06:40.728727 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:06:40.728701 5378 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c7035375-d666-498d-b9e7-f699e1e86c99-kube-api-access-lcb5t" pod "c7035375-d666-498d-b9e7-f699e1e86c99" (UID: "c7035375-d666-498d-b9e7-f699e1e86c99"). InnerVolumeSpecName "kube-api-access-lcb5t". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:06:40.728981 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:06:40.728948 5378 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c7035375-d666-498d-b9e7-f699e1e86c99-istio-envoy" pod "c7035375-d666-498d-b9e7-f699e1e86c99" (UID: "c7035375-d666-498d-b9e7-f699e1e86c99"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:06:40.729098 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:06:40.728996 5378 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c7035375-d666-498d-b9e7-f699e1e86c99-istio-podinfo" pod "c7035375-d666-498d-b9e7-f699e1e86c99" (UID: "c7035375-d666-498d-b9e7-f699e1e86c99"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 21:06:40.729400 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:06:40.729374 5378 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c7035375-d666-498d-b9e7-f699e1e86c99-istio-token" pod "c7035375-d666-498d-b9e7-f699e1e86c99" (UID: "c7035375-d666-498d-b9e7-f699e1e86c99"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:06:40.797559 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:06:40.797528 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:06:40.828481 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:06:40.828450 5378 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c7035375-d666-498d-b9e7-f699e1e86c99-istio-podinfo\") on node \"ip-10-0-138-149.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:06:40.828481 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:06:40.828477 5378 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c7035375-d666-498d-b9e7-f699e1e86c99-istio-envoy\") on node \"ip-10-0-138-149.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:06:40.828481 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:06:40.828487 5378 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c7035375-d666-498d-b9e7-f699e1e86c99-istio-token\") on node \"ip-10-0-138-149.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:06:40.828717 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:06:40.828496 5378 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lcb5t\" (UniqueName: \"kubernetes.io/projected/c7035375-d666-498d-b9e7-f699e1e86c99-kube-api-access-lcb5t\") on node \"ip-10-0-138-149.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:06:40.865049 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:06:40.865020 5378 generic.go:356] "Generic (PLEG): container finished" podID="c7035375-d666-498d-b9e7-f699e1e86c99" containerID="51c283842e2cefe34949229234561d3e96098541d98c1f3de02529a6df501442" exitCode=0 Oct 30 21:06:40.865222 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:06:40.865064 5378 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-2cf21005-openshift-default-5fdcf57455-c7jwb" event={"ID":"c7035375-d666-498d-b9e7-f699e1e86c99","Type":"ContainerDied","Data":"51c283842e2cefe34949229234561d3e96098541d98c1f3de02529a6df501442"} Oct 30 21:06:40.865222 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:06:40.865103 5378 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-2cf21005-openshift-default-5fdcf57455-c7jwb" Oct 30 21:06:40.865222 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:06:40.865127 5378 scope.go:122] "RemoveContainer" containerID="51c283842e2cefe34949229234561d3e96098541d98c1f3de02529a6df501442" Oct 30 21:06:40.865411 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:06:40.865117 5378 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-2cf21005-openshift-default-5fdcf57455-c7jwb" event={"ID":"c7035375-d666-498d-b9e7-f699e1e86c99","Type":"ContainerDied","Data":"e30b2b2df50ae6f8835d26d496c18b1b80bd047343b7f067451c91b66531e82c"} Oct 30 21:06:40.866004 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:06:40.865984 5378 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w1-4d1110-openshift-default-6576975b6c-gvzcm" Oct 30 21:06:40.868770 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:06:40.868752 5378 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w1-4d1110-openshift-default-6576975b6c-gvzcm" Oct 30 21:06:40.877887 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:06:40.877867 5378 scope.go:122] "RemoveContainer" containerID="59bfd2be3c8f8ea9773d7c0dfc4caacef2573c6729b786c89f506e047f94d370" Oct 30 21:06:40.891859 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:06:40.891839 5378 scope.go:122] "RemoveContainer" containerID="51c283842e2cefe34949229234561d3e96098541d98c1f3de02529a6df501442" Oct 30 21:06:40.892141 ip-10-0-138-149 kubenswrapper[5378]: E1030 21:06:40.892123 5378 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"51c283842e2cefe34949229234561d3e96098541d98c1f3de02529a6df501442\": container with ID starting with 51c283842e2cefe34949229234561d3e96098541d98c1f3de02529a6df501442 not found: ID does not exist" containerID="51c283842e2cefe34949229234561d3e96098541d98c1f3de02529a6df501442" Oct 30 21:06:40.892213 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:06:40.892153 5378 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"51c283842e2cefe34949229234561d3e96098541d98c1f3de02529a6df501442"} err="failed to get container status \"51c283842e2cefe34949229234561d3e96098541d98c1f3de02529a6df501442\": rpc error: code = NotFound desc = could not find container \"51c283842e2cefe34949229234561d3e96098541d98c1f3de02529a6df501442\": container with ID starting with 51c283842e2cefe34949229234561d3e96098541d98c1f3de02529a6df501442 not found: ID does not exist" Oct 30 21:06:40.892213 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:06:40.892180 5378 scope.go:122] "RemoveContainer" containerID="59bfd2be3c8f8ea9773d7c0dfc4caacef2573c6729b786c89f506e047f94d370" Oct 30 21:06:40.892458 ip-10-0-138-149 kubenswrapper[5378]: E1030 21:06:40.892438 5378 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"59bfd2be3c8f8ea9773d7c0dfc4caacef2573c6729b786c89f506e047f94d370\": container with ID starting with 59bfd2be3c8f8ea9773d7c0dfc4caacef2573c6729b786c89f506e047f94d370 not found: ID does not exist" containerID="59bfd2be3c8f8ea9773d7c0dfc4caacef2573c6729b786c89f506e047f94d370" Oct 30 21:06:40.892499 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:06:40.892465 5378 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"59bfd2be3c8f8ea9773d7c0dfc4caacef2573c6729b786c89f506e047f94d370"} err="failed to get container status \"59bfd2be3c8f8ea9773d7c0dfc4caacef2573c6729b786c89f506e047f94d370\": rpc error: code = NotFound desc = could not find container \"59bfd2be3c8f8ea9773d7c0dfc4caacef2573c6729b786c89f506e047f94d370\": container with ID starting with 59bfd2be3c8f8ea9773d7c0dfc4caacef2573c6729b786c89f506e047f94d370 not found: ID does not exist" Oct 30 21:06:41.166545 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:06:41.166510 5378 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-2cf21005-openshift-default-5fdcf57455-c7jwb"] Oct 30 21:06:41.170322 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:06:41.170289 5378 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-2cf21005-openshift-default-5fdcf57455-c7jwb"] Oct 30 21:06:41.744680 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:06:41.744649 5378 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c7035375-d666-498d-b9e7-f699e1e86c99" path="/var/lib/kubelet/pods/c7035375-d666-498d-b9e7-f699e1e86c99/volumes" Oct 30 21:06:42.510911 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:06:42.510877 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:06:42.748930 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:06:42.748898 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:06:42.997294 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:06:42.987376 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:06:43.031251 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:06:43.031174 5378 generic.go:356] "Generic (PLEG): container finished" podID="eafa34dc-6d4b-4507-a2a3-989ad8cf7de2" containerID="36a0351f1605358413b4ed86dc714b151758b0961124deaed1360602b7804c2a" exitCode=0 Oct 30 21:06:43.031552 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:06:43.031527 5378 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-bf9c0727-openshift-default-7fb479dd89-jzzdk" event={"ID":"eafa34dc-6d4b-4507-a2a3-989ad8cf7de2","Type":"ContainerDied","Data":"36a0351f1605358413b4ed86dc714b151758b0961124deaed1360602b7804c2a"} Oct 30 21:06:43.031792 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:06:43.031759 5378 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-bf9c0727-openshift-default-7fb479dd89-jzzdk" event={"ID":"eafa34dc-6d4b-4507-a2a3-989ad8cf7de2","Type":"ContainerDied","Data":"3cf3c06515afb92df7a8bd3560033ff45a29ff39fe29ee05fa78897da9b2d799"} Oct 30 21:06:43.031914 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:06:43.031898 5378 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3cf3c06515afb92df7a8bd3560033ff45a29ff39fe29ee05fa78897da9b2d799" Oct 30 21:06:43.032021 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:06:43.032010 5378 scope.go:122] "RemoveContainer" containerID="c90939ef377fe76a9fbc3ed0753ebe88aaaad74d368b45d4a2658581909adb44" Oct 30 21:06:43.094639 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:06:43.086480 5378 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-bf9c0727-openshift-default-7fb479dd89-jzzdk" Oct 30 21:06:43.113717 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:06:43.107927 5378 generic.go:356] "Generic (PLEG): container finished" podID="88bbd407-197c-4ba6-b31f-fc097ec5fbf3" containerID="434ca2e3b8666ac602b32ac4097c0bb6d841dcdeac21e53e8c262b96fa49f606" exitCode=0 Oct 30 21:06:43.123751 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:06:43.123706 5378 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-bf9c0727-openshift-default-7fb479dd89-lrsjm" event={"ID":"88bbd407-197c-4ba6-b31f-fc097ec5fbf3","Type":"ContainerDied","Data":"434ca2e3b8666ac602b32ac4097c0bb6d841dcdeac21e53e8c262b96fa49f606"} Oct 30 21:06:43.193598 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:06:43.189461 5378 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/eafa34dc-6d4b-4507-a2a3-989ad8cf7de2-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/eafa34dc-6d4b-4507-a2a3-989ad8cf7de2-istio-podinfo\") pod \"eafa34dc-6d4b-4507-a2a3-989ad8cf7de2\" (UID: \"eafa34dc-6d4b-4507-a2a3-989ad8cf7de2\") " Oct 30 21:06:43.193598 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:06:43.189627 5378 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/eafa34dc-6d4b-4507-a2a3-989ad8cf7de2-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/eafa34dc-6d4b-4507-a2a3-989ad8cf7de2-workload-certs\") pod \"eafa34dc-6d4b-4507-a2a3-989ad8cf7de2\" (UID: \"eafa34dc-6d4b-4507-a2a3-989ad8cf7de2\") " Oct 30 21:06:43.193846 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:06:43.193713 5378 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/eafa34dc-6d4b-4507-a2a3-989ad8cf7de2-istio-podinfo" pod "eafa34dc-6d4b-4507-a2a3-989ad8cf7de2" (UID: "eafa34dc-6d4b-4507-a2a3-989ad8cf7de2"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 21:06:43.216142 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:06:43.194122 5378 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eafa34dc-6d4b-4507-a2a3-989ad8cf7de2-workload-certs" pod "eafa34dc-6d4b-4507-a2a3-989ad8cf7de2" (UID: "eafa34dc-6d4b-4507-a2a3-989ad8cf7de2"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:06:43.216142 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:06:43.194663 5378 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/eafa34dc-6d4b-4507-a2a3-989ad8cf7de2-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/eafa34dc-6d4b-4507-a2a3-989ad8cf7de2-istio-data\") pod \"eafa34dc-6d4b-4507-a2a3-989ad8cf7de2\" (UID: \"eafa34dc-6d4b-4507-a2a3-989ad8cf7de2\") " Oct 30 21:06:43.216142 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:06:43.194772 5378 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/eafa34dc-6d4b-4507-a2a3-989ad8cf7de2-istio-token\" (UniqueName: \"kubernetes.io/projected/eafa34dc-6d4b-4507-a2a3-989ad8cf7de2-istio-token\") pod \"eafa34dc-6d4b-4507-a2a3-989ad8cf7de2\" (UID: \"eafa34dc-6d4b-4507-a2a3-989ad8cf7de2\") " Oct 30 21:06:43.216142 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:06:43.194810 5378 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/eafa34dc-6d4b-4507-a2a3-989ad8cf7de2-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/eafa34dc-6d4b-4507-a2a3-989ad8cf7de2-workload-socket\") pod \"eafa34dc-6d4b-4507-a2a3-989ad8cf7de2\" (UID: \"eafa34dc-6d4b-4507-a2a3-989ad8cf7de2\") " Oct 30 21:06:43.216142 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:06:43.195804 5378 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/eafa34dc-6d4b-4507-a2a3-989ad8cf7de2-istio-podinfo\") on node \"ip-10-0-138-149.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:06:43.216142 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:06:43.195828 5378 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/eafa34dc-6d4b-4507-a2a3-989ad8cf7de2-workload-certs\") on node \"ip-10-0-138-149.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:06:43.216142 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:06:43.196732 5378 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eafa34dc-6d4b-4507-a2a3-989ad8cf7de2-istio-data" pod "eafa34dc-6d4b-4507-a2a3-989ad8cf7de2" (UID: "eafa34dc-6d4b-4507-a2a3-989ad8cf7de2"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:06:43.216142 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:06:43.197866 5378 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eafa34dc-6d4b-4507-a2a3-989ad8cf7de2-workload-socket" pod "eafa34dc-6d4b-4507-a2a3-989ad8cf7de2" (UID: "eafa34dc-6d4b-4507-a2a3-989ad8cf7de2"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:06:43.223127 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:06:43.219668 5378 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w6-dc6125-74d649bf67-vlsfb"] Oct 30 21:06:43.223127 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:06:43.220716 5378 cpu_manager.go:414] "containerMap: removing container" podUID="c7035375-d666-498d-b9e7-f699e1e86c99" containerName="istio-proxy" Oct 30 21:06:43.223127 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:06:43.220740 5378 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c7035375-d666-498d-b9e7-f699e1e86c99" containerName="istio-proxy" Oct 30 21:06:43.223127 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:06:43.220755 5378 cpu_manager.go:414] "containerMap: removing container" podUID="eafa34dc-6d4b-4507-a2a3-989ad8cf7de2" containerName="istio-proxy" Oct 30 21:06:43.223127 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:06:43.220763 5378 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="eafa34dc-6d4b-4507-a2a3-989ad8cf7de2" containerName="istio-proxy" Oct 30 21:06:43.223127 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:06:43.220970 5378 memory_manager.go:365] "RemoveStaleState removing state" podUID="eafa34dc-6d4b-4507-a2a3-989ad8cf7de2" containerName="istio-proxy" Oct 30 21:06:43.223127 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:06:43.220995 5378 memory_manager.go:365] "RemoveStaleState removing state" podUID="c7035375-d666-498d-b9e7-f699e1e86c99" containerName="istio-proxy" Oct 30 21:06:43.223127 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:06:43.221005 5378 memory_manager.go:365] "RemoveStaleState removing state" podUID="eafa34dc-6d4b-4507-a2a3-989ad8cf7de2" containerName="istio-proxy" Oct 30 21:06:43.224762 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:06:43.224744 5378 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-dc6125-74d649bf67-vlsfb" Oct 30 21:06:43.230070 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:06:43.229665 5378 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eafa34dc-6d4b-4507-a2a3-989ad8cf7de2-istio-token" pod "eafa34dc-6d4b-4507-a2a3-989ad8cf7de2" (UID: "eafa34dc-6d4b-4507-a2a3-989ad8cf7de2"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:06:43.236831 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:06:43.236792 5378 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w6-dc6125-74d649bf67-vlsfb"] Oct 30 21:06:43.278271 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:06:43.278182 5378 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w6-dc6125\"" Oct 30 21:06:43.425805 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:06:43.370222 5378 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/eafa34dc-6d4b-4507-a2a3-989ad8cf7de2-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/eafa34dc-6d4b-4507-a2a3-989ad8cf7de2-istiod-ca-cert\") pod \"eafa34dc-6d4b-4507-a2a3-989ad8cf7de2\" (UID: \"eafa34dc-6d4b-4507-a2a3-989ad8cf7de2\") " Oct 30 21:06:43.425805 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:06:43.370300 5378 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/eafa34dc-6d4b-4507-a2a3-989ad8cf7de2-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/eafa34dc-6d4b-4507-a2a3-989ad8cf7de2-credential-socket\") pod \"eafa34dc-6d4b-4507-a2a3-989ad8cf7de2\" (UID: \"eafa34dc-6d4b-4507-a2a3-989ad8cf7de2\") " Oct 30 21:06:43.425805 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:06:43.370376 5378 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/eafa34dc-6d4b-4507-a2a3-989ad8cf7de2-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/eafa34dc-6d4b-4507-a2a3-989ad8cf7de2-istio-envoy\") pod \"eafa34dc-6d4b-4507-a2a3-989ad8cf7de2\" (UID: \"eafa34dc-6d4b-4507-a2a3-989ad8cf7de2\") " Oct 30 21:06:43.425805 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:06:43.370625 5378 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/eafa34dc-6d4b-4507-a2a3-989ad8cf7de2-kube-api-access-kqdmr\" (UniqueName: \"kubernetes.io/projected/eafa34dc-6d4b-4507-a2a3-989ad8cf7de2-kube-api-access-kqdmr\") pod \"eafa34dc-6d4b-4507-a2a3-989ad8cf7de2\" (UID: \"eafa34dc-6d4b-4507-a2a3-989ad8cf7de2\") " Oct 30 21:06:43.425805 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:06:43.371257 5378 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/eafa34dc-6d4b-4507-a2a3-989ad8cf7de2-istio-token\") on node \"ip-10-0-138-149.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:06:43.425805 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:06:43.371278 5378 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/eafa34dc-6d4b-4507-a2a3-989ad8cf7de2-workload-socket\") on node \"ip-10-0-138-149.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:06:43.425805 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:06:43.371292 5378 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/eafa34dc-6d4b-4507-a2a3-989ad8cf7de2-istio-data\") on node \"ip-10-0-138-149.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:06:43.425805 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:06:43.401444 5378 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/eafa34dc-6d4b-4507-a2a3-989ad8cf7de2-istiod-ca-cert" pod "eafa34dc-6d4b-4507-a2a3-989ad8cf7de2" (UID: "eafa34dc-6d4b-4507-a2a3-989ad8cf7de2"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:06:43.425805 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:06:43.403592 5378 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eafa34dc-6d4b-4507-a2a3-989ad8cf7de2-credential-socket" pod "eafa34dc-6d4b-4507-a2a3-989ad8cf7de2" (UID: "eafa34dc-6d4b-4507-a2a3-989ad8cf7de2"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:06:43.458977 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:06:43.458908 5378 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eafa34dc-6d4b-4507-a2a3-989ad8cf7de2-kube-api-access-kqdmr" pod "eafa34dc-6d4b-4507-a2a3-989ad8cf7de2" (UID: "eafa34dc-6d4b-4507-a2a3-989ad8cf7de2"). InnerVolumeSpecName "kube-api-access-kqdmr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:06:43.492778 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:06:43.464694 5378 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eafa34dc-6d4b-4507-a2a3-989ad8cf7de2-istio-envoy" pod "eafa34dc-6d4b-4507-a2a3-989ad8cf7de2" (UID: "eafa34dc-6d4b-4507-a2a3-989ad8cf7de2"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:06:43.492778 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:06:43.466471 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:06:43.492778 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:06:43.478807 5378 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hxrq7\" (UniqueName: \"kubernetes.io/projected/5ff92f96-e5bc-4b04-893b-16efd49f9d3f-kube-api-access-hxrq7\") pod \"payload-pre-processing-e2e-shared-a-w6-dc6125-74d649bf67-vlsfb\" (UID: \"5ff92f96-e5bc-4b04-893b-16efd49f9d3f\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-dc6125-74d649bf67-vlsfb" Oct 30 21:06:43.492778 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:06:43.478978 5378 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5ff92f96-e5bc-4b04-893b-16efd49f9d3f-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w6-dc6125-74d649bf67-vlsfb\" (UID: \"5ff92f96-e5bc-4b04-893b-16efd49f9d3f\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-dc6125-74d649bf67-vlsfb" Oct 30 21:06:43.492778 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:06:43.479211 5378 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/eafa34dc-6d4b-4507-a2a3-989ad8cf7de2-istiod-ca-cert\") on node \"ip-10-0-138-149.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:06:43.492778 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:06:43.479227 5378 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/eafa34dc-6d4b-4507-a2a3-989ad8cf7de2-credential-socket\") on node \"ip-10-0-138-149.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:06:43.492778 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:06:43.479240 5378 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/eafa34dc-6d4b-4507-a2a3-989ad8cf7de2-istio-envoy\") on node \"ip-10-0-138-149.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:06:43.492778 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:06:43.479251 5378 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kqdmr\" (UniqueName: \"kubernetes.io/projected/eafa34dc-6d4b-4507-a2a3-989ad8cf7de2-kube-api-access-kqdmr\") on node \"ip-10-0-138-149.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:06:43.517690 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:06:43.503162 5378 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w6-dc6125-6644bb494b-2v578"] Oct 30 21:06:43.517690 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:06:43.504211 5378 cpu_manager.go:414] "containerMap: removing container" podUID="c7035375-d666-498d-b9e7-f699e1e86c99" containerName="istio-proxy" Oct 30 21:06:43.517690 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:06:43.504234 5378 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c7035375-d666-498d-b9e7-f699e1e86c99" containerName="istio-proxy" Oct 30 21:06:43.517690 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:06:43.504253 5378 cpu_manager.go:414] "containerMap: removing container" podUID="eafa34dc-6d4b-4507-a2a3-989ad8cf7de2" containerName="istio-proxy" Oct 30 21:06:43.517690 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:06:43.504261 5378 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="eafa34dc-6d4b-4507-a2a3-989ad8cf7de2" containerName="istio-proxy" Oct 30 21:06:43.517690 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:06:43.504486 5378 memory_manager.go:365] "RemoveStaleState removing state" podUID="c7035375-d666-498d-b9e7-f699e1e86c99" containerName="istio-proxy" Oct 30 21:06:43.521593 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:06:43.519874 5378 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w6-dc6125-6644bb494b-2v578" Oct 30 21:06:43.528586 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:06:43.524824 5378 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w6-dc6125-6644bb494b-2v578"] Oct 30 21:06:43.585236 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:06:43.582383 5378 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w6-dc6125-dockercfg-t52gt\"" Oct 30 21:06:43.585236 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:06:43.583613 5378 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5ff92f96-e5bc-4b04-893b-16efd49f9d3f-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w6-dc6125-74d649bf67-vlsfb\" (UID: \"5ff92f96-e5bc-4b04-893b-16efd49f9d3f\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-dc6125-74d649bf67-vlsfb" Oct 30 21:06:43.585236 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:06:43.583872 5378 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hxrq7\" (UniqueName: \"kubernetes.io/projected/5ff92f96-e5bc-4b04-893b-16efd49f9d3f-kube-api-access-hxrq7\") pod \"payload-pre-processing-e2e-shared-a-w6-dc6125-74d649bf67-vlsfb\" (UID: \"5ff92f96-e5bc-4b04-893b-16efd49f9d3f\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-dc6125-74d649bf67-vlsfb" Oct 30 21:06:43.585236 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:06:43.585153 5378 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5ff92f96-e5bc-4b04-893b-16efd49f9d3f-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w6-dc6125-74d649bf67-vlsfb\" (UID: \"5ff92f96-e5bc-4b04-893b-16efd49f9d3f\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-dc6125-74d649bf67-vlsfb" Oct 30 21:06:43.660224 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:06:43.660179 5378 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hxrq7\" (UniqueName: \"kubernetes.io/projected/5ff92f96-e5bc-4b04-893b-16efd49f9d3f-kube-api-access-hxrq7\") pod \"payload-pre-processing-e2e-shared-a-w6-dc6125-74d649bf67-vlsfb\" (UID: \"5ff92f96-e5bc-4b04-893b-16efd49f9d3f\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-dc6125-74d649bf67-vlsfb" Oct 30 21:06:43.685941 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:06:43.685856 5378 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8w7rb\" (UniqueName: \"kubernetes.io/projected/168599a3-7daf-4eba-a259-978486b56bd4-kube-api-access-8w7rb\") pod \"payload-processing-e2e-shared-a-w6-dc6125-6644bb494b-2v578\" (UID: \"168599a3-7daf-4eba-a259-978486b56bd4\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w6-dc6125-6644bb494b-2v578" Oct 30 21:06:43.686084 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:06:43.686022 5378 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/168599a3-7daf-4eba-a259-978486b56bd4-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w6-dc6125-6644bb494b-2v578\" (UID: \"168599a3-7daf-4eba-a259-978486b56bd4\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w6-dc6125-6644bb494b-2v578" Oct 30 21:06:43.756791 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:06:43.756764 5378 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-bf9c0727-openshift-default-7fb479dd89-lrsjm" Oct 30 21:06:43.787902 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:06:43.787867 5378 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/168599a3-7daf-4eba-a259-978486b56bd4-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w6-dc6125-6644bb494b-2v578\" (UID: \"168599a3-7daf-4eba-a259-978486b56bd4\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w6-dc6125-6644bb494b-2v578" Oct 30 21:06:43.788079 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:06:43.788066 5378 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8w7rb\" (UniqueName: \"kubernetes.io/projected/168599a3-7daf-4eba-a259-978486b56bd4-kube-api-access-8w7rb\") pod \"payload-processing-e2e-shared-a-w6-dc6125-6644bb494b-2v578\" (UID: \"168599a3-7daf-4eba-a259-978486b56bd4\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w6-dc6125-6644bb494b-2v578" Oct 30 21:06:43.788544 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:06:43.788519 5378 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/168599a3-7daf-4eba-a259-978486b56bd4-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w6-dc6125-6644bb494b-2v578\" (UID: \"168599a3-7daf-4eba-a259-978486b56bd4\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w6-dc6125-6644bb494b-2v578" Oct 30 21:06:43.845921 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:06:43.845869 5378 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8w7rb\" (UniqueName: \"kubernetes.io/projected/168599a3-7daf-4eba-a259-978486b56bd4-kube-api-access-8w7rb\") pod \"payload-processing-e2e-shared-a-w6-dc6125-6644bb494b-2v578\" (UID: \"168599a3-7daf-4eba-a259-978486b56bd4\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w6-dc6125-6644bb494b-2v578" Oct 30 21:06:43.847237 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:06:43.847198 5378 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-dc6125-74d649bf67-vlsfb" Oct 30 21:06:43.868848 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:06:43.868825 5378 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w6-dc6125-6644bb494b-2v578" Oct 30 21:06:43.890649 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:06:43.889675 5378 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/88bbd407-197c-4ba6-b31f-fc097ec5fbf3-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/88bbd407-197c-4ba6-b31f-fc097ec5fbf3-istio-envoy\") pod \"88bbd407-197c-4ba6-b31f-fc097ec5fbf3\" (UID: \"88bbd407-197c-4ba6-b31f-fc097ec5fbf3\") " Oct 30 21:06:43.890649 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:06:43.889731 5378 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/88bbd407-197c-4ba6-b31f-fc097ec5fbf3-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/88bbd407-197c-4ba6-b31f-fc097ec5fbf3-istio-data\") pod \"88bbd407-197c-4ba6-b31f-fc097ec5fbf3\" (UID: \"88bbd407-197c-4ba6-b31f-fc097ec5fbf3\") " Oct 30 21:06:43.890649 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:06:43.890367 5378 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/88bbd407-197c-4ba6-b31f-fc097ec5fbf3-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/88bbd407-197c-4ba6-b31f-fc097ec5fbf3-credential-socket\") pod \"88bbd407-197c-4ba6-b31f-fc097ec5fbf3\" (UID: \"88bbd407-197c-4ba6-b31f-fc097ec5fbf3\") " Oct 30 21:06:43.890649 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:06:43.890500 5378 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/88bbd407-197c-4ba6-b31f-fc097ec5fbf3-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/88bbd407-197c-4ba6-b31f-fc097ec5fbf3-workload-socket\") pod \"88bbd407-197c-4ba6-b31f-fc097ec5fbf3\" (UID: \"88bbd407-197c-4ba6-b31f-fc097ec5fbf3\") " Oct 30 21:06:43.890989 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:06:43.890757 5378 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/88bbd407-197c-4ba6-b31f-fc097ec5fbf3-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/88bbd407-197c-4ba6-b31f-fc097ec5fbf3-workload-certs\") pod \"88bbd407-197c-4ba6-b31f-fc097ec5fbf3\" (UID: \"88bbd407-197c-4ba6-b31f-fc097ec5fbf3\") " Oct 30 21:06:43.890989 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:06:43.890836 5378 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/88bbd407-197c-4ba6-b31f-fc097ec5fbf3-istio-token\" (UniqueName: \"kubernetes.io/projected/88bbd407-197c-4ba6-b31f-fc097ec5fbf3-istio-token\") pod \"88bbd407-197c-4ba6-b31f-fc097ec5fbf3\" (UID: \"88bbd407-197c-4ba6-b31f-fc097ec5fbf3\") " Oct 30 21:06:43.890989 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:06:43.890930 5378 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/88bbd407-197c-4ba6-b31f-fc097ec5fbf3-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/88bbd407-197c-4ba6-b31f-fc097ec5fbf3-istiod-ca-cert\") pod \"88bbd407-197c-4ba6-b31f-fc097ec5fbf3\" (UID: \"88bbd407-197c-4ba6-b31f-fc097ec5fbf3\") " Oct 30 21:06:43.890989 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:06:43.890975 5378 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/88bbd407-197c-4ba6-b31f-fc097ec5fbf3-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/88bbd407-197c-4ba6-b31f-fc097ec5fbf3-istio-podinfo\") pod \"88bbd407-197c-4ba6-b31f-fc097ec5fbf3\" (UID: \"88bbd407-197c-4ba6-b31f-fc097ec5fbf3\") " Oct 30 21:06:43.891216 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:06:43.891007 5378 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/88bbd407-197c-4ba6-b31f-fc097ec5fbf3-kube-api-access-6sfdj\" (UniqueName: \"kubernetes.io/projected/88bbd407-197c-4ba6-b31f-fc097ec5fbf3-kube-api-access-6sfdj\") pod \"88bbd407-197c-4ba6-b31f-fc097ec5fbf3\" (UID: \"88bbd407-197c-4ba6-b31f-fc097ec5fbf3\") " Oct 30 21:06:43.892376 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:06:43.892348 5378 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/88bbd407-197c-4ba6-b31f-fc097ec5fbf3-workload-certs" pod "88bbd407-197c-4ba6-b31f-fc097ec5fbf3" (UID: "88bbd407-197c-4ba6-b31f-fc097ec5fbf3"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:06:43.892554 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:06:43.892534 5378 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/88bbd407-197c-4ba6-b31f-fc097ec5fbf3-istio-data" pod "88bbd407-197c-4ba6-b31f-fc097ec5fbf3" (UID: "88bbd407-197c-4ba6-b31f-fc097ec5fbf3"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:06:43.892736 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:06:43.892716 5378 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/88bbd407-197c-4ba6-b31f-fc097ec5fbf3-credential-socket" pod "88bbd407-197c-4ba6-b31f-fc097ec5fbf3" (UID: "88bbd407-197c-4ba6-b31f-fc097ec5fbf3"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:06:43.892917 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:06:43.892897 5378 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/88bbd407-197c-4ba6-b31f-fc097ec5fbf3-workload-socket" pod "88bbd407-197c-4ba6-b31f-fc097ec5fbf3" (UID: "88bbd407-197c-4ba6-b31f-fc097ec5fbf3"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:06:43.893190 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:06:43.893169 5378 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/88bbd407-197c-4ba6-b31f-fc097ec5fbf3-istiod-ca-cert" pod "88bbd407-197c-4ba6-b31f-fc097ec5fbf3" (UID: "88bbd407-197c-4ba6-b31f-fc097ec5fbf3"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:06:43.897556 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:06:43.897522 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:06:43.897872 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:06:43.897848 5378 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/88bbd407-197c-4ba6-b31f-fc097ec5fbf3-istio-token" pod "88bbd407-197c-4ba6-b31f-fc097ec5fbf3" (UID: "88bbd407-197c-4ba6-b31f-fc097ec5fbf3"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:06:43.898796 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:06:43.898765 5378 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/88bbd407-197c-4ba6-b31f-fc097ec5fbf3-kube-api-access-6sfdj" pod "88bbd407-197c-4ba6-b31f-fc097ec5fbf3" (UID: "88bbd407-197c-4ba6-b31f-fc097ec5fbf3"). InnerVolumeSpecName "kube-api-access-6sfdj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:06:43.898892 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:06:43.898814 5378 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/88bbd407-197c-4ba6-b31f-fc097ec5fbf3-istio-podinfo" pod "88bbd407-197c-4ba6-b31f-fc097ec5fbf3" (UID: "88bbd407-197c-4ba6-b31f-fc097ec5fbf3"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 21:06:43.899829 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:06:43.899806 5378 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/88bbd407-197c-4ba6-b31f-fc097ec5fbf3-istio-envoy" pod "88bbd407-197c-4ba6-b31f-fc097ec5fbf3" (UID: "88bbd407-197c-4ba6-b31f-fc097ec5fbf3"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:06:43.993083 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:06:43.993054 5378 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/88bbd407-197c-4ba6-b31f-fc097ec5fbf3-credential-socket\") on node \"ip-10-0-138-149.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:06:43.993083 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:06:43.993083 5378 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/88bbd407-197c-4ba6-b31f-fc097ec5fbf3-workload-socket\") on node \"ip-10-0-138-149.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:06:43.993272 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:06:43.993093 5378 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/88bbd407-197c-4ba6-b31f-fc097ec5fbf3-workload-certs\") on node \"ip-10-0-138-149.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:06:43.993272 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:06:43.993103 5378 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/88bbd407-197c-4ba6-b31f-fc097ec5fbf3-istio-token\") on node \"ip-10-0-138-149.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:06:43.993272 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:06:43.993112 5378 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/88bbd407-197c-4ba6-b31f-fc097ec5fbf3-istiod-ca-cert\") on node \"ip-10-0-138-149.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:06:43.993272 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:06:43.993122 5378 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/88bbd407-197c-4ba6-b31f-fc097ec5fbf3-istio-podinfo\") on node \"ip-10-0-138-149.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:06:43.993272 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:06:43.993131 5378 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6sfdj\" (UniqueName: \"kubernetes.io/projected/88bbd407-197c-4ba6-b31f-fc097ec5fbf3-kube-api-access-6sfdj\") on node \"ip-10-0-138-149.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:06:43.993272 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:06:43.993139 5378 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/88bbd407-197c-4ba6-b31f-fc097ec5fbf3-istio-envoy\") on node \"ip-10-0-138-149.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:06:43.993272 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:06:43.993148 5378 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/88bbd407-197c-4ba6-b31f-fc097ec5fbf3-istio-data\") on node \"ip-10-0-138-149.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:06:44.116908 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:06:44.116876 5378 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-bf9c0727-openshift-default-7fb479dd89-jzzdk" Oct 30 21:06:44.118812 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:06:44.118787 5378 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-bf9c0727-openshift-default-7fb479dd89-lrsjm" Oct 30 21:06:44.119015 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:06:44.118723 5378 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-bf9c0727-openshift-default-7fb479dd89-lrsjm" event={"ID":"88bbd407-197c-4ba6-b31f-fc097ec5fbf3","Type":"ContainerDied","Data":"0199052dac8b0bd2cbb89c06de4f16cfae80a146fed6046f5f824a58ab2a37c8"} Oct 30 21:06:44.119159 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:06:44.119059 5378 scope.go:122] "RemoveContainer" containerID="434ca2e3b8666ac602b32ac4097c0bb6d841dcdeac21e53e8c262b96fa49f606" Oct 30 21:06:44.156276 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:06:44.156246 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:06:44.246964 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:06:44.246934 5378 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w6-dc6125-74d649bf67-vlsfb"] Oct 30 21:06:44.247268 ip-10-0-138-149 kubenswrapper[5378]: W1030 21:06:44.247236 5378 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5ff92f96_e5bc_4b04_893b_16efd49f9d3f.slice/crio-0cc8cede770432d7556933712d081b303ad2fa199c0028d5af935a277f65d762 WatchSource:0}: Error finding container 0cc8cede770432d7556933712d081b303ad2fa199c0028d5af935a277f65d762: Status 404 returned error can't find the container with id 0cc8cede770432d7556933712d081b303ad2fa199c0028d5af935a277f65d762 Oct 30 21:06:44.296410 ip-10-0-138-149 kubenswrapper[5378]: W1030 21:06:44.296385 5378 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod168599a3_7daf_4eba_a259_978486b56bd4.slice/crio-bbaca4227c2b51fab16e5f55f3e6e891bc90f96c1c661b63070c28d78b7f2aed WatchSource:0}: Error finding container bbaca4227c2b51fab16e5f55f3e6e891bc90f96c1c661b63070c28d78b7f2aed: Status 404 returned error can't find the container with id bbaca4227c2b51fab16e5f55f3e6e891bc90f96c1c661b63070c28d78b7f2aed Oct 30 21:06:44.298024 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:06:44.298000 5378 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w6-dc6125-6644bb494b-2v578"] Oct 30 21:06:44.300470 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:06:44.300443 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:06:44.392890 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:06:44.392859 5378 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-derive-bf9c0727-openshift-default-7fb479dd89-jzzdk"] Oct 30 21:06:44.397429 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:06:44.397406 5378 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-derive-bf9c0727-openshift-default-7fb479dd89-jzzdk"] Oct 30 21:06:44.486153 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:06:44.486125 5378 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w2-255f05-openshift-default-7db8bf49f-6bfml"] Oct 30 21:06:44.486855 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:06:44.486840 5378 cpu_manager.go:414] "containerMap: removing container" podUID="88bbd407-197c-4ba6-b31f-fc097ec5fbf3" containerName="istio-proxy" Oct 30 21:06:44.486899 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:06:44.486856 5378 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="88bbd407-197c-4ba6-b31f-fc097ec5fbf3" containerName="istio-proxy" Oct 30 21:06:44.486964 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:06:44.486955 5378 memory_manager.go:365] "RemoveStaleState removing state" podUID="88bbd407-197c-4ba6-b31f-fc097ec5fbf3" containerName="istio-proxy" Oct 30 21:06:44.489370 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:06:44.489353 5378 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-255f05-openshift-default-7db8bf49f-6bfml" Oct 30 21:06:44.511278 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:06:44.511250 5378 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-derive-bf9c0727-openshift-default-7fb479dd89-lrsjm"] Oct 30 21:06:44.516541 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:06:44.516515 5378 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-derive-bf9c0727-openshift-default-7fb479dd89-lrsjm"] Oct 30 21:06:44.551704 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:06:44.551679 5378 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w2-255f05-openshift-default-dockercfg-dvsg4\"" Oct 30 21:06:44.553103 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:06:44.553075 5378 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-255f05-openshift-default-7db8bf49f-6bfml"] Oct 30 21:06:44.592958 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:06:44.592923 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:06:44.608035 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:06:44.608008 5378 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4955d910-8544-4a2b-a1fd-b414734aed56-istio-podinfo\") pod \"e2e-shared-b-w2-255f05-openshift-default-7db8bf49f-6bfml\" (UID: \"4955d910-8544-4a2b-a1fd-b414734aed56\") " pod="openshift-ingress/e2e-shared-b-w2-255f05-openshift-default-7db8bf49f-6bfml" Oct 30 21:06:44.608156 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:06:44.608078 5378 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4955d910-8544-4a2b-a1fd-b414734aed56-credential-socket\") pod \"e2e-shared-b-w2-255f05-openshift-default-7db8bf49f-6bfml\" (UID: \"4955d910-8544-4a2b-a1fd-b414734aed56\") " pod="openshift-ingress/e2e-shared-b-w2-255f05-openshift-default-7db8bf49f-6bfml" Oct 30 21:06:44.608256 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:06:44.608235 5378 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4955d910-8544-4a2b-a1fd-b414734aed56-istio-token\") pod \"e2e-shared-b-w2-255f05-openshift-default-7db8bf49f-6bfml\" (UID: \"4955d910-8544-4a2b-a1fd-b414734aed56\") " pod="openshift-ingress/e2e-shared-b-w2-255f05-openshift-default-7db8bf49f-6bfml" Oct 30 21:06:44.608295 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:06:44.608278 5378 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4955d910-8544-4a2b-a1fd-b414734aed56-workload-certs\") pod \"e2e-shared-b-w2-255f05-openshift-default-7db8bf49f-6bfml\" (UID: \"4955d910-8544-4a2b-a1fd-b414734aed56\") " pod="openshift-ingress/e2e-shared-b-w2-255f05-openshift-default-7db8bf49f-6bfml" Oct 30 21:06:44.608327 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:06:44.608302 5378 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4955d910-8544-4a2b-a1fd-b414734aed56-workload-socket\") pod \"e2e-shared-b-w2-255f05-openshift-default-7db8bf49f-6bfml\" (UID: \"4955d910-8544-4a2b-a1fd-b414734aed56\") " pod="openshift-ingress/e2e-shared-b-w2-255f05-openshift-default-7db8bf49f-6bfml" Oct 30 21:06:44.608397 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:06:44.608381 5378 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4955d910-8544-4a2b-a1fd-b414734aed56-istio-envoy\") pod \"e2e-shared-b-w2-255f05-openshift-default-7db8bf49f-6bfml\" (UID: \"4955d910-8544-4a2b-a1fd-b414734aed56\") " pod="openshift-ingress/e2e-shared-b-w2-255f05-openshift-default-7db8bf49f-6bfml" Oct 30 21:06:44.608487 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:06:44.608466 5378 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4955d910-8544-4a2b-a1fd-b414734aed56-istio-data\") pod \"e2e-shared-b-w2-255f05-openshift-default-7db8bf49f-6bfml\" (UID: \"4955d910-8544-4a2b-a1fd-b414734aed56\") " pod="openshift-ingress/e2e-shared-b-w2-255f05-openshift-default-7db8bf49f-6bfml" Oct 30 21:06:44.608526 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:06:44.608496 5378 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4955d910-8544-4a2b-a1fd-b414734aed56-istiod-ca-cert\") pod \"e2e-shared-b-w2-255f05-openshift-default-7db8bf49f-6bfml\" (UID: \"4955d910-8544-4a2b-a1fd-b414734aed56\") " pod="openshift-ingress/e2e-shared-b-w2-255f05-openshift-default-7db8bf49f-6bfml" Oct 30 21:06:44.608580 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:06:44.608536 5378 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9mtg6\" (UniqueName: \"kubernetes.io/projected/4955d910-8544-4a2b-a1fd-b414734aed56-kube-api-access-9mtg6\") pod \"e2e-shared-b-w2-255f05-openshift-default-7db8bf49f-6bfml\" (UID: \"4955d910-8544-4a2b-a1fd-b414734aed56\") " pod="openshift-ingress/e2e-shared-b-w2-255f05-openshift-default-7db8bf49f-6bfml" Oct 30 21:06:44.723587 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:06:44.719988 5378 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4955d910-8544-4a2b-a1fd-b414734aed56-istio-token\") pod \"e2e-shared-b-w2-255f05-openshift-default-7db8bf49f-6bfml\" (UID: \"4955d910-8544-4a2b-a1fd-b414734aed56\") " pod="openshift-ingress/e2e-shared-b-w2-255f05-openshift-default-7db8bf49f-6bfml" Oct 30 21:06:44.723587 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:06:44.720061 5378 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4955d910-8544-4a2b-a1fd-b414734aed56-workload-certs\") pod \"e2e-shared-b-w2-255f05-openshift-default-7db8bf49f-6bfml\" (UID: \"4955d910-8544-4a2b-a1fd-b414734aed56\") " pod="openshift-ingress/e2e-shared-b-w2-255f05-openshift-default-7db8bf49f-6bfml" Oct 30 21:06:44.723587 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:06:44.720087 5378 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4955d910-8544-4a2b-a1fd-b414734aed56-workload-socket\") pod \"e2e-shared-b-w2-255f05-openshift-default-7db8bf49f-6bfml\" (UID: \"4955d910-8544-4a2b-a1fd-b414734aed56\") " pod="openshift-ingress/e2e-shared-b-w2-255f05-openshift-default-7db8bf49f-6bfml" Oct 30 21:06:44.723587 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:06:44.720140 5378 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4955d910-8544-4a2b-a1fd-b414734aed56-istio-envoy\") pod \"e2e-shared-b-w2-255f05-openshift-default-7db8bf49f-6bfml\" (UID: \"4955d910-8544-4a2b-a1fd-b414734aed56\") " pod="openshift-ingress/e2e-shared-b-w2-255f05-openshift-default-7db8bf49f-6bfml" Oct 30 21:06:44.723587 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:06:44.720216 5378 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4955d910-8544-4a2b-a1fd-b414734aed56-istio-data\") pod \"e2e-shared-b-w2-255f05-openshift-default-7db8bf49f-6bfml\" (UID: \"4955d910-8544-4a2b-a1fd-b414734aed56\") " pod="openshift-ingress/e2e-shared-b-w2-255f05-openshift-default-7db8bf49f-6bfml" Oct 30 21:06:44.723587 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:06:44.720239 5378 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4955d910-8544-4a2b-a1fd-b414734aed56-istiod-ca-cert\") pod \"e2e-shared-b-w2-255f05-openshift-default-7db8bf49f-6bfml\" (UID: \"4955d910-8544-4a2b-a1fd-b414734aed56\") " pod="openshift-ingress/e2e-shared-b-w2-255f05-openshift-default-7db8bf49f-6bfml" Oct 30 21:06:44.723587 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:06:44.720299 5378 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9mtg6\" (UniqueName: \"kubernetes.io/projected/4955d910-8544-4a2b-a1fd-b414734aed56-kube-api-access-9mtg6\") pod \"e2e-shared-b-w2-255f05-openshift-default-7db8bf49f-6bfml\" (UID: \"4955d910-8544-4a2b-a1fd-b414734aed56\") " pod="openshift-ingress/e2e-shared-b-w2-255f05-openshift-default-7db8bf49f-6bfml" Oct 30 21:06:44.723587 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:06:44.720388 5378 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4955d910-8544-4a2b-a1fd-b414734aed56-istio-podinfo\") pod \"e2e-shared-b-w2-255f05-openshift-default-7db8bf49f-6bfml\" (UID: \"4955d910-8544-4a2b-a1fd-b414734aed56\") " pod="openshift-ingress/e2e-shared-b-w2-255f05-openshift-default-7db8bf49f-6bfml" Oct 30 21:06:44.723587 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:06:44.720488 5378 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4955d910-8544-4a2b-a1fd-b414734aed56-credential-socket\") pod \"e2e-shared-b-w2-255f05-openshift-default-7db8bf49f-6bfml\" (UID: \"4955d910-8544-4a2b-a1fd-b414734aed56\") " pod="openshift-ingress/e2e-shared-b-w2-255f05-openshift-default-7db8bf49f-6bfml" Oct 30 21:06:44.723587 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:06:44.721062 5378 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4955d910-8544-4a2b-a1fd-b414734aed56-credential-socket\") pod \"e2e-shared-b-w2-255f05-openshift-default-7db8bf49f-6bfml\" (UID: \"4955d910-8544-4a2b-a1fd-b414734aed56\") " pod="openshift-ingress/e2e-shared-b-w2-255f05-openshift-default-7db8bf49f-6bfml" Oct 30 21:06:44.723587 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:06:44.721650 5378 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4955d910-8544-4a2b-a1fd-b414734aed56-workload-certs\") pod \"e2e-shared-b-w2-255f05-openshift-default-7db8bf49f-6bfml\" (UID: \"4955d910-8544-4a2b-a1fd-b414734aed56\") " pod="openshift-ingress/e2e-shared-b-w2-255f05-openshift-default-7db8bf49f-6bfml" Oct 30 21:06:44.723587 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:06:44.721884 5378 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4955d910-8544-4a2b-a1fd-b414734aed56-workload-socket\") pod \"e2e-shared-b-w2-255f05-openshift-default-7db8bf49f-6bfml\" (UID: \"4955d910-8544-4a2b-a1fd-b414734aed56\") " pod="openshift-ingress/e2e-shared-b-w2-255f05-openshift-default-7db8bf49f-6bfml" Oct 30 21:06:44.769271 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:06:44.726141 5378 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4955d910-8544-4a2b-a1fd-b414734aed56-istiod-ca-cert\") pod \"e2e-shared-b-w2-255f05-openshift-default-7db8bf49f-6bfml\" (UID: \"4955d910-8544-4a2b-a1fd-b414734aed56\") " pod="openshift-ingress/e2e-shared-b-w2-255f05-openshift-default-7db8bf49f-6bfml" Oct 30 21:06:44.790131 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:06:44.734984 5378 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4955d910-8544-4a2b-a1fd-b414734aed56-istio-data\") pod \"e2e-shared-b-w2-255f05-openshift-default-7db8bf49f-6bfml\" (UID: \"4955d910-8544-4a2b-a1fd-b414734aed56\") " pod="openshift-ingress/e2e-shared-b-w2-255f05-openshift-default-7db8bf49f-6bfml" Oct 30 21:06:44.790363 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:06:44.749897 5378 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4955d910-8544-4a2b-a1fd-b414734aed56-istio-envoy\") pod \"e2e-shared-b-w2-255f05-openshift-default-7db8bf49f-6bfml\" (UID: \"4955d910-8544-4a2b-a1fd-b414734aed56\") " pod="openshift-ingress/e2e-shared-b-w2-255f05-openshift-default-7db8bf49f-6bfml" Oct 30 21:06:44.790492 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:06:44.750180 5378 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4955d910-8544-4a2b-a1fd-b414734aed56-istio-podinfo\") pod \"e2e-shared-b-w2-255f05-openshift-default-7db8bf49f-6bfml\" (UID: \"4955d910-8544-4a2b-a1fd-b414734aed56\") " pod="openshift-ingress/e2e-shared-b-w2-255f05-openshift-default-7db8bf49f-6bfml" Oct 30 21:06:44.797290 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:06:44.797259 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:06:44.812300 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:06:44.812266 5378 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4955d910-8544-4a2b-a1fd-b414734aed56-istio-token\") pod \"e2e-shared-b-w2-255f05-openshift-default-7db8bf49f-6bfml\" (UID: \"4955d910-8544-4a2b-a1fd-b414734aed56\") " pod="openshift-ingress/e2e-shared-b-w2-255f05-openshift-default-7db8bf49f-6bfml" Oct 30 21:06:44.830692 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:06:44.830629 5378 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9mtg6\" (UniqueName: \"kubernetes.io/projected/4955d910-8544-4a2b-a1fd-b414734aed56-kube-api-access-9mtg6\") pod \"e2e-shared-b-w2-255f05-openshift-default-7db8bf49f-6bfml\" (UID: \"4955d910-8544-4a2b-a1fd-b414734aed56\") " pod="openshift-ingress/e2e-shared-b-w2-255f05-openshift-default-7db8bf49f-6bfml" Oct 30 21:06:45.114820 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:06:45.114778 5378 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-255f05-openshift-default-7db8bf49f-6bfml" Oct 30 21:06:45.128518 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:06:45.128489 5378 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-dc6125-74d649bf67-vlsfb" event={"ID":"5ff92f96-e5bc-4b04-893b-16efd49f9d3f","Type":"ContainerStarted","Data":"119bfe8aa3cd04e4d95f140e1a87b0542f4940384e5eb1292d5ecffb20e995bf"} Oct 30 21:06:45.128660 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:06:45.128549 5378 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-dc6125-74d649bf67-vlsfb" event={"ID":"5ff92f96-e5bc-4b04-893b-16efd49f9d3f","Type":"ContainerStarted","Data":"0cc8cede770432d7556933712d081b303ad2fa199c0028d5af935a277f65d762"} Oct 30 21:06:45.128840 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:06:45.128667 5378 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-dc6125-74d649bf67-vlsfb" Oct 30 21:06:45.130287 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:06:45.130260 5378 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w6-dc6125-6644bb494b-2v578" event={"ID":"168599a3-7daf-4eba-a259-978486b56bd4","Type":"ContainerStarted","Data":"a087ee550d15f63a1420aec405fb77c4bbdd4785f2dc958ce88cf68789e9763c"} Oct 30 21:06:45.130422 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:06:45.130297 5378 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w6-dc6125-6644bb494b-2v578" event={"ID":"168599a3-7daf-4eba-a259-978486b56bd4","Type":"ContainerStarted","Data":"bbaca4227c2b51fab16e5f55f3e6e891bc90f96c1c661b63070c28d78b7f2aed"} Oct 30 21:06:45.130422 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:06:45.130358 5378 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w6-dc6125-6644bb494b-2v578" Oct 30 21:06:45.256060 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:06:45.256006 5378 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-dc6125-74d649bf67-vlsfb" podStartSLOduration=2.255990617 podStartE2EDuration="2.255990617s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:06:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:06:45.252726048 +0000 UTC m=+1695.680839991" watchObservedRunningTime="2026-10-30 21:06:45.255990617 +0000 UTC m=+1695.684104570" Oct 30 21:06:45.402013 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:06:45.391240 5378 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-a-w6-dc6125-6644bb494b-2v578" podStartSLOduration=2.39121802 podStartE2EDuration="2.39121802s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:06:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:06:45.385632438 +0000 UTC m=+1695.813746357" watchObservedRunningTime="2026-10-30 21:06:45.39121802 +0000 UTC m=+1695.819331941" Oct 30 21:06:45.537586 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:06:45.528904 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:06:45.611546 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:06:45.611511 5378 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-255f05-openshift-default-7db8bf49f-6bfml"] Oct 30 21:06:45.611938 ip-10-0-138-149 kubenswrapper[5378]: W1030 21:06:45.611916 5378 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4955d910_8544_4a2b_a1fd_b414734aed56.slice/crio-60f7f7e917a7eacc2f8b89615553e3b3091aa25dbcb5915b630b570d9cc3b751 WatchSource:0}: Error finding container 60f7f7e917a7eacc2f8b89615553e3b3091aa25dbcb5915b630b570d9cc3b751: Status 404 returned error can't find the container with id 60f7f7e917a7eacc2f8b89615553e3b3091aa25dbcb5915b630b570d9cc3b751 Oct 30 21:06:45.614745 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:06:45.614709 5378 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 21:06:45.614821 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:06:45.614784 5378 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 21:06:45.614869 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:06:45.614821 5378 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 21:06:45.650728 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:06:45.650702 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:06:45.746901 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:06:45.746869 5378 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="88bbd407-197c-4ba6-b31f-fc097ec5fbf3" path="/var/lib/kubelet/pods/88bbd407-197c-4ba6-b31f-fc097ec5fbf3/volumes" Oct 30 21:06:45.747517 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:06:45.747494 5378 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="eafa34dc-6d4b-4507-a2a3-989ad8cf7de2" path="/var/lib/kubelet/pods/eafa34dc-6d4b-4507-a2a3-989ad8cf7de2/volumes" Oct 30 21:06:46.139048 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:06:46.139009 5378 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-255f05-openshift-default-7db8bf49f-6bfml" event={"ID":"4955d910-8544-4a2b-a1fd-b414734aed56","Type":"ContainerStarted","Data":"f436cafc6a27c4e22cde96a1784d61e2c40ce352ae7696fc0974d6435e60a1bf"} Oct 30 21:06:46.139474 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:06:46.139068 5378 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-255f05-openshift-default-7db8bf49f-6bfml" event={"ID":"4955d910-8544-4a2b-a1fd-b414734aed56","Type":"ContainerStarted","Data":"60f7f7e917a7eacc2f8b89615553e3b3091aa25dbcb5915b630b570d9cc3b751"} Oct 30 21:06:46.258720 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:06:46.258667 5378 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w2-255f05-openshift-default-7db8bf49f-6bfml" podStartSLOduration=2.258653853 podStartE2EDuration="2.258653853s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:06:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:06:46.256156275 +0000 UTC m=+1696.684270194" watchObservedRunningTime="2026-10-30 21:06:46.258653853 +0000 UTC m=+1696.686767771" Oct 30 21:06:47.115942 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:06:47.115912 5378 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w2-255f05-openshift-default-7db8bf49f-6bfml" Oct 30 21:06:47.122047 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:06:47.122018 5378 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w2-255f05-openshift-default-7db8bf49f-6bfml" Oct 30 21:06:47.145868 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:06:47.145842 5378 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w2-255f05-openshift-default-7db8bf49f-6bfml" Oct 30 21:06:47.146744 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:06:47.146725 5378 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w2-255f05-openshift-default-7db8bf49f-6bfml" Oct 30 21:06:48.109520 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:06:48.109489 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:06:48.838250 ip-10-0-138-149 kubenswrapper[5378]: E1030 21:06:48.838213 5378 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q6mx7" containerName="extractor" Oct 30 21:06:49.587418 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:06:49.587389 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:06:49.792242 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:06:49.792206 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:06:49.925605 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:06:49.925537 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:06:50.667076 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:06:50.667043 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:06:50.869839 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:06:50.869808 5378 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-88da25-5ddcdff7f9-5p4dg" Oct 30 21:06:51.566621 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:06:51.566556 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:06:51.875467 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:06:51.875436 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:06:52.193501 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:06:52.193370 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:06:52.764967 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:06:52.764934 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:06:52.951015 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:06:52.941072 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:06:53.200332 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:06:53.200297 5378 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w1-4d1110-5cfd7bf99f-mxznk"] Oct 30 21:06:53.205837 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:06:53.205805 5378 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-4d1110-5cfd7bf99f-mxznk" Oct 30 21:06:53.231339 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:06:53.231300 5378 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b8b5625b-b0ec-4e9c-8b80-338ea02008f9-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w1-4d1110-5cfd7bf99f-mxznk\" (UID: \"b8b5625b-b0ec-4e9c-8b80-338ea02008f9\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-4d1110-5cfd7bf99f-mxznk" Oct 30 21:06:53.231695 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:06:53.231669 5378 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sg5fl\" (UniqueName: \"kubernetes.io/projected/b8b5625b-b0ec-4e9c-8b80-338ea02008f9-kube-api-access-sg5fl\") pod \"payload-pre-processing-e2e-shared-a-w1-4d1110-5cfd7bf99f-mxznk\" (UID: \"b8b5625b-b0ec-4e9c-8b80-338ea02008f9\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-4d1110-5cfd7bf99f-mxznk" Oct 30 21:06:53.263212 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:06:53.263168 5378 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w1-4d1110-5cfd7bf99f-mxznk"] Oct 30 21:06:53.312588 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:06:53.304022 5378 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w1-4d1110\"" Oct 30 21:06:53.312588 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:06:53.304199 5378 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w1-4d1110-dockercfg-szjp2\"" Oct 30 21:06:53.463752 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:06:53.463665 5378 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w1-4d1110-694854c68-k9bh2"] Oct 30 21:06:53.474816 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:06:53.471896 5378 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b8b5625b-b0ec-4e9c-8b80-338ea02008f9-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w1-4d1110-5cfd7bf99f-mxznk\" (UID: \"b8b5625b-b0ec-4e9c-8b80-338ea02008f9\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-4d1110-5cfd7bf99f-mxznk" Oct 30 21:06:53.474816 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:06:53.472054 5378 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sg5fl\" (UniqueName: \"kubernetes.io/projected/b8b5625b-b0ec-4e9c-8b80-338ea02008f9-kube-api-access-sg5fl\") pod \"payload-pre-processing-e2e-shared-a-w1-4d1110-5cfd7bf99f-mxznk\" (UID: \"b8b5625b-b0ec-4e9c-8b80-338ea02008f9\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-4d1110-5cfd7bf99f-mxznk" Oct 30 21:06:53.474816 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:06:53.473322 5378 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b8b5625b-b0ec-4e9c-8b80-338ea02008f9-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w1-4d1110-5cfd7bf99f-mxznk\" (UID: \"b8b5625b-b0ec-4e9c-8b80-338ea02008f9\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-4d1110-5cfd7bf99f-mxznk" Oct 30 21:06:53.482890 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:06:53.482864 5378 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w1-4d1110-694854c68-k9bh2"] Oct 30 21:06:53.483210 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:06:53.483190 5378 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-4d1110-694854c68-k9bh2" Oct 30 21:06:53.548288 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:06:53.548149 5378 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sg5fl\" (UniqueName: \"kubernetes.io/projected/b8b5625b-b0ec-4e9c-8b80-338ea02008f9-kube-api-access-sg5fl\") pod \"payload-pre-processing-e2e-shared-a-w1-4d1110-5cfd7bf99f-mxznk\" (UID: \"b8b5625b-b0ec-4e9c-8b80-338ea02008f9\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-4d1110-5cfd7bf99f-mxznk" Oct 30 21:06:53.566301 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:06:53.566267 5378 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-4d1110-5cfd7bf99f-mxznk" Oct 30 21:06:53.646805 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:06:53.646257 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:06:53.674614 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:06:53.674551 5378 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5a5302ee-2e3c-427b-a222-940f0a913372-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w1-4d1110-694854c68-k9bh2\" (UID: \"5a5302ee-2e3c-427b-a222-940f0a913372\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w1-4d1110-694854c68-k9bh2" Oct 30 21:06:53.674810 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:06:53.674782 5378 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v45ps\" (UniqueName: \"kubernetes.io/projected/5a5302ee-2e3c-427b-a222-940f0a913372-kube-api-access-v45ps\") pod \"payload-processing-e2e-shared-a-w1-4d1110-694854c68-k9bh2\" (UID: \"5a5302ee-2e3c-427b-a222-940f0a913372\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w1-4d1110-694854c68-k9bh2" Oct 30 21:06:53.776085 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:06:53.775995 5378 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5a5302ee-2e3c-427b-a222-940f0a913372-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w1-4d1110-694854c68-k9bh2\" (UID: \"5a5302ee-2e3c-427b-a222-940f0a913372\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w1-4d1110-694854c68-k9bh2" Oct 30 21:06:53.776745 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:06:53.776715 5378 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v45ps\" (UniqueName: \"kubernetes.io/projected/5a5302ee-2e3c-427b-a222-940f0a913372-kube-api-access-v45ps\") pod \"payload-processing-e2e-shared-a-w1-4d1110-694854c68-k9bh2\" (UID: \"5a5302ee-2e3c-427b-a222-940f0a913372\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w1-4d1110-694854c68-k9bh2" Oct 30 21:06:53.778131 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:06:53.778106 5378 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5a5302ee-2e3c-427b-a222-940f0a913372-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w1-4d1110-694854c68-k9bh2\" (UID: \"5a5302ee-2e3c-427b-a222-940f0a913372\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w1-4d1110-694854c68-k9bh2" Oct 30 21:06:53.834106 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:06:53.834079 5378 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v45ps\" (UniqueName: \"kubernetes.io/projected/5a5302ee-2e3c-427b-a222-940f0a913372-kube-api-access-v45ps\") pod \"payload-processing-e2e-shared-a-w1-4d1110-694854c68-k9bh2\" (UID: \"5a5302ee-2e3c-427b-a222-940f0a913372\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w1-4d1110-694854c68-k9bh2" Oct 30 21:06:53.946371 ip-10-0-138-149 kubenswrapper[5378]: W1030 21:06:53.946338 5378 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb8b5625b_b0ec_4e9c_8b80_338ea02008f9.slice/crio-140b404b03f8cb4536b436f62bc5f571b930c0eb9793ffead0af2a645c73cd93 WatchSource:0}: Error finding container 140b404b03f8cb4536b436f62bc5f571b930c0eb9793ffead0af2a645c73cd93: Status 404 returned error can't find the container with id 140b404b03f8cb4536b436f62bc5f571b930c0eb9793ffead0af2a645c73cd93 Oct 30 21:06:53.946985 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:06:53.946952 5378 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w1-4d1110-5cfd7bf99f-mxznk"] Oct 30 21:06:54.125122 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:06:54.125092 5378 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-4d1110-694854c68-k9bh2" Oct 30 21:06:54.254255 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:06:54.254214 5378 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-4d1110-5cfd7bf99f-mxznk" event={"ID":"b8b5625b-b0ec-4e9c-8b80-338ea02008f9","Type":"ContainerStarted","Data":"a6bf94a3efd1b7f0fb91c26f016b4d32f36dadb7c5e68d4c5eec54334139837d"} Oct 30 21:06:54.254455 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:06:54.254292 5378 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-4d1110-5cfd7bf99f-mxznk" Oct 30 21:06:54.254455 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:06:54.254323 5378 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-4d1110-5cfd7bf99f-mxznk" event={"ID":"b8b5625b-b0ec-4e9c-8b80-338ea02008f9","Type":"ContainerStarted","Data":"140b404b03f8cb4536b436f62bc5f571b930c0eb9793ffead0af2a645c73cd93"} Oct 30 21:06:54.372421 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:06:54.372356 5378 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-4d1110-5cfd7bf99f-mxznk" podStartSLOduration=1.372340858 podStartE2EDuration="1.372340858s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:06:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:06:54.370272312 +0000 UTC m=+1704.798386230" watchObservedRunningTime="2026-10-30 21:06:54.372340858 +0000 UTC m=+1704.800454815" Oct 30 21:06:54.510690 ip-10-0-138-149 kubenswrapper[5378]: W1030 21:06:54.509512 5378 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5a5302ee_2e3c_427b_a222_940f0a913372.slice/crio-7e7805d35b79f061e22ef571b2c514ae77e5e26d807d266e4ee008bf4529ead9 WatchSource:0}: Error finding container 7e7805d35b79f061e22ef571b2c514ae77e5e26d807d266e4ee008bf4529ead9: Status 404 returned error can't find the container with id 7e7805d35b79f061e22ef571b2c514ae77e5e26d807d266e4ee008bf4529ead9 Oct 30 21:06:54.510859 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:06:54.510687 5378 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w1-4d1110-694854c68-k9bh2"] Oct 30 21:06:55.263663 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:06:55.263628 5378 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-4d1110-694854c68-k9bh2" event={"ID":"5a5302ee-2e3c-427b-a222-940f0a913372","Type":"ContainerStarted","Data":"cc2bd4ae2f1dd4e16538a348649f0aa24a6da5f156470f318aeced97ed8f43a5"} Oct 30 21:06:55.264051 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:06:55.263725 5378 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-4d1110-694854c68-k9bh2" Oct 30 21:06:55.264051 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:06:55.263754 5378 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-4d1110-694854c68-k9bh2" event={"ID":"5a5302ee-2e3c-427b-a222-940f0a913372","Type":"ContainerStarted","Data":"7e7805d35b79f061e22ef571b2c514ae77e5e26d807d266e4ee008bf4529ead9"} Oct 30 21:06:55.351201 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:06:55.351166 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:06:55.382485 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:06:55.382428 5378 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-4d1110-694854c68-k9bh2" podStartSLOduration=2.382412935 podStartE2EDuration="2.382412935s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:06:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:06:55.380445487 +0000 UTC m=+1705.808559408" watchObservedRunningTime="2026-10-30 21:06:55.382412935 +0000 UTC m=+1705.810526877" Oct 30 21:06:55.524017 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:06:55.523941 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:06:56.142129 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:06:56.142097 5378 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w6-dc6125-6644bb494b-2v578" Oct 30 21:06:56.142301 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:06:56.142182 5378 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-dc6125-74d649bf67-vlsfb" Oct 30 21:06:57.956590 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:06:57.953306 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:06:58.139542 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:06:58.139509 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:06:58.584066 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:06:58.584032 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:06:59.162929 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:06:59.162892 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:07:04.375882 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:07:04.375850 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:07:04.550213 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:07:04.550171 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:07:05.267704 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:07:05.267671 5378 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-4d1110-5cfd7bf99f-mxznk" Oct 30 21:07:06.276345 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:07:06.276311 5378 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-4d1110-694854c68-k9bh2" Oct 30 21:07:09.635483 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:07:09.635402 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:07:10.757938 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:07:10.757910 5378 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-19db3d5f-744ff4fb64-blk7k"] Oct 30 21:07:11.135486 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:07:11.135451 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:07:11.321193 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:07:11.321124 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:07:14.961502 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:07:14.961447 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:07:15.758514 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:07:15.758472 5378 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-mt-19db3d5f-744ff4fb64-blk7k" podUID="0c2654e1-7e67-4697-9613-4fa0aa8aa5d1" containerName="payload-processing" containerID="cri-o://f1e677937bfb755acbf760ddb590dfec3cc3919e2745531f12f31250f772bb79" gracePeriod=25 Oct 30 21:07:16.045246 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:07:16.045218 5378 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-19db3d5f-744ff4fb64-blk7k" Oct 30 21:07:16.213030 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:07:16.212999 5378 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0c2654e1-7e67-4697-9613-4fa0aa8aa5d1-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0c2654e1-7e67-4697-9613-4fa0aa8aa5d1-plugins-config-volume\") pod \"0c2654e1-7e67-4697-9613-4fa0aa8aa5d1\" (UID: \"0c2654e1-7e67-4697-9613-4fa0aa8aa5d1\") " Oct 30 21:07:16.213219 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:07:16.213068 5378 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0c2654e1-7e67-4697-9613-4fa0aa8aa5d1-kube-api-access-lbj2k\" (UniqueName: \"kubernetes.io/projected/0c2654e1-7e67-4697-9613-4fa0aa8aa5d1-kube-api-access-lbj2k\") pod \"0c2654e1-7e67-4697-9613-4fa0aa8aa5d1\" (UID: \"0c2654e1-7e67-4697-9613-4fa0aa8aa5d1\") " Oct 30 21:07:16.213389 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:07:16.213362 5378 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0c2654e1-7e67-4697-9613-4fa0aa8aa5d1-plugins-config-volume" pod "0c2654e1-7e67-4697-9613-4fa0aa8aa5d1" (UID: "0c2654e1-7e67-4697-9613-4fa0aa8aa5d1"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:07:16.213532 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:07:16.213518 5378 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0c2654e1-7e67-4697-9613-4fa0aa8aa5d1-plugins-config-volume\") on node \"ip-10-0-138-149.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:07:16.215169 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:07:16.215149 5378 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0c2654e1-7e67-4697-9613-4fa0aa8aa5d1-kube-api-access-lbj2k" pod "0c2654e1-7e67-4697-9613-4fa0aa8aa5d1" (UID: "0c2654e1-7e67-4697-9613-4fa0aa8aa5d1"). InnerVolumeSpecName "kube-api-access-lbj2k". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:07:16.314618 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:07:16.314494 5378 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lbj2k\" (UniqueName: \"kubernetes.io/projected/0c2654e1-7e67-4697-9613-4fa0aa8aa5d1-kube-api-access-lbj2k\") on node \"ip-10-0-138-149.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:07:16.473990 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:07:16.473948 5378 generic.go:356] "Generic (PLEG): container finished" podID="0c2654e1-7e67-4697-9613-4fa0aa8aa5d1" containerID="f1e677937bfb755acbf760ddb590dfec3cc3919e2745531f12f31250f772bb79" exitCode=0 Oct 30 21:07:16.474166 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:07:16.474001 5378 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-19db3d5f-744ff4fb64-blk7k" Oct 30 21:07:16.474166 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:07:16.474032 5378 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-19db3d5f-744ff4fb64-blk7k" event={"ID":"0c2654e1-7e67-4697-9613-4fa0aa8aa5d1","Type":"ContainerDied","Data":"f1e677937bfb755acbf760ddb590dfec3cc3919e2745531f12f31250f772bb79"} Oct 30 21:07:16.474166 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:07:16.474120 5378 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-19db3d5f-744ff4fb64-blk7k" event={"ID":"0c2654e1-7e67-4697-9613-4fa0aa8aa5d1","Type":"ContainerDied","Data":"d1c05bc2b0b73b7d7262ff3466dd9984175d71a81f8c335f383cad66f6f8eff3"} Oct 30 21:07:16.474166 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:07:16.474154 5378 scope.go:122] "RemoveContainer" containerID="f1e677937bfb755acbf760ddb590dfec3cc3919e2745531f12f31250f772bb79" Oct 30 21:07:16.485537 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:07:16.485521 5378 scope.go:122] "RemoveContainer" containerID="f1e677937bfb755acbf760ddb590dfec3cc3919e2745531f12f31250f772bb79" Oct 30 21:07:16.485818 ip-10-0-138-149 kubenswrapper[5378]: E1030 21:07:16.485796 5378 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f1e677937bfb755acbf760ddb590dfec3cc3919e2745531f12f31250f772bb79\": container with ID starting with f1e677937bfb755acbf760ddb590dfec3cc3919e2745531f12f31250f772bb79 not found: ID does not exist" containerID="f1e677937bfb755acbf760ddb590dfec3cc3919e2745531f12f31250f772bb79" Oct 30 21:07:16.485870 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:07:16.485825 5378 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f1e677937bfb755acbf760ddb590dfec3cc3919e2745531f12f31250f772bb79"} err="failed to get container status \"f1e677937bfb755acbf760ddb590dfec3cc3919e2745531f12f31250f772bb79\": rpc error: code = NotFound desc = could not find container \"f1e677937bfb755acbf760ddb590dfec3cc3919e2745531f12f31250f772bb79\": container with ID starting with f1e677937bfb755acbf760ddb590dfec3cc3919e2745531f12f31250f772bb79 not found: ID does not exist" Oct 30 21:07:16.645017 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:07:16.644991 5378 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-19db3d5f-744ff4fb64-blk7k"] Oct 30 21:07:16.695320 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:07:16.695290 5378 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-19db3d5f-744ff4fb64-blk7k"] Oct 30 21:07:17.743241 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:07:17.743209 5378 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0c2654e1-7e67-4697-9613-4fa0aa8aa5d1" path="/var/lib/kubelet/pods/0c2654e1-7e67-4697-9613-4fa0aa8aa5d1/volumes" Oct 30 21:07:17.811678 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:07:17.811652 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:07:17.932633 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:07:17.932603 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:07:18.617560 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:07:18.617529 5378 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-r9g59"] Oct 30 21:07:18.839160 ip-10-0-138-149 kubenswrapper[5378]: E1030 21:07:18.839124 5378 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q6mx7" containerName="extractor" Oct 30 21:07:20.414165 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:07:20.414126 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:07:21.334357 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:07:21.334312 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:07:25.775281 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:07:25.775248 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:07:30.623481 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:07:30.623449 5378 scope.go:122] "RemoveContainer" containerID="3a7bffd1f19133cb813a22d614b54d28e11d8c3bf803b1a620bdfec52c6027dc" Oct 30 21:07:30.635784 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:07:30.635757 5378 scope.go:122] "RemoveContainer" containerID="f9515dce096602d9ad8d96b2c07ead6895651c7ccc1299d7116c10fac676013a" Oct 30 21:07:31.067908 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:07:31.067825 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:07:35.725915 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:07:35.725875 5378 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-88da25-785647f46c-rb4g9"] Oct 30 21:07:35.726665 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:07:35.726649 5378 cpu_manager.go:414] "containerMap: removing container" podUID="0c2654e1-7e67-4697-9613-4fa0aa8aa5d1" containerName="payload-processing" Oct 30 21:07:35.726727 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:07:35.726667 5378 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0c2654e1-7e67-4697-9613-4fa0aa8aa5d1" containerName="payload-processing" Oct 30 21:07:35.726797 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:07:35.726787 5378 memory_manager.go:365] "RemoveStaleState removing state" podUID="0c2654e1-7e67-4697-9613-4fa0aa8aa5d1" containerName="payload-processing" Oct 30 21:07:35.733093 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:07:35.733072 5378 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-88da25-785647f46c-rb4g9" Oct 30 21:07:35.790497 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:07:35.790471 5378 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-w4-88da25-serving-cert\"" Oct 30 21:07:35.830147 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:07:35.830112 5378 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a9c904b3-edb2-426a-ac6a-136a4e6afba0-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w4-88da25-785647f46c-rb4g9\" (UID: \"a9c904b3-edb2-426a-ac6a-136a4e6afba0\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-88da25-785647f46c-rb4g9" Oct 30 21:07:35.830836 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:07:35.830219 5378 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a9c904b3-edb2-426a-ac6a-136a4e6afba0-metrics-certs\") pod \"maas-api-e2e-shared-b-w4-88da25-785647f46c-rb4g9\" (UID: \"a9c904b3-edb2-426a-ac6a-136a4e6afba0\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-88da25-785647f46c-rb4g9" Oct 30 21:07:35.830836 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:07:35.830507 5378 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a9c904b3-edb2-426a-ac6a-136a4e6afba0-maas-api-tls\") pod \"maas-api-e2e-shared-b-w4-88da25-785647f46c-rb4g9\" (UID: \"a9c904b3-edb2-426a-ac6a-136a4e6afba0\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-88da25-785647f46c-rb4g9" Oct 30 21:07:35.830993 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:07:35.830880 5378 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pjxpp\" (UniqueName: \"kubernetes.io/projected/a9c904b3-edb2-426a-ac6a-136a4e6afba0-kube-api-access-pjxpp\") pod \"maas-api-e2e-shared-b-w4-88da25-785647f46c-rb4g9\" (UID: \"a9c904b3-edb2-426a-ac6a-136a4e6afba0\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-88da25-785647f46c-rb4g9" Oct 30 21:07:35.846975 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:07:35.846949 5378 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-88da25-785647f46c-rb4g9"] Oct 30 21:07:35.890580 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:07:35.890540 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:07:35.932251 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:07:35.932228 5378 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a9c904b3-edb2-426a-ac6a-136a4e6afba0-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w4-88da25-785647f46c-rb4g9\" (UID: \"a9c904b3-edb2-426a-ac6a-136a4e6afba0\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-88da25-785647f46c-rb4g9" Oct 30 21:07:35.932391 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:07:35.932269 5378 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a9c904b3-edb2-426a-ac6a-136a4e6afba0-metrics-certs\") pod \"maas-api-e2e-shared-b-w4-88da25-785647f46c-rb4g9\" (UID: \"a9c904b3-edb2-426a-ac6a-136a4e6afba0\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-88da25-785647f46c-rb4g9" Oct 30 21:07:35.932391 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:07:35.932332 5378 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a9c904b3-edb2-426a-ac6a-136a4e6afba0-maas-api-tls\") pod \"maas-api-e2e-shared-b-w4-88da25-785647f46c-rb4g9\" (UID: \"a9c904b3-edb2-426a-ac6a-136a4e6afba0\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-88da25-785647f46c-rb4g9" Oct 30 21:07:35.932391 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:07:35.932370 5378 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pjxpp\" (UniqueName: \"kubernetes.io/projected/a9c904b3-edb2-426a-ac6a-136a4e6afba0-kube-api-access-pjxpp\") pod \"maas-api-e2e-shared-b-w4-88da25-785647f46c-rb4g9\" (UID: \"a9c904b3-edb2-426a-ac6a-136a4e6afba0\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-88da25-785647f46c-rb4g9" Oct 30 21:07:35.932619 ip-10-0-138-149 kubenswrapper[5378]: E1030 21:07:35.932453 5378 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-88da25-serving-cert: secret "maas-api-e2e-shared-b-w4-88da25-serving-cert" not found Oct 30 21:07:35.932619 ip-10-0-138-149 kubenswrapper[5378]: E1030 21:07:35.932543 5378 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a9c904b3-edb2-426a-ac6a-136a4e6afba0-maas-api-tls podName:a9c904b3-edb2-426a-ac6a-136a4e6afba0 nodeName:}" failed. No retries permitted until 2026-10-30 21:07:36.432522091 +0000 UTC m=+1746.860636009 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/a9c904b3-edb2-426a-ac6a-136a4e6afba0-maas-api-tls") pod "maas-api-e2e-shared-b-w4-88da25-785647f46c-rb4g9" (UID: "a9c904b3-edb2-426a-ac6a-136a4e6afba0") : secret "maas-api-e2e-shared-b-w4-88da25-serving-cert" not found Oct 30 21:07:35.932971 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:07:35.932950 5378 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a9c904b3-edb2-426a-ac6a-136a4e6afba0-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w4-88da25-785647f46c-rb4g9\" (UID: \"a9c904b3-edb2-426a-ac6a-136a4e6afba0\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-88da25-785647f46c-rb4g9" Oct 30 21:07:35.934880 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:07:35.934862 5378 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a9c904b3-edb2-426a-ac6a-136a4e6afba0-metrics-certs\") pod \"maas-api-e2e-shared-b-w4-88da25-785647f46c-rb4g9\" (UID: \"a9c904b3-edb2-426a-ac6a-136a4e6afba0\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-88da25-785647f46c-rb4g9" Oct 30 21:07:36.002220 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:07:36.002139 5378 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pjxpp\" (UniqueName: \"kubernetes.io/projected/a9c904b3-edb2-426a-ac6a-136a4e6afba0-kube-api-access-pjxpp\") pod \"maas-api-e2e-shared-b-w4-88da25-785647f46c-rb4g9\" (UID: \"a9c904b3-edb2-426a-ac6a-136a4e6afba0\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-88da25-785647f46c-rb4g9" Oct 30 21:07:36.033172 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:07:36.033106 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:07:36.184223 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:07:36.184018 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:07:36.444400 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:07:36.444363 5378 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a9c904b3-edb2-426a-ac6a-136a4e6afba0-maas-api-tls\") pod \"maas-api-e2e-shared-b-w4-88da25-785647f46c-rb4g9\" (UID: \"a9c904b3-edb2-426a-ac6a-136a4e6afba0\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-88da25-785647f46c-rb4g9" Oct 30 21:07:36.446857 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:07:36.446833 5378 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a9c904b3-edb2-426a-ac6a-136a4e6afba0-maas-api-tls\") pod \"maas-api-e2e-shared-b-w4-88da25-785647f46c-rb4g9\" (UID: \"a9c904b3-edb2-426a-ac6a-136a4e6afba0\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-88da25-785647f46c-rb4g9" Oct 30 21:07:36.649514 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:07:36.649479 5378 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-88da25-785647f46c-rb4g9" Oct 30 21:07:37.032625 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:07:37.032596 5378 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-88da25-785647f46c-rb4g9"] Oct 30 21:07:37.032962 ip-10-0-138-149 kubenswrapper[5378]: W1030 21:07:37.032911 5378 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda9c904b3_edb2_426a_ac6a_136a4e6afba0.slice/crio-b836ebafb6632969db44c4a01418d7df5d53ad86877279244cc6ee43c10294b3 WatchSource:0}: Error finding container b836ebafb6632969db44c4a01418d7df5d53ad86877279244cc6ee43c10294b3: Status 404 returned error can't find the container with id b836ebafb6632969db44c4a01418d7df5d53ad86877279244cc6ee43c10294b3 Oct 30 21:07:37.034366 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:07:37.034345 5378 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 30 21:07:37.711202 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:07:37.711168 5378 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-88da25-785647f46c-rb4g9" event={"ID":"a9c904b3-edb2-426a-ac6a-136a4e6afba0","Type":"ContainerStarted","Data":"b836ebafb6632969db44c4a01418d7df5d53ad86877279244cc6ee43c10294b3"} Oct 30 21:07:38.719106 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:07:38.719072 5378 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-88da25-785647f46c-rb4g9" event={"ID":"a9c904b3-edb2-426a-ac6a-136a4e6afba0","Type":"ContainerStarted","Data":"fd82c3d6c0c0c909006f171457c846ce2daf7bce0596dc8d8a1cc553c44d7828"} Oct 30 21:07:38.719489 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:07:38.719203 5378 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-88da25-785647f46c-rb4g9" Oct 30 21:07:38.850605 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:07:38.844094 5378 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-88da25-785647f46c-rb4g9" podStartSLOduration=3.034529116 podStartE2EDuration="3.844076028s" totalImagesPullingTime="809.546912ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:07:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:07:38.843225309 +0000 UTC m=+1749.271339228" watchObservedRunningTime="2026-10-30 21:07:38.844076028 +0000 UTC m=+1749.272189946" Oct 30 21:07:39.745629 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:07:39.745595 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:07:40.383107 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:07:40.383071 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:07:40.808582 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:07:40.808492 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:07:41.078768 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:07:41.078680 5378 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-dc6125-5854c458b7-gdqwh"] Oct 30 21:07:41.084348 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:07:41.084326 5378 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-dc6125-5854c458b7-gdqwh" Oct 30 21:07:41.137463 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:07:41.137438 5378 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-w6-dc6125-serving-cert\"" Oct 30 21:07:41.165092 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:07:41.165054 5378 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qxhjr\" (UniqueName: \"kubernetes.io/projected/7e2d9d3a-d5d8-4187-bf82-2b451b8224f4-kube-api-access-qxhjr\") pod \"maas-api-e2e-shared-a-w6-dc6125-5854c458b7-gdqwh\" (UID: \"7e2d9d3a-d5d8-4187-bf82-2b451b8224f4\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-dc6125-5854c458b7-gdqwh" Oct 30 21:07:41.166202 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:07:41.165983 5378 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7e2d9d3a-d5d8-4187-bf82-2b451b8224f4-maas-api-tls\") pod \"maas-api-e2e-shared-a-w6-dc6125-5854c458b7-gdqwh\" (UID: \"7e2d9d3a-d5d8-4187-bf82-2b451b8224f4\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-dc6125-5854c458b7-gdqwh" Oct 30 21:07:41.166202 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:07:41.166136 5378 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7e2d9d3a-d5d8-4187-bf82-2b451b8224f4-metrics-certs\") pod \"maas-api-e2e-shared-a-w6-dc6125-5854c458b7-gdqwh\" (UID: \"7e2d9d3a-d5d8-4187-bf82-2b451b8224f4\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-dc6125-5854c458b7-gdqwh" Oct 30 21:07:41.166355 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:07:41.166284 5378 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7e2d9d3a-d5d8-4187-bf82-2b451b8224f4-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w6-dc6125-5854c458b7-gdqwh\" (UID: \"7e2d9d3a-d5d8-4187-bf82-2b451b8224f4\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-dc6125-5854c458b7-gdqwh" Oct 30 21:07:41.193807 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:07:41.193777 5378 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-dc6125-5854c458b7-gdqwh"] Oct 30 21:07:41.268124 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:07:41.268095 5378 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7e2d9d3a-d5d8-4187-bf82-2b451b8224f4-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w6-dc6125-5854c458b7-gdqwh\" (UID: \"7e2d9d3a-d5d8-4187-bf82-2b451b8224f4\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-dc6125-5854c458b7-gdqwh" Oct 30 21:07:41.268300 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:07:41.268168 5378 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qxhjr\" (UniqueName: \"kubernetes.io/projected/7e2d9d3a-d5d8-4187-bf82-2b451b8224f4-kube-api-access-qxhjr\") pod \"maas-api-e2e-shared-a-w6-dc6125-5854c458b7-gdqwh\" (UID: \"7e2d9d3a-d5d8-4187-bf82-2b451b8224f4\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-dc6125-5854c458b7-gdqwh" Oct 30 21:07:41.268392 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:07:41.268325 5378 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7e2d9d3a-d5d8-4187-bf82-2b451b8224f4-maas-api-tls\") pod \"maas-api-e2e-shared-a-w6-dc6125-5854c458b7-gdqwh\" (UID: \"7e2d9d3a-d5d8-4187-bf82-2b451b8224f4\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-dc6125-5854c458b7-gdqwh" Oct 30 21:07:41.268459 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:07:41.268403 5378 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7e2d9d3a-d5d8-4187-bf82-2b451b8224f4-metrics-certs\") pod \"maas-api-e2e-shared-a-w6-dc6125-5854c458b7-gdqwh\" (UID: \"7e2d9d3a-d5d8-4187-bf82-2b451b8224f4\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-dc6125-5854c458b7-gdqwh" Oct 30 21:07:41.268459 ip-10-0-138-149 kubenswrapper[5378]: E1030 21:07:41.268417 5378 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-dc6125-serving-cert: secret "maas-api-e2e-shared-a-w6-dc6125-serving-cert" not found Oct 30 21:07:41.268612 ip-10-0-138-149 kubenswrapper[5378]: E1030 21:07:41.268482 5378 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7e2d9d3a-d5d8-4187-bf82-2b451b8224f4-maas-api-tls podName:7e2d9d3a-d5d8-4187-bf82-2b451b8224f4 nodeName:}" failed. No retries permitted until 2026-10-30 21:07:41.768468418 +0000 UTC m=+1752.196582332 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/7e2d9d3a-d5d8-4187-bf82-2b451b8224f4-maas-api-tls") pod "maas-api-e2e-shared-a-w6-dc6125-5854c458b7-gdqwh" (UID: "7e2d9d3a-d5d8-4187-bf82-2b451b8224f4") : secret "maas-api-e2e-shared-a-w6-dc6125-serving-cert" not found Oct 30 21:07:41.268887 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:07:41.268865 5378 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7e2d9d3a-d5d8-4187-bf82-2b451b8224f4-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w6-dc6125-5854c458b7-gdqwh\" (UID: \"7e2d9d3a-d5d8-4187-bf82-2b451b8224f4\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-dc6125-5854c458b7-gdqwh" Oct 30 21:07:41.270778 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:07:41.270758 5378 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7e2d9d3a-d5d8-4187-bf82-2b451b8224f4-metrics-certs\") pod \"maas-api-e2e-shared-a-w6-dc6125-5854c458b7-gdqwh\" (UID: \"7e2d9d3a-d5d8-4187-bf82-2b451b8224f4\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-dc6125-5854c458b7-gdqwh" Oct 30 21:07:41.273297 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:07:41.273275 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:07:41.327261 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:07:41.327223 5378 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qxhjr\" (UniqueName: \"kubernetes.io/projected/7e2d9d3a-d5d8-4187-bf82-2b451b8224f4-kube-api-access-qxhjr\") pod \"maas-api-e2e-shared-a-w6-dc6125-5854c458b7-gdqwh\" (UID: \"7e2d9d3a-d5d8-4187-bf82-2b451b8224f4\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-dc6125-5854c458b7-gdqwh" Oct 30 21:07:41.446856 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:07:41.446825 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:07:41.659717 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:07:41.659622 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:07:41.778851 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:07:41.778773 5378 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7e2d9d3a-d5d8-4187-bf82-2b451b8224f4-maas-api-tls\") pod \"maas-api-e2e-shared-a-w6-dc6125-5854c458b7-gdqwh\" (UID: \"7e2d9d3a-d5d8-4187-bf82-2b451b8224f4\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-dc6125-5854c458b7-gdqwh" Oct 30 21:07:41.781248 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:07:41.781229 5378 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7e2d9d3a-d5d8-4187-bf82-2b451b8224f4-maas-api-tls\") pod \"maas-api-e2e-shared-a-w6-dc6125-5854c458b7-gdqwh\" (UID: \"7e2d9d3a-d5d8-4187-bf82-2b451b8224f4\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-dc6125-5854c458b7-gdqwh" Oct 30 21:07:41.999539 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:07:41.999509 5378 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-dc6125-5854c458b7-gdqwh" Oct 30 21:07:42.412833 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:07:42.412800 5378 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-dc6125-5854c458b7-gdqwh"] Oct 30 21:07:42.413141 ip-10-0-138-149 kubenswrapper[5378]: W1030 21:07:42.413113 5378 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7e2d9d3a_d5d8_4187_bf82_2b451b8224f4.slice/crio-343ff3e67bfd250e64bf81a07a17e4505bdd5d1f883c7dce6de84f634f4e21c7 WatchSource:0}: Error finding container 343ff3e67bfd250e64bf81a07a17e4505bdd5d1f883c7dce6de84f634f4e21c7: Status 404 returned error can't find the container with id 343ff3e67bfd250e64bf81a07a17e4505bdd5d1f883c7dce6de84f634f4e21c7 Oct 30 21:07:42.751115 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:07:42.751023 5378 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-dc6125-5854c458b7-gdqwh" event={"ID":"7e2d9d3a-d5d8-4187-bf82-2b451b8224f4","Type":"ContainerStarted","Data":"343ff3e67bfd250e64bf81a07a17e4505bdd5d1f883c7dce6de84f634f4e21c7"} Oct 30 21:07:43.760791 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:07:43.760742 5378 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-dc6125-5854c458b7-gdqwh" event={"ID":"7e2d9d3a-d5d8-4187-bf82-2b451b8224f4","Type":"ContainerStarted","Data":"edea6ba8e081374d3f91c3f3b33c5ce72ddaf2ab7e6744e582def8210a053210"} Oct 30 21:07:43.761327 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:07:43.760905 5378 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-dc6125-5854c458b7-gdqwh" Oct 30 21:07:43.878033 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:07:43.877976 5378 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-dc6125-5854c458b7-gdqwh" podStartSLOduration=2.084918441 podStartE2EDuration="2.877962869s" totalImagesPullingTime="793.044428ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:07:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:07:43.877045191 +0000 UTC m=+1754.305159111" watchObservedRunningTime="2026-10-30 21:07:43.877962869 +0000 UTC m=+1754.306076835" Oct 30 21:07:44.732135 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:07:44.732104 5378 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-88da25-785647f46c-rb4g9" Oct 30 21:07:45.009202 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:07:45.009124 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:07:45.805670 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:07:45.805622 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:07:45.992765 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:07:45.992727 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:07:46.842987 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:07:46.842943 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:07:48.848790 ip-10-0-138-149 kubenswrapper[5378]: E1030 21:07:48.848753 5378 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q6mx7" containerName="extractor" Oct 30 21:07:49.776002 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:07:49.775968 5378 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-dc6125-5854c458b7-gdqwh" Oct 30 21:07:49.933185 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:07:49.933151 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:07:50.144584 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:07:50.142220 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:07:52.094922 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:07:52.094886 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:07:52.341183 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:07:52.341144 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:07:52.476957 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:07:52.476852 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:07:56.963842 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:07:56.963803 5378 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-19db3d5f-55c59f9b9d-h4zbn"] Oct 30 21:07:56.964250 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:07:56.964033 5378 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-mt-19db3d5f-55c59f9b9d-h4zbn" podUID="6bb33be9-e451-42ba-a7f9-c9a4ebbbf169" containerName="maas-api" containerID="cri-o://d3b98d5bee535121103d0993e8805a8d43afaaed8d2cd2ccab6ddd54f009c5ec" gracePeriod=30 Oct 30 21:07:57.065480 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:07:57.065450 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:07:57.338497 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:07:57.338472 5378 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-19db3d5f-55c59f9b9d-h4zbn" Oct 30 21:07:57.474117 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:07:57.474072 5378 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6bb33be9-e451-42ba-a7f9-c9a4ebbbf169-metrics-certs\" (UniqueName: \"kubernetes.io/secret/6bb33be9-e451-42ba-a7f9-c9a4ebbbf169-metrics-certs\") pod \"6bb33be9-e451-42ba-a7f9-c9a4ebbbf169\" (UID: \"6bb33be9-e451-42ba-a7f9-c9a4ebbbf169\") " Oct 30 21:07:57.474550 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:07:57.474520 5378 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6bb33be9-e451-42ba-a7f9-c9a4ebbbf169-kube-api-access-mjb2k\" (UniqueName: \"kubernetes.io/projected/6bb33be9-e451-42ba-a7f9-c9a4ebbbf169-kube-api-access-mjb2k\") pod \"6bb33be9-e451-42ba-a7f9-c9a4ebbbf169\" (UID: \"6bb33be9-e451-42ba-a7f9-c9a4ebbbf169\") " Oct 30 21:07:57.474818 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:07:57.474794 5378 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6bb33be9-e451-42ba-a7f9-c9a4ebbbf169-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6bb33be9-e451-42ba-a7f9-c9a4ebbbf169-openshift-service-ca\") pod \"6bb33be9-e451-42ba-a7f9-c9a4ebbbf169\" (UID: \"6bb33be9-e451-42ba-a7f9-c9a4ebbbf169\") " Oct 30 21:07:57.475026 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:07:57.475005 5378 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6bb33be9-e451-42ba-a7f9-c9a4ebbbf169-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6bb33be9-e451-42ba-a7f9-c9a4ebbbf169-maas-api-tls\") pod \"6bb33be9-e451-42ba-a7f9-c9a4ebbbf169\" (UID: \"6bb33be9-e451-42ba-a7f9-c9a4ebbbf169\") " Oct 30 21:07:57.475747 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:07:57.475681 5378 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6bb33be9-e451-42ba-a7f9-c9a4ebbbf169-openshift-service-ca" pod "6bb33be9-e451-42ba-a7f9-c9a4ebbbf169" (UID: "6bb33be9-e451-42ba-a7f9-c9a4ebbbf169"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:07:57.476247 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:07:57.476223 5378 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6bb33be9-e451-42ba-a7f9-c9a4ebbbf169-openshift-service-ca\") on node \"ip-10-0-138-149.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:07:57.518421 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:07:57.518358 5378 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6bb33be9-e451-42ba-a7f9-c9a4ebbbf169-maas-api-tls" pod "6bb33be9-e451-42ba-a7f9-c9a4ebbbf169" (UID: "6bb33be9-e451-42ba-a7f9-c9a4ebbbf169"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:07:57.522724 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:07:57.521333 5378 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6bb33be9-e451-42ba-a7f9-c9a4ebbbf169-metrics-certs" pod "6bb33be9-e451-42ba-a7f9-c9a4ebbbf169" (UID: "6bb33be9-e451-42ba-a7f9-c9a4ebbbf169"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:07:57.525320 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:07:57.524807 5378 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6bb33be9-e451-42ba-a7f9-c9a4ebbbf169-kube-api-access-mjb2k" pod "6bb33be9-e451-42ba-a7f9-c9a4ebbbf169" (UID: "6bb33be9-e451-42ba-a7f9-c9a4ebbbf169"). InnerVolumeSpecName "kube-api-access-mjb2k". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:07:57.587738 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:07:57.580209 5378 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6bb33be9-e451-42ba-a7f9-c9a4ebbbf169-maas-api-tls\") on node \"ip-10-0-138-149.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:07:57.587738 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:07:57.580247 5378 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6bb33be9-e451-42ba-a7f9-c9a4ebbbf169-metrics-certs\") on node \"ip-10-0-138-149.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:07:57.587738 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:07:57.580263 5378 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mjb2k\" (UniqueName: \"kubernetes.io/projected/6bb33be9-e451-42ba-a7f9-c9a4ebbbf169-kube-api-access-mjb2k\") on node \"ip-10-0-138-149.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:07:57.864306 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:07:57.864274 5378 generic.go:356] "Generic (PLEG): container finished" podID="6bb33be9-e451-42ba-a7f9-c9a4ebbbf169" containerID="d3b98d5bee535121103d0993e8805a8d43afaaed8d2cd2ccab6ddd54f009c5ec" exitCode=0 Oct 30 21:07:57.864466 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:07:57.864325 5378 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-19db3d5f-55c59f9b9d-h4zbn" Oct 30 21:07:57.864466 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:07:57.864362 5378 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-19db3d5f-55c59f9b9d-h4zbn" event={"ID":"6bb33be9-e451-42ba-a7f9-c9a4ebbbf169","Type":"ContainerDied","Data":"d3b98d5bee535121103d0993e8805a8d43afaaed8d2cd2ccab6ddd54f009c5ec"} Oct 30 21:07:57.864466 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:07:57.864415 5378 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-19db3d5f-55c59f9b9d-h4zbn" event={"ID":"6bb33be9-e451-42ba-a7f9-c9a4ebbbf169","Type":"ContainerDied","Data":"3f9928697fc5c4fa17c4212524033d5c380408716294236a138d377d18f4b695"} Oct 30 21:07:57.864466 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:07:57.864443 5378 scope.go:122] "RemoveContainer" containerID="d3b98d5bee535121103d0993e8805a8d43afaaed8d2cd2ccab6ddd54f009c5ec" Oct 30 21:07:57.875091 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:07:57.875072 5378 scope.go:122] "RemoveContainer" containerID="d3b98d5bee535121103d0993e8805a8d43afaaed8d2cd2ccab6ddd54f009c5ec" Oct 30 21:07:57.875347 ip-10-0-138-149 kubenswrapper[5378]: E1030 21:07:57.875325 5378 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d3b98d5bee535121103d0993e8805a8d43afaaed8d2cd2ccab6ddd54f009c5ec\": container with ID starting with d3b98d5bee535121103d0993e8805a8d43afaaed8d2cd2ccab6ddd54f009c5ec not found: ID does not exist" containerID="d3b98d5bee535121103d0993e8805a8d43afaaed8d2cd2ccab6ddd54f009c5ec" Oct 30 21:07:57.875422 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:07:57.875354 5378 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d3b98d5bee535121103d0993e8805a8d43afaaed8d2cd2ccab6ddd54f009c5ec"} err="failed to get container status \"d3b98d5bee535121103d0993e8805a8d43afaaed8d2cd2ccab6ddd54f009c5ec\": rpc error: code = NotFound desc = could not find container \"d3b98d5bee535121103d0993e8805a8d43afaaed8d2cd2ccab6ddd54f009c5ec\": container with ID starting with d3b98d5bee535121103d0993e8805a8d43afaaed8d2cd2ccab6ddd54f009c5ec not found: ID does not exist" Oct 30 21:07:58.085333 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:07:58.085298 5378 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-19db3d5f-55c59f9b9d-h4zbn"] Oct 30 21:07:58.096601 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:07:58.092344 5378 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-19db3d5f-55c59f9b9d-h4zbn"] Oct 30 21:07:58.735141 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:07:58.735107 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:07:59.000319 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:07:59.000246 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:07:59.244652 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:07:59.244609 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:07:59.683241 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:07:59.683212 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:07:59.745204 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:07:59.745175 5378 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6bb33be9-e451-42ba-a7f9-c9a4ebbbf169" path="/var/lib/kubelet/pods/6bb33be9-e451-42ba-a7f9-c9a4ebbbf169/volumes" Oct 30 21:08:02.968182 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:08:02.968148 5378 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-r9g59"] Oct 30 21:08:04.307589 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:08:04.300109 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:08:04.493590 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:08:04.488144 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:08:06.416893 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:08:06.416859 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:08:06.586608 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:08:06.584207 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:08:06.801843 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:08:06.801757 5378 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-19db3d5f-openshift-default-7f4c69b8fc-vjtzt"] Oct 30 21:08:06.802635 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:08:06.802525 5378 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-19db3d5f-openshift-default-7f4c69b8fc-vjtzt" podUID="0406854a-c204-4495-bdc8-23afaad6457f" containerName="istio-proxy" containerID="cri-o://43dad921d5d07df6927ef56819f534c8b66f7abac3bd91b97c2593ac511fc94c" gracePeriod=30 Oct 30 21:08:06.807987 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:08:06.807965 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:08:07.751200 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:08:07.751165 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:08:08.603384 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:08:08.603261 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:08:09.068493 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:08:09.068409 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:08:09.139839 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:08:09.139800 5378 patch_prober.go:29] interesting pod/e2e-mt-19db3d5f-openshift-default-7f4c69b8fc-vjtzt container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 21:08:09.140023 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:08:09.139864 5378 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-mt-19db3d5f-openshift-default-7f4c69b8fc-vjtzt" podUID="0406854a-c204-4495-bdc8-23afaad6457f" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 21:08:09.992712 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:08:09.992676 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:08:12.106416 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:08:12.106387 5378 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-19db3d5f-openshift-default-7f4c69b8fc-vjtzt" Oct 30 21:08:12.152941 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:08:12.152905 5378 generic.go:356] "Generic (PLEG): container finished" podID="0406854a-c204-4495-bdc8-23afaad6457f" containerID="43dad921d5d07df6927ef56819f534c8b66f7abac3bd91b97c2593ac511fc94c" exitCode=0 Oct 30 21:08:12.153142 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:08:12.152949 5378 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-19db3d5f-openshift-default-7f4c69b8fc-vjtzt" event={"ID":"0406854a-c204-4495-bdc8-23afaad6457f","Type":"ContainerDied","Data":"43dad921d5d07df6927ef56819f534c8b66f7abac3bd91b97c2593ac511fc94c"} Oct 30 21:08:12.153142 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:08:12.152973 5378 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-19db3d5f-openshift-default-7f4c69b8fc-vjtzt" Oct 30 21:08:12.153142 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:08:12.153009 5378 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-19db3d5f-openshift-default-7f4c69b8fc-vjtzt" event={"ID":"0406854a-c204-4495-bdc8-23afaad6457f","Type":"ContainerDied","Data":"49bbf42185959c1f4347f9f36083124555bf7182b71a3df89a875dc329bf792c"} Oct 30 21:08:12.153142 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:08:12.153038 5378 scope.go:122] "RemoveContainer" containerID="43dad921d5d07df6927ef56819f534c8b66f7abac3bd91b97c2593ac511fc94c" Oct 30 21:08:12.163707 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:08:12.163680 5378 scope.go:122] "RemoveContainer" containerID="8793bedb56a074407b42ba1d90aedb3e2ea15f4e71c7cdabf051ca723976299e" Oct 30 21:08:12.173488 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:08:12.173465 5378 scope.go:122] "RemoveContainer" containerID="43dad921d5d07df6927ef56819f534c8b66f7abac3bd91b97c2593ac511fc94c" Oct 30 21:08:12.173807 ip-10-0-138-149 kubenswrapper[5378]: E1030 21:08:12.173785 5378 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"43dad921d5d07df6927ef56819f534c8b66f7abac3bd91b97c2593ac511fc94c\": container with ID starting with 43dad921d5d07df6927ef56819f534c8b66f7abac3bd91b97c2593ac511fc94c not found: ID does not exist" containerID="43dad921d5d07df6927ef56819f534c8b66f7abac3bd91b97c2593ac511fc94c" Oct 30 21:08:12.173876 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:08:12.173817 5378 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"43dad921d5d07df6927ef56819f534c8b66f7abac3bd91b97c2593ac511fc94c"} err="failed to get container status \"43dad921d5d07df6927ef56819f534c8b66f7abac3bd91b97c2593ac511fc94c\": rpc error: code = NotFound desc = could not find container \"43dad921d5d07df6927ef56819f534c8b66f7abac3bd91b97c2593ac511fc94c\": container with ID starting with 43dad921d5d07df6927ef56819f534c8b66f7abac3bd91b97c2593ac511fc94c not found: ID does not exist" Oct 30 21:08:12.173876 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:08:12.173839 5378 scope.go:122] "RemoveContainer" containerID="8793bedb56a074407b42ba1d90aedb3e2ea15f4e71c7cdabf051ca723976299e" Oct 30 21:08:12.174087 ip-10-0-138-149 kubenswrapper[5378]: E1030 21:08:12.174072 5378 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8793bedb56a074407b42ba1d90aedb3e2ea15f4e71c7cdabf051ca723976299e\": container with ID starting with 8793bedb56a074407b42ba1d90aedb3e2ea15f4e71c7cdabf051ca723976299e not found: ID does not exist" containerID="8793bedb56a074407b42ba1d90aedb3e2ea15f4e71c7cdabf051ca723976299e" Oct 30 21:08:12.174136 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:08:12.174091 5378 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8793bedb56a074407b42ba1d90aedb3e2ea15f4e71c7cdabf051ca723976299e"} err="failed to get container status \"8793bedb56a074407b42ba1d90aedb3e2ea15f4e71c7cdabf051ca723976299e\": rpc error: code = NotFound desc = could not find container \"8793bedb56a074407b42ba1d90aedb3e2ea15f4e71c7cdabf051ca723976299e\": container with ID starting with 8793bedb56a074407b42ba1d90aedb3e2ea15f4e71c7cdabf051ca723976299e not found: ID does not exist" Oct 30 21:08:12.207972 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:08:12.207946 5378 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0406854a-c204-4495-bdc8-23afaad6457f-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0406854a-c204-4495-bdc8-23afaad6457f-istio-data\") pod \"0406854a-c204-4495-bdc8-23afaad6457f\" (UID: \"0406854a-c204-4495-bdc8-23afaad6457f\") " Oct 30 21:08:12.208119 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:08:12.208006 5378 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0406854a-c204-4495-bdc8-23afaad6457f-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0406854a-c204-4495-bdc8-23afaad6457f-credential-socket\") pod \"0406854a-c204-4495-bdc8-23afaad6457f\" (UID: \"0406854a-c204-4495-bdc8-23afaad6457f\") " Oct 30 21:08:12.208206 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:08:12.208136 5378 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0406854a-c204-4495-bdc8-23afaad6457f-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0406854a-c204-4495-bdc8-23afaad6457f-istio-envoy\") pod \"0406854a-c204-4495-bdc8-23afaad6457f\" (UID: \"0406854a-c204-4495-bdc8-23afaad6457f\") " Oct 30 21:08:12.208277 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:08:12.208260 5378 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0406854a-c204-4495-bdc8-23afaad6457f-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0406854a-c204-4495-bdc8-23afaad6457f-workload-socket\") pod \"0406854a-c204-4495-bdc8-23afaad6457f\" (UID: \"0406854a-c204-4495-bdc8-23afaad6457f\") " Oct 30 21:08:12.208343 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:08:12.208328 5378 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0406854a-c204-4495-bdc8-23afaad6457f-kube-api-access-f6szh\" (UniqueName: \"kubernetes.io/projected/0406854a-c204-4495-bdc8-23afaad6457f-kube-api-access-f6szh\") pod \"0406854a-c204-4495-bdc8-23afaad6457f\" (UID: \"0406854a-c204-4495-bdc8-23afaad6457f\") " Oct 30 21:08:12.208402 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:08:12.208389 5378 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0406854a-c204-4495-bdc8-23afaad6457f-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0406854a-c204-4495-bdc8-23afaad6457f-istiod-ca-cert\") pod \"0406854a-c204-4495-bdc8-23afaad6457f\" (UID: \"0406854a-c204-4495-bdc8-23afaad6457f\") " Oct 30 21:08:12.208465 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:08:12.208258 5378 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0406854a-c204-4495-bdc8-23afaad6457f-istio-data" pod "0406854a-c204-4495-bdc8-23afaad6457f" (UID: "0406854a-c204-4495-bdc8-23afaad6457f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:08:12.208465 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:08:12.208456 5378 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/0406854a-c204-4495-bdc8-23afaad6457f-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0406854a-c204-4495-bdc8-23afaad6457f-istio-podinfo\") pod \"0406854a-c204-4495-bdc8-23afaad6457f\" (UID: \"0406854a-c204-4495-bdc8-23afaad6457f\") " Oct 30 21:08:12.208597 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:08:12.208276 5378 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0406854a-c204-4495-bdc8-23afaad6457f-credential-socket" pod "0406854a-c204-4495-bdc8-23afaad6457f" (UID: "0406854a-c204-4495-bdc8-23afaad6457f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:08:12.208597 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:08:12.208483 5378 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0406854a-c204-4495-bdc8-23afaad6457f-workload-socket" pod "0406854a-c204-4495-bdc8-23afaad6457f" (UID: "0406854a-c204-4495-bdc8-23afaad6457f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:08:12.208697 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:08:12.208590 5378 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0406854a-c204-4495-bdc8-23afaad6457f-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0406854a-c204-4495-bdc8-23afaad6457f-workload-certs\") pod \"0406854a-c204-4495-bdc8-23afaad6457f\" (UID: \"0406854a-c204-4495-bdc8-23afaad6457f\") " Oct 30 21:08:12.208760 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:08:12.208731 5378 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0406854a-c204-4495-bdc8-23afaad6457f-istio-token\" (UniqueName: \"kubernetes.io/projected/0406854a-c204-4495-bdc8-23afaad6457f-istio-token\") pod \"0406854a-c204-4495-bdc8-23afaad6457f\" (UID: \"0406854a-c204-4495-bdc8-23afaad6457f\") " Oct 30 21:08:12.208874 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:08:12.208850 5378 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0406854a-c204-4495-bdc8-23afaad6457f-istiod-ca-cert" pod "0406854a-c204-4495-bdc8-23afaad6457f" (UID: "0406854a-c204-4495-bdc8-23afaad6457f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:08:12.209319 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:08:12.209296 5378 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0406854a-c204-4495-bdc8-23afaad6457f-workload-certs" pod "0406854a-c204-4495-bdc8-23afaad6457f" (UID: "0406854a-c204-4495-bdc8-23afaad6457f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:08:12.209472 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:08:12.209454 5378 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0406854a-c204-4495-bdc8-23afaad6457f-workload-socket\") on node \"ip-10-0-138-149.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:08:12.209580 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:08:12.209482 5378 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0406854a-c204-4495-bdc8-23afaad6457f-istiod-ca-cert\") on node \"ip-10-0-138-149.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:08:12.209580 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:08:12.209498 5378 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0406854a-c204-4495-bdc8-23afaad6457f-workload-certs\") on node \"ip-10-0-138-149.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:08:12.209580 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:08:12.209514 5378 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0406854a-c204-4495-bdc8-23afaad6457f-istio-data\") on node \"ip-10-0-138-149.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:08:12.209580 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:08:12.209528 5378 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0406854a-c204-4495-bdc8-23afaad6457f-credential-socket\") on node \"ip-10-0-138-149.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:08:12.211686 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:08:12.211659 5378 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0406854a-c204-4495-bdc8-23afaad6457f-istio-envoy" pod "0406854a-c204-4495-bdc8-23afaad6457f" (UID: "0406854a-c204-4495-bdc8-23afaad6457f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:08:12.211919 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:08:12.211896 5378 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0406854a-c204-4495-bdc8-23afaad6457f-kube-api-access-f6szh" pod "0406854a-c204-4495-bdc8-23afaad6457f" (UID: "0406854a-c204-4495-bdc8-23afaad6457f"). InnerVolumeSpecName "kube-api-access-f6szh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:08:12.211987 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:08:12.211954 5378 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/0406854a-c204-4495-bdc8-23afaad6457f-istio-podinfo" pod "0406854a-c204-4495-bdc8-23afaad6457f" (UID: "0406854a-c204-4495-bdc8-23afaad6457f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 21:08:12.212289 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:08:12.212267 5378 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0406854a-c204-4495-bdc8-23afaad6457f-istio-token" pod "0406854a-c204-4495-bdc8-23afaad6457f" (UID: "0406854a-c204-4495-bdc8-23afaad6457f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:08:12.310734 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:08:12.310699 5378 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0406854a-c204-4495-bdc8-23afaad6457f-istio-envoy\") on node \"ip-10-0-138-149.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:08:12.310734 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:08:12.310734 5378 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f6szh\" (UniqueName: \"kubernetes.io/projected/0406854a-c204-4495-bdc8-23afaad6457f-kube-api-access-f6szh\") on node \"ip-10-0-138-149.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:08:12.310939 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:08:12.310746 5378 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0406854a-c204-4495-bdc8-23afaad6457f-istio-podinfo\") on node \"ip-10-0-138-149.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:08:12.310939 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:08:12.310755 5378 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0406854a-c204-4495-bdc8-23afaad6457f-istio-token\") on node \"ip-10-0-138-149.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:08:12.358009 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:08:12.357976 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:08:12.547159 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:08:12.547131 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:08:12.588963 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:08:12.588926 5378 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-19db3d5f-openshift-default-7f4c69b8fc-vjtzt"] Oct 30 21:08:12.595521 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:08:12.595483 5378 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-19db3d5f-openshift-default-7f4c69b8fc-vjtzt"] Oct 30 21:08:12.679263 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:08:12.679179 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:08:13.744942 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:08:13.744904 5378 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0406854a-c204-4495-bdc8-23afaad6457f" path="/var/lib/kubelet/pods/0406854a-c204-4495-bdc8-23afaad6457f/volumes" Oct 30 21:08:15.177627 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:08:15.177558 5378 generic.go:356] "Generic (PLEG): container finished" podID="191dc733-cb36-4efc-9572-e645e2eb59d8" containerID="345ba2ed9062b536d084e4b4475ec52a5838cdb19de04968cd6e6b8f301aa257" exitCode=0 Oct 30 21:08:15.178044 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:08:15.177644 5378 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-88da25-openshift-default-6c4ccc4cd9-tcsdd" event={"ID":"191dc733-cb36-4efc-9572-e645e2eb59d8","Type":"ContainerDied","Data":"345ba2ed9062b536d084e4b4475ec52a5838cdb19de04968cd6e6b8f301aa257"} Oct 30 21:08:15.178101 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:08:15.178081 5378 scope.go:122] "RemoveContainer" containerID="345ba2ed9062b536d084e4b4475ec52a5838cdb19de04968cd6e6b8f301aa257" Oct 30 21:08:15.179691 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:08:15.179661 5378 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 21:08:15.179816 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:08:15.179745 5378 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 21:08:15.179816 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:08:15.179794 5378 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 21:08:15.389496 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:08:15.389462 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:08:15.695304 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:08:15.695272 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:08:15.940829 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:08:15.940791 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:08:16.104459 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:08:16.104297 5378 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-9926f7-646b87d4d7-pgzdg"] Oct 30 21:08:16.105993 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:08:16.105962 5378 cpu_manager.go:414] "containerMap: removing container" podUID="0406854a-c204-4495-bdc8-23afaad6457f" containerName="istio-proxy" Oct 30 21:08:16.106526 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:08:16.106505 5378 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0406854a-c204-4495-bdc8-23afaad6457f" containerName="istio-proxy" Oct 30 21:08:16.106850 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:08:16.106832 5378 cpu_manager.go:414] "containerMap: removing container" podUID="0406854a-c204-4495-bdc8-23afaad6457f" containerName="istio-proxy" Oct 30 21:08:16.106974 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:08:16.106960 5378 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0406854a-c204-4495-bdc8-23afaad6457f" containerName="istio-proxy" Oct 30 21:08:16.107083 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:08:16.107072 5378 cpu_manager.go:414] "containerMap: removing container" podUID="6bb33be9-e451-42ba-a7f9-c9a4ebbbf169" containerName="maas-api" Oct 30 21:08:16.107175 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:08:16.107164 5378 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6bb33be9-e451-42ba-a7f9-c9a4ebbbf169" containerName="maas-api" Oct 30 21:08:16.107609 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:08:16.107464 5378 memory_manager.go:365] "RemoveStaleState removing state" podUID="0406854a-c204-4495-bdc8-23afaad6457f" containerName="istio-proxy" Oct 30 21:08:16.107749 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:08:16.107734 5378 memory_manager.go:365] "RemoveStaleState removing state" podUID="6bb33be9-e451-42ba-a7f9-c9a4ebbbf169" containerName="maas-api" Oct 30 21:08:16.107860 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:08:16.107847 5378 memory_manager.go:365] "RemoveStaleState removing state" podUID="0406854a-c204-4495-bdc8-23afaad6457f" containerName="istio-proxy" Oct 30 21:08:16.121309 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:08:16.121278 5378 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-9926f7-646b87d4d7-pgzdg" Oct 30 21:08:16.132684 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:08:16.130945 5378 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lfs2k\" (UniqueName: \"kubernetes.io/projected/2fd814bb-f2a6-4ae3-a53c-6108f380f607-kube-api-access-lfs2k\") pod \"payload-pre-processing-e2e-api-a-9926f7-646b87d4d7-pgzdg\" (UID: \"2fd814bb-f2a6-4ae3-a53c-6108f380f607\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-9926f7-646b87d4d7-pgzdg" Oct 30 21:08:16.132684 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:08:16.131038 5378 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2fd814bb-f2a6-4ae3-a53c-6108f380f607-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-9926f7-646b87d4d7-pgzdg\" (UID: \"2fd814bb-f2a6-4ae3-a53c-6108f380f607\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-9926f7-646b87d4d7-pgzdg" Oct 30 21:08:16.161679 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:08:16.161649 5378 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-9926f7-646b87d4d7-pgzdg"] Oct 30 21:08:16.178417 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:08:16.178377 5378 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-a-9926f7-dockercfg-tv8vd\"" Oct 30 21:08:16.178889 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:08:16.178634 5378 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-a-9926f7\"" Oct 30 21:08:16.201586 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:08:16.192058 5378 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-88da25-openshift-default-6c4ccc4cd9-tcsdd" event={"ID":"191dc733-cb36-4efc-9572-e645e2eb59d8","Type":"ContainerStarted","Data":"b32bbef49274b5f0ba3e132a3db6b2019f999b73007e1ed694bd0b5142d741e9"} Oct 30 21:08:16.363635 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:08:16.359263 5378 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lfs2k\" (UniqueName: \"kubernetes.io/projected/2fd814bb-f2a6-4ae3-a53c-6108f380f607-kube-api-access-lfs2k\") pod \"payload-pre-processing-e2e-api-a-9926f7-646b87d4d7-pgzdg\" (UID: \"2fd814bb-f2a6-4ae3-a53c-6108f380f607\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-9926f7-646b87d4d7-pgzdg" Oct 30 21:08:16.363635 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:08:16.359372 5378 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2fd814bb-f2a6-4ae3-a53c-6108f380f607-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-9926f7-646b87d4d7-pgzdg\" (UID: \"2fd814bb-f2a6-4ae3-a53c-6108f380f607\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-9926f7-646b87d4d7-pgzdg" Oct 30 21:08:16.380610 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:08:16.369384 5378 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2fd814bb-f2a6-4ae3-a53c-6108f380f607-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-9926f7-646b87d4d7-pgzdg\" (UID: \"2fd814bb-f2a6-4ae3-a53c-6108f380f607\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-9926f7-646b87d4d7-pgzdg" Oct 30 21:08:16.443328 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:08:16.443291 5378 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-9926f7-57bb5db5fb-b224f"] Oct 30 21:08:16.453766 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:08:16.452258 5378 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-9926f7-57bb5db5fb-b224f"] Oct 30 21:08:16.453766 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:08:16.452541 5378 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-9926f7-57bb5db5fb-b224f" Oct 30 21:08:16.454965 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:08:16.454906 5378 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lfs2k\" (UniqueName: \"kubernetes.io/projected/2fd814bb-f2a6-4ae3-a53c-6108f380f607-kube-api-access-lfs2k\") pod \"payload-pre-processing-e2e-api-a-9926f7-646b87d4d7-pgzdg\" (UID: \"2fd814bb-f2a6-4ae3-a53c-6108f380f607\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-9926f7-646b87d4d7-pgzdg" Oct 30 21:08:16.461355 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:08:16.460978 5378 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6d33dc48-4424-477b-ae64-bfc39c7faaa3-plugins-config-volume\") pod \"payload-processing-e2e-api-a-9926f7-57bb5db5fb-b224f\" (UID: \"6d33dc48-4424-477b-ae64-bfc39c7faaa3\") " pod="openshift-ingress/payload-processing-e2e-api-a-9926f7-57bb5db5fb-b224f" Oct 30 21:08:16.461355 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:08:16.461235 5378 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tphpj\" (UniqueName: \"kubernetes.io/projected/6d33dc48-4424-477b-ae64-bfc39c7faaa3-kube-api-access-tphpj\") pod \"payload-processing-e2e-api-a-9926f7-57bb5db5fb-b224f\" (UID: \"6d33dc48-4424-477b-ae64-bfc39c7faaa3\") " pod="openshift-ingress/payload-processing-e2e-api-a-9926f7-57bb5db5fb-b224f" Oct 30 21:08:16.576589 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:08:16.563814 5378 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tphpj\" (UniqueName: \"kubernetes.io/projected/6d33dc48-4424-477b-ae64-bfc39c7faaa3-kube-api-access-tphpj\") pod \"payload-processing-e2e-api-a-9926f7-57bb5db5fb-b224f\" (UID: \"6d33dc48-4424-477b-ae64-bfc39c7faaa3\") " pod="openshift-ingress/payload-processing-e2e-api-a-9926f7-57bb5db5fb-b224f" Oct 30 21:08:16.576589 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:08:16.563903 5378 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6d33dc48-4424-477b-ae64-bfc39c7faaa3-plugins-config-volume\") pod \"payload-processing-e2e-api-a-9926f7-57bb5db5fb-b224f\" (UID: \"6d33dc48-4424-477b-ae64-bfc39c7faaa3\") " pod="openshift-ingress/payload-processing-e2e-api-a-9926f7-57bb5db5fb-b224f" Oct 30 21:08:16.576589 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:08:16.574041 5378 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6d33dc48-4424-477b-ae64-bfc39c7faaa3-plugins-config-volume\") pod \"payload-processing-e2e-api-a-9926f7-57bb5db5fb-b224f\" (UID: \"6d33dc48-4424-477b-ae64-bfc39c7faaa3\") " pod="openshift-ingress/payload-processing-e2e-api-a-9926f7-57bb5db5fb-b224f" Oct 30 21:08:16.577644 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:08:16.577118 5378 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w4-88da25-openshift-default-6c4ccc4cd9-tcsdd" Oct 30 21:08:16.627951 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:08:16.627885 5378 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tphpj\" (UniqueName: \"kubernetes.io/projected/6d33dc48-4424-477b-ae64-bfc39c7faaa3-kube-api-access-tphpj\") pod \"payload-processing-e2e-api-a-9926f7-57bb5db5fb-b224f\" (UID: \"6d33dc48-4424-477b-ae64-bfc39c7faaa3\") " pod="openshift-ingress/payload-processing-e2e-api-a-9926f7-57bb5db5fb-b224f" Oct 30 21:08:16.752138 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:08:16.752097 5378 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-9926f7-646b87d4d7-pgzdg" Oct 30 21:08:16.816967 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:08:16.816939 5378 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-9926f7-57bb5db5fb-b224f" Oct 30 21:08:16.817957 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:08:16.817926 5378 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w4-88da25-openshift-default-6c4ccc4cd9-tcsdd" Oct 30 21:08:16.819928 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:08:16.819892 5378 patch_prober.go:29] interesting pod/e2e-shared-b-w4-88da25-openshift-default-6c4ccc4cd9-tcsdd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.121:15021/healthz/ready\": dial tcp 10.133.0.121:15021: connect: connection refused" start-of-body= Oct 30 21:08:16.820039 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:08:16.819994 5378 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-88da25-openshift-default-6c4ccc4cd9-tcsdd" podUID="191dc733-cb36-4efc-9572-e645e2eb59d8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.121:15021/healthz/ready\": dial tcp 10.133.0.121:15021: connect: connection refused" Oct 30 21:08:17.180412 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:08:17.180240 5378 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-9926f7-646b87d4d7-pgzdg"] Oct 30 21:08:17.182129 ip-10-0-138-149 kubenswrapper[5378]: W1030 21:08:17.182092 5378 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2fd814bb_f2a6_4ae3_a53c_6108f380f607.slice/crio-220b40cabe0ea327951f88952db811071c0b1a215d521f42240af84958e98669 WatchSource:0}: Error finding container 220b40cabe0ea327951f88952db811071c0b1a215d521f42240af84958e98669: Status 404 returned error can't find the container with id 220b40cabe0ea327951f88952db811071c0b1a215d521f42240af84958e98669 Oct 30 21:08:17.199302 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:08:17.199246 5378 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-9926f7-646b87d4d7-pgzdg" event={"ID":"2fd814bb-f2a6-4ae3-a53c-6108f380f607","Type":"ContainerStarted","Data":"220b40cabe0ea327951f88952db811071c0b1a215d521f42240af84958e98669"} Oct 30 21:08:17.281966 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:08:17.281934 5378 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-9926f7-57bb5db5fb-b224f"] Oct 30 21:08:17.284812 ip-10-0-138-149 kubenswrapper[5378]: W1030 21:08:17.284781 5378 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6d33dc48_4424_477b_ae64_bfc39c7faaa3.slice/crio-fddc5f3afee6ab0bfefeedfa7f5b416b8704dbd139e76c61ccf84ed308bb9abb WatchSource:0}: Error finding container fddc5f3afee6ab0bfefeedfa7f5b416b8704dbd139e76c61ccf84ed308bb9abb: Status 404 returned error can't find the container with id fddc5f3afee6ab0bfefeedfa7f5b416b8704dbd139e76c61ccf84ed308bb9abb Oct 30 21:08:17.817516 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:08:17.817478 5378 patch_prober.go:29] interesting pod/e2e-shared-b-w4-88da25-openshift-default-6c4ccc4cd9-tcsdd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.121:15021/healthz/ready\": dial tcp 10.133.0.121:15021: connect: connection refused" start-of-body= Oct 30 21:08:17.817717 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:08:17.817527 5378 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-88da25-openshift-default-6c4ccc4cd9-tcsdd" podUID="191dc733-cb36-4efc-9572-e645e2eb59d8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.121:15021/healthz/ready\": dial tcp 10.133.0.121:15021: connect: connection refused" Oct 30 21:08:18.208249 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:08:18.208203 5378 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-9926f7-646b87d4d7-pgzdg" event={"ID":"2fd814bb-f2a6-4ae3-a53c-6108f380f607","Type":"ContainerStarted","Data":"16c7b014c1e6335bf5a943d7aafdc9279ae20b9a34da594caf26aef8ed563753"} Oct 30 21:08:18.208760 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:08:18.208272 5378 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-9926f7-646b87d4d7-pgzdg" Oct 30 21:08:18.209944 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:08:18.209915 5378 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-9926f7-57bb5db5fb-b224f" event={"ID":"6d33dc48-4424-477b-ae64-bfc39c7faaa3","Type":"ContainerStarted","Data":"4a9df14a558134ad49179773491f61f140605eab1ce581abcfc1ef8509dc20ce"} Oct 30 21:08:18.210079 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:08:18.209961 5378 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-9926f7-57bb5db5fb-b224f" event={"ID":"6d33dc48-4424-477b-ae64-bfc39c7faaa3","Type":"ContainerStarted","Data":"fddc5f3afee6ab0bfefeedfa7f5b416b8704dbd139e76c61ccf84ed308bb9abb"} Oct 30 21:08:18.210079 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:08:18.209982 5378 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-a-9926f7-57bb5db5fb-b224f" Oct 30 21:08:18.324452 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:08:18.324393 5378 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-a-9926f7-646b87d4d7-pgzdg" podStartSLOduration=2.324379799 podStartE2EDuration="2.324379799s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:08:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:08:18.323516824 +0000 UTC m=+1788.751630765" watchObservedRunningTime="2026-10-30 21:08:18.324379799 +0000 UTC m=+1788.752493717" Oct 30 21:08:18.447226 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:08:18.447085 5378 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-a-9926f7-57bb5db5fb-b224f" podStartSLOduration=2.447065625 podStartE2EDuration="2.447065625s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:08:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:08:18.442187177 +0000 UTC m=+1788.870301097" watchObservedRunningTime="2026-10-30 21:08:18.447065625 +0000 UTC m=+1788.875179548" Oct 30 21:08:18.817533 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:08:18.817504 5378 patch_prober.go:29] interesting pod/e2e-shared-b-w4-88da25-openshift-default-6c4ccc4cd9-tcsdd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.121:15021/healthz/ready\": dial tcp 10.133.0.121:15021: connect: connection refused" start-of-body= Oct 30 21:08:18.817737 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:08:18.817552 5378 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-88da25-openshift-default-6c4ccc4cd9-tcsdd" podUID="191dc733-cb36-4efc-9572-e645e2eb59d8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.121:15021/healthz/ready\": dial tcp 10.133.0.121:15021: connect: connection refused" Oct 30 21:08:18.837852 ip-10-0-138-149 kubenswrapper[5378]: E1030 21:08:18.837819 5378 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q6mx7" containerName="extractor" Oct 30 21:08:19.817641 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:08:19.817612 5378 patch_prober.go:29] interesting pod/e2e-shared-b-w4-88da25-openshift-default-6c4ccc4cd9-tcsdd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.121:15021/healthz/ready\": dial tcp 10.133.0.121:15021: connect: connection refused" start-of-body= Oct 30 21:08:19.818036 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:08:19.817660 5378 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-88da25-openshift-default-6c4ccc4cd9-tcsdd" podUID="191dc733-cb36-4efc-9572-e645e2eb59d8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.121:15021/healthz/ready\": dial tcp 10.133.0.121:15021: connect: connection refused" Oct 30 21:08:20.817366 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:08:20.817327 5378 patch_prober.go:29] interesting pod/e2e-shared-b-w4-88da25-openshift-default-6c4ccc4cd9-tcsdd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.121:15021/healthz/ready\": dial tcp 10.133.0.121:15021: connect: connection refused" start-of-body= Oct 30 21:08:20.817549 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:08:20.817413 5378 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-88da25-openshift-default-6c4ccc4cd9-tcsdd" podUID="191dc733-cb36-4efc-9572-e645e2eb59d8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.121:15021/healthz/ready\": dial tcp 10.133.0.121:15021: connect: connection refused" Oct 30 21:08:20.968012 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:08:20.967895 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:08:21.818166 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:08:21.818134 5378 patch_prober.go:29] interesting pod/e2e-shared-b-w4-88da25-openshift-default-6c4ccc4cd9-tcsdd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.121:15021/healthz/ready\": dial tcp 10.133.0.121:15021: connect: connection refused" start-of-body= Oct 30 21:08:21.818358 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:08:21.818192 5378 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-88da25-openshift-default-6c4ccc4cd9-tcsdd" podUID="191dc733-cb36-4efc-9572-e645e2eb59d8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.121:15021/healthz/ready\": dial tcp 10.133.0.121:15021: connect: connection refused" Oct 30 21:08:22.822244 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:08:22.822209 5378 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w4-88da25-openshift-default-6c4ccc4cd9-tcsdd" Oct 30 21:08:22.823137 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:08:22.823116 5378 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w4-88da25-openshift-default-6c4ccc4cd9-tcsdd" Oct 30 21:08:25.784448 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:08:25.784387 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:08:29.219855 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:08:29.219821 5378 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-9926f7-646b87d4d7-pgzdg" Oct 30 21:08:29.220343 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:08:29.219891 5378 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-a-9926f7-57bb5db5fb-b224f" Oct 30 21:08:29.962598 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:08:29.962558 5378 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-7x9jg_c2ff359c-cf92-4a5b-8610-f6eb41acbe73/ovn-acl-logging/0.log" Oct 30 21:08:29.970817 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:08:29.970795 5378 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-7x9jg_c2ff359c-cf92-4a5b-8610-f6eb41acbe73/ovn-acl-logging/0.log" Oct 30 21:08:30.836791 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:08:30.836759 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:08:35.495032 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:08:35.495000 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:08:36.271872 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:08:36.271832 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:08:40.212168 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:08:40.212072 5378 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-r9g59"] Oct 30 21:08:40.366746 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:08:40.366706 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:08:40.548641 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:08:40.545201 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:08:43.103513 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:08:43.103464 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:08:43.237022 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:08:43.236991 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:08:45.501591 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:08:45.487239 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:08:46.165589 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:08:46.163553 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:08:46.299152 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:08:46.299111 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:08:48.187722 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:08:48.187676 5378 patch_prober.go:29] interesting pod/e2e-shared-b-w2-255f05-openshift-default-7db8bf49f-6bfml container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.127:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 21:08:48.188210 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:08:48.187745 5378 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-w2-255f05-openshift-default-7db8bf49f-6bfml" podUID="4955d910-8544-4a2b-a1fd-b414734aed56" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.127:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 21:08:48.839886 ip-10-0-138-149 kubenswrapper[5378]: E1030 21:08:48.839834 5378 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q6mx7" containerName="extractor" Oct 30 21:08:50.358873 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:08:50.358832 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:08:51.518476 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:08:51.518442 5378 generic.go:356] "Generic (PLEG): container finished" podID="1b385f61-b9d9-4f5d-8e71-7d86cd0d867d" containerID="b75ef819ca3ba0cace281a1b6a7a8378147dfa4b114d39242abfe2a7d5c9bb44" exitCode=0 Oct 30 21:08:51.519040 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:08:51.518518 5378 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-4d1110-openshift-default-6576975b6c-gvzcm" event={"ID":"1b385f61-b9d9-4f5d-8e71-7d86cd0d867d","Type":"ContainerDied","Data":"b75ef819ca3ba0cace281a1b6a7a8378147dfa4b114d39242abfe2a7d5c9bb44"} Oct 30 21:08:51.519122 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:08:51.519099 5378 scope.go:122] "RemoveContainer" containerID="b75ef819ca3ba0cace281a1b6a7a8378147dfa4b114d39242abfe2a7d5c9bb44" Oct 30 21:08:51.520431 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:08:51.520410 5378 generic.go:356] "Generic (PLEG): container finished" podID="4955d910-8544-4a2b-a1fd-b414734aed56" containerID="f436cafc6a27c4e22cde96a1784d61e2c40ce352ae7696fc0974d6435e60a1bf" exitCode=0 Oct 30 21:08:51.520539 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:08:51.520457 5378 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-255f05-openshift-default-7db8bf49f-6bfml" event={"ID":"4955d910-8544-4a2b-a1fd-b414734aed56","Type":"ContainerDied","Data":"f436cafc6a27c4e22cde96a1784d61e2c40ce352ae7696fc0974d6435e60a1bf"} Oct 30 21:08:51.520815 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:08:51.520797 5378 scope.go:122] "RemoveContainer" containerID="f436cafc6a27c4e22cde96a1784d61e2c40ce352ae7696fc0974d6435e60a1bf" Oct 30 21:08:51.521009 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:08:51.520983 5378 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 21:08:51.521091 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:08:51.521051 5378 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 21:08:51.521091 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:08:51.521087 5378 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 21:08:51.522344 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:08:51.522309 5378 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 21:08:51.522412 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:08:51.522395 5378 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 21:08:51.522450 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:08:51.522437 5378 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 21:08:52.534172 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:08:52.534137 5378 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-4d1110-openshift-default-6576975b6c-gvzcm" event={"ID":"1b385f61-b9d9-4f5d-8e71-7d86cd0d867d","Type":"ContainerStarted","Data":"048a3802a64f4473e97950a182b7b9278ad0397bbbb2b485c86e47964adaf321"} Oct 30 21:08:52.536464 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:08:52.536437 5378 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-255f05-openshift-default-7db8bf49f-6bfml" event={"ID":"4955d910-8544-4a2b-a1fd-b414734aed56","Type":"ContainerStarted","Data":"54e9e27c0e5662cdc7708cccc54fa81ed8abb32708b17e78bee0d401d7486463"} Oct 30 21:08:53.045929 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:08:53.045894 5378 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w1-4d1110-openshift-default-6576975b6c-gvzcm" Oct 30 21:08:53.046958 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:08:53.046921 5378 patch_prober.go:29] interesting pod/e2e-shared-a-w1-4d1110-openshift-default-6576975b6c-gvzcm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.123:15021/healthz/ready\": dial tcp 10.133.0.123:15021: connect: connection refused" start-of-body= Oct 30 21:08:53.047089 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:08:53.046970 5378 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-4d1110-openshift-default-6576975b6c-gvzcm" podUID="1b385f61-b9d9-4f5d-8e71-7d86cd0d867d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.123:15021/healthz/ready\": dial tcp 10.133.0.123:15021: connect: connection refused" Oct 30 21:08:53.115684 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:08:53.115643 5378 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w2-255f05-openshift-default-7db8bf49f-6bfml" Oct 30 21:08:53.115896 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:08:53.115873 5378 patch_prober.go:29] interesting pod/e2e-shared-b-w2-255f05-openshift-default-7db8bf49f-6bfml container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.127:15021/healthz/ready\": dial tcp 10.133.0.127:15021: connect: connection refused" start-of-body= Oct 30 21:08:53.115945 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:08:53.115922 5378 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-255f05-openshift-default-7db8bf49f-6bfml" podUID="4955d910-8544-4a2b-a1fd-b414734aed56" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.127:15021/healthz/ready\": dial tcp 10.133.0.127:15021: connect: connection refused" Oct 30 21:08:54.046219 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:08:54.046177 5378 patch_prober.go:29] interesting pod/e2e-shared-a-w1-4d1110-openshift-default-6576975b6c-gvzcm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.123:15021/healthz/ready\": dial tcp 10.133.0.123:15021: connect: connection refused" start-of-body= Oct 30 21:08:54.046732 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:08:54.046235 5378 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-4d1110-openshift-default-6576975b6c-gvzcm" podUID="1b385f61-b9d9-4f5d-8e71-7d86cd0d867d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.123:15021/healthz/ready\": dial tcp 10.133.0.123:15021: connect: connection refused" Oct 30 21:08:54.115128 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:08:54.115097 5378 patch_prober.go:29] interesting pod/e2e-shared-b-w2-255f05-openshift-default-7db8bf49f-6bfml container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.127:15021/healthz/ready\": dial tcp 10.133.0.127:15021: connect: connection refused" start-of-body= Oct 30 21:08:54.115358 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:08:54.115337 5378 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-255f05-openshift-default-7db8bf49f-6bfml" podUID="4955d910-8544-4a2b-a1fd-b414734aed56" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.127:15021/healthz/ready\": dial tcp 10.133.0.127:15021: connect: connection refused" Oct 30 21:08:55.046411 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:08:55.046368 5378 patch_prober.go:29] interesting pod/e2e-shared-a-w1-4d1110-openshift-default-6576975b6c-gvzcm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.123:15021/healthz/ready\": dial tcp 10.133.0.123:15021: connect: connection refused" start-of-body= Oct 30 21:08:55.046978 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:08:55.046448 5378 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-4d1110-openshift-default-6576975b6c-gvzcm" podUID="1b385f61-b9d9-4f5d-8e71-7d86cd0d867d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.123:15021/healthz/ready\": dial tcp 10.133.0.123:15021: connect: connection refused" Oct 30 21:08:55.116795 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:08:55.115736 5378 patch_prober.go:29] interesting pod/e2e-shared-b-w2-255f05-openshift-default-7db8bf49f-6bfml container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.127:15021/healthz/ready\": dial tcp 10.133.0.127:15021: connect: connection refused" start-of-body= Oct 30 21:08:55.116795 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:08:55.115830 5378 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-255f05-openshift-default-7db8bf49f-6bfml" podUID="4955d910-8544-4a2b-a1fd-b414734aed56" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.127:15021/healthz/ready\": dial tcp 10.133.0.127:15021: connect: connection refused" Oct 30 21:08:55.506328 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:08:55.506157 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:08:55.884852 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:08:55.883681 5378 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w1-4d1110-openshift-default-6576975b6c-gvzcm" Oct 30 21:08:56.046750 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:08:56.046708 5378 patch_prober.go:29] interesting pod/e2e-shared-a-w1-4d1110-openshift-default-6576975b6c-gvzcm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.123:15021/healthz/ready\": dial tcp 10.133.0.123:15021: connect: connection refused" start-of-body= Oct 30 21:08:56.047137 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:08:56.046773 5378 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-4d1110-openshift-default-6576975b6c-gvzcm" podUID="1b385f61-b9d9-4f5d-8e71-7d86cd0d867d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.123:15021/healthz/ready\": dial tcp 10.133.0.123:15021: connect: connection refused" Oct 30 21:08:56.115642 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:08:56.115607 5378 patch_prober.go:29] interesting pod/e2e-shared-b-w2-255f05-openshift-default-7db8bf49f-6bfml container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.127:15021/healthz/ready\": dial tcp 10.133.0.127:15021: connect: connection refused" start-of-body= Oct 30 21:08:56.115816 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:08:56.115662 5378 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-255f05-openshift-default-7db8bf49f-6bfml" podUID="4955d910-8544-4a2b-a1fd-b414734aed56" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.127:15021/healthz/ready\": dial tcp 10.133.0.127:15021: connect: connection refused" Oct 30 21:08:57.046360 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:08:57.046322 5378 patch_prober.go:29] interesting pod/e2e-shared-a-w1-4d1110-openshift-default-6576975b6c-gvzcm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.123:15021/healthz/ready\": dial tcp 10.133.0.123:15021: connect: connection refused" start-of-body= Oct 30 21:08:57.046523 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:08:57.046385 5378 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-4d1110-openshift-default-6576975b6c-gvzcm" podUID="1b385f61-b9d9-4f5d-8e71-7d86cd0d867d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.123:15021/healthz/ready\": dial tcp 10.133.0.123:15021: connect: connection refused" Oct 30 21:08:57.115275 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:08:57.115239 5378 patch_prober.go:29] interesting pod/e2e-shared-b-w2-255f05-openshift-default-7db8bf49f-6bfml container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.127:15021/healthz/ready\": dial tcp 10.133.0.127:15021: connect: connection refused" start-of-body= Oct 30 21:08:57.115700 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:08:57.115295 5378 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-255f05-openshift-default-7db8bf49f-6bfml" podUID="4955d910-8544-4a2b-a1fd-b414734aed56" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.127:15021/healthz/ready\": dial tcp 10.133.0.127:15021: connect: connection refused" Oct 30 21:08:58.051509 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:08:58.051469 5378 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w1-4d1110-openshift-default-6576975b6c-gvzcm" Oct 30 21:08:58.052486 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:08:58.052466 5378 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w1-4d1110-openshift-default-6576975b6c-gvzcm" Oct 30 21:08:58.119910 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:08:58.119869 5378 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w2-255f05-openshift-default-7db8bf49f-6bfml" Oct 30 21:08:58.120417 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:08:58.120266 5378 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w2-255f05-openshift-default-7db8bf49f-6bfml" Oct 30 21:08:58.120996 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:08:58.120969 5378 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w2-255f05-openshift-default-7db8bf49f-6bfml" Oct 30 21:09:00.922017 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:09:00.921979 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:09:01.075623 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:09:01.075587 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:09:01.219257 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:09:01.219154 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:09:03.743884 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:09:03.743671 5378 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-f9f277/simulator-w1-f9f277-kserve-5d9685f85c-2h78x" podUID="dc662e9d-914c-4dfb-a86f-0fb048f7b690" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": context deadline exceeded" Oct 30 21:09:03.774690 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:09:03.774643 5378 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-781058/distinct-w0-781058-kserve-8bdf9bfcd-hzdfk" podUID="a9678fc2-1c56-4d45-9007-eee5049e0f48" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:09:03.810587 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:09:03.801874 5378 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-d061d0/distinct-w2-d061d0-kserve-679c7cd7c4-8czrv" podUID="2c139b9f-c7df-4675-b431-95d4136095e5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/ready\": context deadline exceeded" Oct 30 21:09:03.854247 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:09:03.849059 5378 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-1fdaa7/premium-w4-1fdaa7-kserve-55dffbd5cd-gvpxq" podUID="ec282a4b-742f-4382-a296-335dcd2e9636" containerName="main" probeResult="failure" output="Get \"https://10.133.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:09:03.854247 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:09:03.849420 5378 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-1fdaa7/simulator-w4-1fdaa7-kserve-9866d468c-fwmsk" podUID="09219832-010e-4ba8-bfb9-5a2884b358e1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:09:03.864702 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:09:03.864655 5378 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-8fa2a7/simulator-w6-8fa2a7-kserve-6cbf5bd894-lfpcq" podUID="f409f469-71fc-445b-bdc8-b8e84a4d1dc6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:09:05.176167 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:09:05.176130 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:09:05.621833 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:09:05.621778 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:09:06.459772 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:09:06.459733 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:09:07.098647 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:09:07.085200 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:09:07.891032 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:09:07.890996 5378 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w6-7eaf74-openshift-default-69c8f48765-tmsjl"] Oct 30 21:09:07.895552 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:09:07.895526 5378 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w6-7eaf74-openshift-default-69c8f48765-tmsjl" Oct 30 21:09:07.954896 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:09:07.954843 5378 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w6-7eaf74-openshift-default-dockercfg-n66xg\"" Oct 30 21:09:07.998477 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:09:07.998439 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:09:08.007111 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:09:08.007061 5378 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w6-7eaf74-openshift-default-69c8f48765-tmsjl"] Oct 30 21:09:08.083338 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:09:08.083289 5378 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3559349f-6be4-4ef6-8061-59c7d1a8b678-istio-envoy\") pod \"e2e-shared-b-w6-7eaf74-openshift-default-69c8f48765-tmsjl\" (UID: \"3559349f-6be4-4ef6-8061-59c7d1a8b678\") " pod="openshift-ingress/e2e-shared-b-w6-7eaf74-openshift-default-69c8f48765-tmsjl" Oct 30 21:09:08.083637 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:09:08.083605 5378 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3559349f-6be4-4ef6-8061-59c7d1a8b678-istio-data\") pod \"e2e-shared-b-w6-7eaf74-openshift-default-69c8f48765-tmsjl\" (UID: \"3559349f-6be4-4ef6-8061-59c7d1a8b678\") " pod="openshift-ingress/e2e-shared-b-w6-7eaf74-openshift-default-69c8f48765-tmsjl" Oct 30 21:09:08.083854 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:09:08.083835 5378 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3559349f-6be4-4ef6-8061-59c7d1a8b678-credential-socket\") pod \"e2e-shared-b-w6-7eaf74-openshift-default-69c8f48765-tmsjl\" (UID: \"3559349f-6be4-4ef6-8061-59c7d1a8b678\") " pod="openshift-ingress/e2e-shared-b-w6-7eaf74-openshift-default-69c8f48765-tmsjl" Oct 30 21:09:08.084025 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:09:08.083997 5378 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3559349f-6be4-4ef6-8061-59c7d1a8b678-istiod-ca-cert\") pod \"e2e-shared-b-w6-7eaf74-openshift-default-69c8f48765-tmsjl\" (UID: \"3559349f-6be4-4ef6-8061-59c7d1a8b678\") " pod="openshift-ingress/e2e-shared-b-w6-7eaf74-openshift-default-69c8f48765-tmsjl" Oct 30 21:09:08.084247 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:09:08.084232 5378 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3559349f-6be4-4ef6-8061-59c7d1a8b678-istio-token\") pod \"e2e-shared-b-w6-7eaf74-openshift-default-69c8f48765-tmsjl\" (UID: \"3559349f-6be4-4ef6-8061-59c7d1a8b678\") " pod="openshift-ingress/e2e-shared-b-w6-7eaf74-openshift-default-69c8f48765-tmsjl" Oct 30 21:09:08.084412 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:09:08.084397 5378 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c7mmd\" (UniqueName: \"kubernetes.io/projected/3559349f-6be4-4ef6-8061-59c7d1a8b678-kube-api-access-c7mmd\") pod \"e2e-shared-b-w6-7eaf74-openshift-default-69c8f48765-tmsjl\" (UID: \"3559349f-6be4-4ef6-8061-59c7d1a8b678\") " pod="openshift-ingress/e2e-shared-b-w6-7eaf74-openshift-default-69c8f48765-tmsjl" Oct 30 21:09:08.085048 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:09:08.085003 5378 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3559349f-6be4-4ef6-8061-59c7d1a8b678-workload-certs\") pod \"e2e-shared-b-w6-7eaf74-openshift-default-69c8f48765-tmsjl\" (UID: \"3559349f-6be4-4ef6-8061-59c7d1a8b678\") " pod="openshift-ingress/e2e-shared-b-w6-7eaf74-openshift-default-69c8f48765-tmsjl" Oct 30 21:09:08.085267 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:09:08.085249 5378 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3559349f-6be4-4ef6-8061-59c7d1a8b678-workload-socket\") pod \"e2e-shared-b-w6-7eaf74-openshift-default-69c8f48765-tmsjl\" (UID: \"3559349f-6be4-4ef6-8061-59c7d1a8b678\") " pod="openshift-ingress/e2e-shared-b-w6-7eaf74-openshift-default-69c8f48765-tmsjl" Oct 30 21:09:08.085431 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:09:08.085416 5378 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3559349f-6be4-4ef6-8061-59c7d1a8b678-istio-podinfo\") pod \"e2e-shared-b-w6-7eaf74-openshift-default-69c8f48765-tmsjl\" (UID: \"3559349f-6be4-4ef6-8061-59c7d1a8b678\") " pod="openshift-ingress/e2e-shared-b-w6-7eaf74-openshift-default-69c8f48765-tmsjl" Oct 30 21:09:08.188458 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:09:08.187450 5378 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3559349f-6be4-4ef6-8061-59c7d1a8b678-istio-envoy\") pod \"e2e-shared-b-w6-7eaf74-openshift-default-69c8f48765-tmsjl\" (UID: \"3559349f-6be4-4ef6-8061-59c7d1a8b678\") " pod="openshift-ingress/e2e-shared-b-w6-7eaf74-openshift-default-69c8f48765-tmsjl" Oct 30 21:09:08.188458 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:09:08.187504 5378 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3559349f-6be4-4ef6-8061-59c7d1a8b678-istio-data\") pod \"e2e-shared-b-w6-7eaf74-openshift-default-69c8f48765-tmsjl\" (UID: \"3559349f-6be4-4ef6-8061-59c7d1a8b678\") " pod="openshift-ingress/e2e-shared-b-w6-7eaf74-openshift-default-69c8f48765-tmsjl" Oct 30 21:09:08.188458 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:09:08.187576 5378 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3559349f-6be4-4ef6-8061-59c7d1a8b678-credential-socket\") pod \"e2e-shared-b-w6-7eaf74-openshift-default-69c8f48765-tmsjl\" (UID: \"3559349f-6be4-4ef6-8061-59c7d1a8b678\") " pod="openshift-ingress/e2e-shared-b-w6-7eaf74-openshift-default-69c8f48765-tmsjl" Oct 30 21:09:08.188458 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:09:08.187611 5378 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3559349f-6be4-4ef6-8061-59c7d1a8b678-istiod-ca-cert\") pod \"e2e-shared-b-w6-7eaf74-openshift-default-69c8f48765-tmsjl\" (UID: \"3559349f-6be4-4ef6-8061-59c7d1a8b678\") " pod="openshift-ingress/e2e-shared-b-w6-7eaf74-openshift-default-69c8f48765-tmsjl" Oct 30 21:09:08.188458 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:09:08.187678 5378 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3559349f-6be4-4ef6-8061-59c7d1a8b678-istio-token\") pod \"e2e-shared-b-w6-7eaf74-openshift-default-69c8f48765-tmsjl\" (UID: \"3559349f-6be4-4ef6-8061-59c7d1a8b678\") " pod="openshift-ingress/e2e-shared-b-w6-7eaf74-openshift-default-69c8f48765-tmsjl" Oct 30 21:09:08.188458 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:09:08.187724 5378 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c7mmd\" (UniqueName: \"kubernetes.io/projected/3559349f-6be4-4ef6-8061-59c7d1a8b678-kube-api-access-c7mmd\") pod \"e2e-shared-b-w6-7eaf74-openshift-default-69c8f48765-tmsjl\" (UID: \"3559349f-6be4-4ef6-8061-59c7d1a8b678\") " pod="openshift-ingress/e2e-shared-b-w6-7eaf74-openshift-default-69c8f48765-tmsjl" Oct 30 21:09:08.188458 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:09:08.187809 5378 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3559349f-6be4-4ef6-8061-59c7d1a8b678-workload-certs\") pod \"e2e-shared-b-w6-7eaf74-openshift-default-69c8f48765-tmsjl\" (UID: \"3559349f-6be4-4ef6-8061-59c7d1a8b678\") " pod="openshift-ingress/e2e-shared-b-w6-7eaf74-openshift-default-69c8f48765-tmsjl" Oct 30 21:09:08.188458 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:09:08.187855 5378 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3559349f-6be4-4ef6-8061-59c7d1a8b678-workload-socket\") pod \"e2e-shared-b-w6-7eaf74-openshift-default-69c8f48765-tmsjl\" (UID: \"3559349f-6be4-4ef6-8061-59c7d1a8b678\") " pod="openshift-ingress/e2e-shared-b-w6-7eaf74-openshift-default-69c8f48765-tmsjl" Oct 30 21:09:08.188458 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:09:08.187914 5378 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3559349f-6be4-4ef6-8061-59c7d1a8b678-istio-podinfo\") pod \"e2e-shared-b-w6-7eaf74-openshift-default-69c8f48765-tmsjl\" (UID: \"3559349f-6be4-4ef6-8061-59c7d1a8b678\") " pod="openshift-ingress/e2e-shared-b-w6-7eaf74-openshift-default-69c8f48765-tmsjl" Oct 30 21:09:08.188997 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:09:08.188529 5378 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3559349f-6be4-4ef6-8061-59c7d1a8b678-istio-data\") pod \"e2e-shared-b-w6-7eaf74-openshift-default-69c8f48765-tmsjl\" (UID: \"3559349f-6be4-4ef6-8061-59c7d1a8b678\") " pod="openshift-ingress/e2e-shared-b-w6-7eaf74-openshift-default-69c8f48765-tmsjl" Oct 30 21:09:08.188997 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:09:08.188862 5378 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3559349f-6be4-4ef6-8061-59c7d1a8b678-credential-socket\") pod \"e2e-shared-b-w6-7eaf74-openshift-default-69c8f48765-tmsjl\" (UID: \"3559349f-6be4-4ef6-8061-59c7d1a8b678\") " pod="openshift-ingress/e2e-shared-b-w6-7eaf74-openshift-default-69c8f48765-tmsjl" Oct 30 21:09:08.189742 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:09:08.189435 5378 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3559349f-6be4-4ef6-8061-59c7d1a8b678-workload-certs\") pod \"e2e-shared-b-w6-7eaf74-openshift-default-69c8f48765-tmsjl\" (UID: \"3559349f-6be4-4ef6-8061-59c7d1a8b678\") " pod="openshift-ingress/e2e-shared-b-w6-7eaf74-openshift-default-69c8f48765-tmsjl" Oct 30 21:09:08.189742 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:09:08.189502 5378 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3559349f-6be4-4ef6-8061-59c7d1a8b678-istiod-ca-cert\") pod \"e2e-shared-b-w6-7eaf74-openshift-default-69c8f48765-tmsjl\" (UID: \"3559349f-6be4-4ef6-8061-59c7d1a8b678\") " pod="openshift-ingress/e2e-shared-b-w6-7eaf74-openshift-default-69c8f48765-tmsjl" Oct 30 21:09:08.189920 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:09:08.189781 5378 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3559349f-6be4-4ef6-8061-59c7d1a8b678-workload-socket\") pod \"e2e-shared-b-w6-7eaf74-openshift-default-69c8f48765-tmsjl\" (UID: \"3559349f-6be4-4ef6-8061-59c7d1a8b678\") " pod="openshift-ingress/e2e-shared-b-w6-7eaf74-openshift-default-69c8f48765-tmsjl" Oct 30 21:09:08.191092 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:09:08.191073 5378 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3559349f-6be4-4ef6-8061-59c7d1a8b678-istio-podinfo\") pod \"e2e-shared-b-w6-7eaf74-openshift-default-69c8f48765-tmsjl\" (UID: \"3559349f-6be4-4ef6-8061-59c7d1a8b678\") " pod="openshift-ingress/e2e-shared-b-w6-7eaf74-openshift-default-69c8f48765-tmsjl" Oct 30 21:09:08.191760 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:09:08.191726 5378 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3559349f-6be4-4ef6-8061-59c7d1a8b678-istio-envoy\") pod \"e2e-shared-b-w6-7eaf74-openshift-default-69c8f48765-tmsjl\" (UID: \"3559349f-6be4-4ef6-8061-59c7d1a8b678\") " pod="openshift-ingress/e2e-shared-b-w6-7eaf74-openshift-default-69c8f48765-tmsjl" Oct 30 21:09:08.200471 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:09:08.200436 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:09:08.248268 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:09:08.248238 5378 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3559349f-6be4-4ef6-8061-59c7d1a8b678-istio-token\") pod \"e2e-shared-b-w6-7eaf74-openshift-default-69c8f48765-tmsjl\" (UID: \"3559349f-6be4-4ef6-8061-59c7d1a8b678\") " pod="openshift-ingress/e2e-shared-b-w6-7eaf74-openshift-default-69c8f48765-tmsjl" Oct 30 21:09:08.248459 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:09:08.248310 5378 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c7mmd\" (UniqueName: \"kubernetes.io/projected/3559349f-6be4-4ef6-8061-59c7d1a8b678-kube-api-access-c7mmd\") pod \"e2e-shared-b-w6-7eaf74-openshift-default-69c8f48765-tmsjl\" (UID: \"3559349f-6be4-4ef6-8061-59c7d1a8b678\") " pod="openshift-ingress/e2e-shared-b-w6-7eaf74-openshift-default-69c8f48765-tmsjl" Oct 30 21:09:08.272483 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:09:08.272445 5378 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-a-ef7e2d-openshift-default-6b6dbc765f-679n5"] Oct 30 21:09:08.276469 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:09:08.276450 5378 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-ef7e2d-openshift-default-6b6dbc765f-679n5" Oct 30 21:09:08.331630 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:09:08.331490 5378 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-a-ef7e2d-openshift-default-dockercfg-fhxft\"" Oct 30 21:09:08.339341 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:09:08.339312 5378 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-ef7e2d-openshift-default-6b6dbc765f-679n5"] Oct 30 21:09:08.364378 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:09:08.364343 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:09:08.391741 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:09:08.391706 5378 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a5f9447f-85f1-40c0-b794-3a39315da3cc-istio-data\") pod \"e2e-api-a-ef7e2d-openshift-default-6b6dbc765f-679n5\" (UID: \"a5f9447f-85f1-40c0-b794-3a39315da3cc\") " pod="openshift-ingress/e2e-api-a-ef7e2d-openshift-default-6b6dbc765f-679n5" Oct 30 21:09:08.391907 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:09:08.391837 5378 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a5f9447f-85f1-40c0-b794-3a39315da3cc-workload-socket\") pod \"e2e-api-a-ef7e2d-openshift-default-6b6dbc765f-679n5\" (UID: \"a5f9447f-85f1-40c0-b794-3a39315da3cc\") " pod="openshift-ingress/e2e-api-a-ef7e2d-openshift-default-6b6dbc765f-679n5" Oct 30 21:09:08.391976 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:09:08.391926 5378 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a5f9447f-85f1-40c0-b794-3a39315da3cc-credential-socket\") pod \"e2e-api-a-ef7e2d-openshift-default-6b6dbc765f-679n5\" (UID: \"a5f9447f-85f1-40c0-b794-3a39315da3cc\") " pod="openshift-ingress/e2e-api-a-ef7e2d-openshift-default-6b6dbc765f-679n5" Oct 30 21:09:08.392039 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:09:08.391996 5378 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a5f9447f-85f1-40c0-b794-3a39315da3cc-istio-token\") pod \"e2e-api-a-ef7e2d-openshift-default-6b6dbc765f-679n5\" (UID: \"a5f9447f-85f1-40c0-b794-3a39315da3cc\") " pod="openshift-ingress/e2e-api-a-ef7e2d-openshift-default-6b6dbc765f-679n5" Oct 30 21:09:08.392090 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:09:08.392035 5378 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a5f9447f-85f1-40c0-b794-3a39315da3cc-istiod-ca-cert\") pod \"e2e-api-a-ef7e2d-openshift-default-6b6dbc765f-679n5\" (UID: \"a5f9447f-85f1-40c0-b794-3a39315da3cc\") " pod="openshift-ingress/e2e-api-a-ef7e2d-openshift-default-6b6dbc765f-679n5" Oct 30 21:09:08.392139 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:09:08.392120 5378 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a5f9447f-85f1-40c0-b794-3a39315da3cc-workload-certs\") pod \"e2e-api-a-ef7e2d-openshift-default-6b6dbc765f-679n5\" (UID: \"a5f9447f-85f1-40c0-b794-3a39315da3cc\") " pod="openshift-ingress/e2e-api-a-ef7e2d-openshift-default-6b6dbc765f-679n5" Oct 30 21:09:08.392175 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:09:08.392166 5378 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a5f9447f-85f1-40c0-b794-3a39315da3cc-istio-envoy\") pod \"e2e-api-a-ef7e2d-openshift-default-6b6dbc765f-679n5\" (UID: \"a5f9447f-85f1-40c0-b794-3a39315da3cc\") " pod="openshift-ingress/e2e-api-a-ef7e2d-openshift-default-6b6dbc765f-679n5" Oct 30 21:09:08.392244 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:09:08.392231 5378 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jh4g2\" (UniqueName: \"kubernetes.io/projected/a5f9447f-85f1-40c0-b794-3a39315da3cc-kube-api-access-jh4g2\") pod \"e2e-api-a-ef7e2d-openshift-default-6b6dbc765f-679n5\" (UID: \"a5f9447f-85f1-40c0-b794-3a39315da3cc\") " pod="openshift-ingress/e2e-api-a-ef7e2d-openshift-default-6b6dbc765f-679n5" Oct 30 21:09:08.392331 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:09:08.392320 5378 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a5f9447f-85f1-40c0-b794-3a39315da3cc-istio-podinfo\") pod \"e2e-api-a-ef7e2d-openshift-default-6b6dbc765f-679n5\" (UID: \"a5f9447f-85f1-40c0-b794-3a39315da3cc\") " pod="openshift-ingress/e2e-api-a-ef7e2d-openshift-default-6b6dbc765f-679n5" Oct 30 21:09:08.496454 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:09:08.496369 5378 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a5f9447f-85f1-40c0-b794-3a39315da3cc-istio-data\") pod \"e2e-api-a-ef7e2d-openshift-default-6b6dbc765f-679n5\" (UID: \"a5f9447f-85f1-40c0-b794-3a39315da3cc\") " pod="openshift-ingress/e2e-api-a-ef7e2d-openshift-default-6b6dbc765f-679n5" Oct 30 21:09:08.496664 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:09:08.496481 5378 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a5f9447f-85f1-40c0-b794-3a39315da3cc-workload-socket\") pod \"e2e-api-a-ef7e2d-openshift-default-6b6dbc765f-679n5\" (UID: \"a5f9447f-85f1-40c0-b794-3a39315da3cc\") " pod="openshift-ingress/e2e-api-a-ef7e2d-openshift-default-6b6dbc765f-679n5" Oct 30 21:09:08.496664 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:09:08.496618 5378 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a5f9447f-85f1-40c0-b794-3a39315da3cc-credential-socket\") pod \"e2e-api-a-ef7e2d-openshift-default-6b6dbc765f-679n5\" (UID: \"a5f9447f-85f1-40c0-b794-3a39315da3cc\") " pod="openshift-ingress/e2e-api-a-ef7e2d-openshift-default-6b6dbc765f-679n5" Oct 30 21:09:08.496664 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:09:08.496643 5378 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a5f9447f-85f1-40c0-b794-3a39315da3cc-istio-token\") pod \"e2e-api-a-ef7e2d-openshift-default-6b6dbc765f-679n5\" (UID: \"a5f9447f-85f1-40c0-b794-3a39315da3cc\") " pod="openshift-ingress/e2e-api-a-ef7e2d-openshift-default-6b6dbc765f-679n5" Oct 30 21:09:08.496875 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:09:08.496668 5378 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a5f9447f-85f1-40c0-b794-3a39315da3cc-istiod-ca-cert\") pod \"e2e-api-a-ef7e2d-openshift-default-6b6dbc765f-679n5\" (UID: \"a5f9447f-85f1-40c0-b794-3a39315da3cc\") " pod="openshift-ingress/e2e-api-a-ef7e2d-openshift-default-6b6dbc765f-679n5" Oct 30 21:09:08.496875 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:09:08.496730 5378 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a5f9447f-85f1-40c0-b794-3a39315da3cc-workload-certs\") pod \"e2e-api-a-ef7e2d-openshift-default-6b6dbc765f-679n5\" (UID: \"a5f9447f-85f1-40c0-b794-3a39315da3cc\") " pod="openshift-ingress/e2e-api-a-ef7e2d-openshift-default-6b6dbc765f-679n5" Oct 30 21:09:08.496875 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:09:08.496785 5378 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a5f9447f-85f1-40c0-b794-3a39315da3cc-istio-envoy\") pod \"e2e-api-a-ef7e2d-openshift-default-6b6dbc765f-679n5\" (UID: \"a5f9447f-85f1-40c0-b794-3a39315da3cc\") " pod="openshift-ingress/e2e-api-a-ef7e2d-openshift-default-6b6dbc765f-679n5" Oct 30 21:09:08.496875 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:09:08.496839 5378 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jh4g2\" (UniqueName: \"kubernetes.io/projected/a5f9447f-85f1-40c0-b794-3a39315da3cc-kube-api-access-jh4g2\") pod \"e2e-api-a-ef7e2d-openshift-default-6b6dbc765f-679n5\" (UID: \"a5f9447f-85f1-40c0-b794-3a39315da3cc\") " pod="openshift-ingress/e2e-api-a-ef7e2d-openshift-default-6b6dbc765f-679n5" Oct 30 21:09:08.497076 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:09:08.496951 5378 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a5f9447f-85f1-40c0-b794-3a39315da3cc-istio-podinfo\") pod \"e2e-api-a-ef7e2d-openshift-default-6b6dbc765f-679n5\" (UID: \"a5f9447f-85f1-40c0-b794-3a39315da3cc\") " pod="openshift-ingress/e2e-api-a-ef7e2d-openshift-default-6b6dbc765f-679n5" Oct 30 21:09:08.499968 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:09:08.498377 5378 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a5f9447f-85f1-40c0-b794-3a39315da3cc-istio-data\") pod \"e2e-api-a-ef7e2d-openshift-default-6b6dbc765f-679n5\" (UID: \"a5f9447f-85f1-40c0-b794-3a39315da3cc\") " pod="openshift-ingress/e2e-api-a-ef7e2d-openshift-default-6b6dbc765f-679n5" Oct 30 21:09:08.499968 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:09:08.498659 5378 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a5f9447f-85f1-40c0-b794-3a39315da3cc-workload-socket\") pod \"e2e-api-a-ef7e2d-openshift-default-6b6dbc765f-679n5\" (UID: \"a5f9447f-85f1-40c0-b794-3a39315da3cc\") " pod="openshift-ingress/e2e-api-a-ef7e2d-openshift-default-6b6dbc765f-679n5" Oct 30 21:09:08.499968 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:09:08.498924 5378 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a5f9447f-85f1-40c0-b794-3a39315da3cc-credential-socket\") pod \"e2e-api-a-ef7e2d-openshift-default-6b6dbc765f-679n5\" (UID: \"a5f9447f-85f1-40c0-b794-3a39315da3cc\") " pod="openshift-ingress/e2e-api-a-ef7e2d-openshift-default-6b6dbc765f-679n5" Oct 30 21:09:08.499968 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:09:08.499144 5378 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a5f9447f-85f1-40c0-b794-3a39315da3cc-workload-certs\") pod \"e2e-api-a-ef7e2d-openshift-default-6b6dbc765f-679n5\" (UID: \"a5f9447f-85f1-40c0-b794-3a39315da3cc\") " pod="openshift-ingress/e2e-api-a-ef7e2d-openshift-default-6b6dbc765f-679n5" Oct 30 21:09:08.499968 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:09:08.499584 5378 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a5f9447f-85f1-40c0-b794-3a39315da3cc-istiod-ca-cert\") pod \"e2e-api-a-ef7e2d-openshift-default-6b6dbc765f-679n5\" (UID: \"a5f9447f-85f1-40c0-b794-3a39315da3cc\") " pod="openshift-ingress/e2e-api-a-ef7e2d-openshift-default-6b6dbc765f-679n5" Oct 30 21:09:08.521338 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:09:08.513487 5378 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w6-7eaf74-openshift-default-69c8f48765-tmsjl" Oct 30 21:09:08.530024 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:09:08.529993 5378 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a5f9447f-85f1-40c0-b794-3a39315da3cc-istio-envoy\") pod \"e2e-api-a-ef7e2d-openshift-default-6b6dbc765f-679n5\" (UID: \"a5f9447f-85f1-40c0-b794-3a39315da3cc\") " pod="openshift-ingress/e2e-api-a-ef7e2d-openshift-default-6b6dbc765f-679n5" Oct 30 21:09:08.531374 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:09:08.531350 5378 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a5f9447f-85f1-40c0-b794-3a39315da3cc-istio-podinfo\") pod \"e2e-api-a-ef7e2d-openshift-default-6b6dbc765f-679n5\" (UID: \"a5f9447f-85f1-40c0-b794-3a39315da3cc\") " pod="openshift-ingress/e2e-api-a-ef7e2d-openshift-default-6b6dbc765f-679n5" Oct 30 21:09:08.566282 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:09:08.566249 5378 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a5f9447f-85f1-40c0-b794-3a39315da3cc-istio-token\") pod \"e2e-api-a-ef7e2d-openshift-default-6b6dbc765f-679n5\" (UID: \"a5f9447f-85f1-40c0-b794-3a39315da3cc\") " pod="openshift-ingress/e2e-api-a-ef7e2d-openshift-default-6b6dbc765f-679n5" Oct 30 21:09:08.575319 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:09:08.575283 5378 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jh4g2\" (UniqueName: \"kubernetes.io/projected/a5f9447f-85f1-40c0-b794-3a39315da3cc-kube-api-access-jh4g2\") pod \"e2e-api-a-ef7e2d-openshift-default-6b6dbc765f-679n5\" (UID: \"a5f9447f-85f1-40c0-b794-3a39315da3cc\") " pod="openshift-ingress/e2e-api-a-ef7e2d-openshift-default-6b6dbc765f-679n5" Oct 30 21:09:08.633191 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:09:08.594519 5378 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-ef7e2d-openshift-default-6b6dbc765f-679n5" Oct 30 21:09:08.645750 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:09:08.645109 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:09:08.993540 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:09:08.993504 5378 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w6-7eaf74-openshift-default-69c8f48765-tmsjl"] Oct 30 21:09:09.038810 ip-10-0-138-149 kubenswrapper[5378]: W1030 21:09:09.038785 5378 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3559349f_6be4_4ef6_8061_59c7d1a8b678.slice/crio-eef112527c199e727a3c6d033c3386cd15026f35ca26e9c137c754e20da072d7 WatchSource:0}: Error finding container eef112527c199e727a3c6d033c3386cd15026f35ca26e9c137c754e20da072d7: Status 404 returned error can't find the container with id eef112527c199e727a3c6d033c3386cd15026f35ca26e9c137c754e20da072d7 Oct 30 21:09:09.041803 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:09:09.041766 5378 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 21:09:09.041903 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:09:09.041836 5378 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 21:09:09.041903 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:09:09.041871 5378 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 21:09:09.056314 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:09:09.056288 5378 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-ef7e2d-openshift-default-6b6dbc765f-679n5"] Oct 30 21:09:09.063760 ip-10-0-138-149 kubenswrapper[5378]: W1030 21:09:09.063730 5378 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda5f9447f_85f1_40c0_b794_3a39315da3cc.slice/crio-2702d21e7604e7d1095494b46ab9c4e9bd63a1f6957d6a4c279dcebcebbc1237 WatchSource:0}: Error finding container 2702d21e7604e7d1095494b46ab9c4e9bd63a1f6957d6a4c279dcebcebbc1237: Status 404 returned error can't find the container with id 2702d21e7604e7d1095494b46ab9c4e9bd63a1f6957d6a4c279dcebcebbc1237 Oct 30 21:09:09.066904 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:09:09.066868 5378 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 21:09:09.067024 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:09:09.066945 5378 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 21:09:09.067024 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:09:09.066981 5378 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 21:09:09.933284 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:09:09.933242 5378 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-ef7e2d-openshift-default-6b6dbc765f-679n5" event={"ID":"a5f9447f-85f1-40c0-b794-3a39315da3cc","Type":"ContainerStarted","Data":"e6437a86d114df1af144051ae1f9213ff0b6b1a6c558b343f17d3a8cecda7675"} Oct 30 21:09:09.933464 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:09:09.933348 5378 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-ef7e2d-openshift-default-6b6dbc765f-679n5" event={"ID":"a5f9447f-85f1-40c0-b794-3a39315da3cc","Type":"ContainerStarted","Data":"2702d21e7604e7d1095494b46ab9c4e9bd63a1f6957d6a4c279dcebcebbc1237"} Oct 30 21:09:09.935049 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:09:09.935021 5378 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w6-7eaf74-openshift-default-69c8f48765-tmsjl" event={"ID":"3559349f-6be4-4ef6-8061-59c7d1a8b678","Type":"ContainerStarted","Data":"7f6ec1fe4760ca648091b49d5fd6b282dd26bfb546798a789e4dc64db5bb8a6b"} Oct 30 21:09:09.935184 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:09:09.935068 5378 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w6-7eaf74-openshift-default-69c8f48765-tmsjl" event={"ID":"3559349f-6be4-4ef6-8061-59c7d1a8b678","Type":"ContainerStarted","Data":"eef112527c199e727a3c6d033c3386cd15026f35ca26e9c137c754e20da072d7"} Oct 30 21:09:10.133078 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:09:10.133019 5378 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-a-ef7e2d-openshift-default-6b6dbc765f-679n5" podStartSLOduration=2.133004302 podStartE2EDuration="2.133004302s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:09:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:09:10.131744755 +0000 UTC m=+1840.559858694" watchObservedRunningTime="2026-10-30 21:09:10.133004302 +0000 UTC m=+1840.561118295" Oct 30 21:09:10.259870 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:09:10.259772 5378 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w6-7eaf74-openshift-default-69c8f48765-tmsjl" podStartSLOduration=3.25975544 podStartE2EDuration="3.25975544s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:09:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:09:10.25692883 +0000 UTC m=+1840.685042820" watchObservedRunningTime="2026-10-30 21:09:10.25975544 +0000 UTC m=+1840.687869359" Oct 30 21:09:10.522409 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:09:10.522319 5378 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w6-7eaf74-openshift-default-69c8f48765-tmsjl" Oct 30 21:09:10.528243 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:09:10.528220 5378 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w6-7eaf74-openshift-default-69c8f48765-tmsjl" Oct 30 21:09:10.603585 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:09:10.603529 5378 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-ef7e2d-openshift-default-6b6dbc765f-679n5" Oct 30 21:09:10.609729 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:09:10.609701 5378 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-ef7e2d-openshift-default-6b6dbc765f-679n5" Oct 30 21:09:10.944189 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:09:10.944161 5378 generic.go:356] "Generic (PLEG): container finished" podID="d3278471-08a8-4d1e-8030-c6256d75265d" containerID="b66cc3f1cfa154e91f53a673aaac48f9fcd49cde853157e43d7fc96cae4a8cf5" exitCode=0 Oct 30 21:09:10.944640 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:09:10.944229 5378 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w6-dc6125-openshift-default-659bcc49bf-ms7zg" event={"ID":"d3278471-08a8-4d1e-8030-c6256d75265d","Type":"ContainerDied","Data":"b66cc3f1cfa154e91f53a673aaac48f9fcd49cde853157e43d7fc96cae4a8cf5"} Oct 30 21:09:10.944724 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:09:10.944674 5378 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w6-7eaf74-openshift-default-69c8f48765-tmsjl" Oct 30 21:09:10.944724 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:09:10.944721 5378 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-ef7e2d-openshift-default-6b6dbc765f-679n5" Oct 30 21:09:10.945053 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:09:10.945030 5378 scope.go:122] "RemoveContainer" containerID="b66cc3f1cfa154e91f53a673aaac48f9fcd49cde853157e43d7fc96cae4a8cf5" Oct 30 21:09:10.945622 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:09:10.945605 5378 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w6-7eaf74-openshift-default-69c8f48765-tmsjl" Oct 30 21:09:10.945721 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:09:10.945693 5378 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-ef7e2d-openshift-default-6b6dbc765f-679n5" Oct 30 21:09:10.946690 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:09:10.946666 5378 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 21:09:10.946749 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:09:10.946727 5378 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 21:09:10.946788 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:09:10.946760 5378 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 21:09:11.979068 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:09:11.978096 5378 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w6-dc6125-openshift-default-659bcc49bf-ms7zg" event={"ID":"d3278471-08a8-4d1e-8030-c6256d75265d","Type":"ContainerStarted","Data":"131c50322efa7a5ed2cdfe258fb864f7e2a5f175926e843d96445986a020710f"} Oct 30 21:09:12.235118 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:09:12.235034 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:09:12.431008 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:09:12.430979 5378 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w6-dc6125-openshift-default-659bcc49bf-ms7zg" Oct 30 21:09:12.431182 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:09:12.431115 5378 patch_prober.go:29] interesting pod/e2e-shared-a-w6-dc6125-openshift-default-659bcc49bf-ms7zg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.122:15021/healthz/ready\": dial tcp 10.133.0.122:15021: connect: connection refused" start-of-body= Oct 30 21:09:12.431182 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:09:12.431159 5378 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w6-dc6125-openshift-default-659bcc49bf-ms7zg" podUID="d3278471-08a8-4d1e-8030-c6256d75265d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.122:15021/healthz/ready\": dial tcp 10.133.0.122:15021: connect: connection refused" Oct 30 21:09:13.178181 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:09:13.178076 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:09:13.324620 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:09:13.324554 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:09:13.430973 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:09:13.430895 5378 patch_prober.go:29] interesting pod/e2e-shared-a-w6-dc6125-openshift-default-659bcc49bf-ms7zg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.122:15021/healthz/ready\": dial tcp 10.133.0.122:15021: connect: connection refused" start-of-body= Oct 30 21:09:13.430973 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:09:13.430956 5378 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w6-dc6125-openshift-default-659bcc49bf-ms7zg" podUID="d3278471-08a8-4d1e-8030-c6256d75265d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.122:15021/healthz/ready\": dial tcp 10.133.0.122:15021: connect: connection refused" Oct 30 21:09:13.479055 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:09:13.479020 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:09:13.820868 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:09:13.820762 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:09:14.135600 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:09:14.135545 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:09:14.295010 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:09:14.294976 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:09:14.430352 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:09:14.430272 5378 patch_prober.go:29] interesting pod/e2e-shared-a-w6-dc6125-openshift-default-659bcc49bf-ms7zg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.122:15021/healthz/ready\": dial tcp 10.133.0.122:15021: connect: connection refused" start-of-body= Oct 30 21:09:14.430352 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:09:14.430339 5378 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w6-dc6125-openshift-default-659bcc49bf-ms7zg" podUID="d3278471-08a8-4d1e-8030-c6256d75265d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.122:15021/healthz/ready\": dial tcp 10.133.0.122:15021: connect: connection refused" Oct 30 21:09:15.430804 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:09:15.430766 5378 patch_prober.go:29] interesting pod/e2e-shared-a-w6-dc6125-openshift-default-659bcc49bf-ms7zg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.122:15021/healthz/ready\": dial tcp 10.133.0.122:15021: connect: connection refused" start-of-body= Oct 30 21:09:15.431225 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:09:15.430840 5378 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w6-dc6125-openshift-default-659bcc49bf-ms7zg" podUID="d3278471-08a8-4d1e-8030-c6256d75265d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.122:15021/healthz/ready\": dial tcp 10.133.0.122:15021: connect: connection refused" Oct 30 21:09:16.430209 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:09:16.430176 5378 patch_prober.go:29] interesting pod/e2e-shared-a-w6-dc6125-openshift-default-659bcc49bf-ms7zg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.122:15021/healthz/ready\": dial tcp 10.133.0.122:15021: connect: connection refused" start-of-body= Oct 30 21:09:16.430418 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:09:16.430237 5378 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w6-dc6125-openshift-default-659bcc49bf-ms7zg" podUID="d3278471-08a8-4d1e-8030-c6256d75265d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.122:15021/healthz/ready\": dial tcp 10.133.0.122:15021: connect: connection refused" Oct 30 21:09:16.441468 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:09:16.441430 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:09:16.733286 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:09:16.733173 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:09:17.432590 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:09:17.430215 5378 patch_prober.go:29] interesting pod/e2e-shared-a-w6-dc6125-openshift-default-659bcc49bf-ms7zg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.122:15021/healthz/ready\": dial tcp 10.133.0.122:15021: connect: connection refused" start-of-body= Oct 30 21:09:17.432590 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:09:17.430273 5378 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w6-dc6125-openshift-default-659bcc49bf-ms7zg" podUID="d3278471-08a8-4d1e-8030-c6256d75265d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.122:15021/healthz/ready\": dial tcp 10.133.0.122:15021: connect: connection refused" Oct 30 21:09:18.192263 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:09:18.192224 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:09:18.435301 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:09:18.435274 5378 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w6-dc6125-openshift-default-659bcc49bf-ms7zg" Oct 30 21:09:18.435612 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:09:18.435588 5378 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w6-dc6125-openshift-default-659bcc49bf-ms7zg" Oct 30 21:09:18.436240 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:09:18.436220 5378 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w6-dc6125-openshift-default-659bcc49bf-ms7zg" Oct 30 21:09:18.851529 ip-10-0-138-149 kubenswrapper[5378]: E1030 21:09:18.851490 5378 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q6mx7" containerName="extractor" Oct 30 21:09:20.590899 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:09:20.590870 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:09:21.245591 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:09:21.245542 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:09:23.229597 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:09:23.220130 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:09:23.375786 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:09:23.375755 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:09:23.517744 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:09:23.517108 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:09:24.141106 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:09:24.141066 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:09:24.328714 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:09:24.328664 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:09:24.754334 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:09:24.754291 5378 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w6-7eaf74-d47bd994b-tww2z"] Oct 30 21:09:24.763234 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:09:24.763207 5378 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w6-7eaf74-d47bd994b-tww2z" Oct 30 21:09:24.914587 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:09:24.882771 5378 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w6-7eaf74\"" Oct 30 21:09:24.914587 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:09:24.884880 5378 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w6-7eaf74-d47bd994b-tww2z"] Oct 30 21:09:24.914587 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:09:24.900207 5378 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w6-7eaf74-dockercfg-bhp7f\"" Oct 30 21:09:24.985716 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:09:24.985592 5378 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/72d82ec7-ffcb-4267-9639-81357c29430d-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w6-7eaf74-d47bd994b-tww2z\" (UID: \"72d82ec7-ffcb-4267-9639-81357c29430d\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w6-7eaf74-d47bd994b-tww2z" Oct 30 21:09:24.985980 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:09:24.985961 5378 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-svzl2\" (UniqueName: \"kubernetes.io/projected/72d82ec7-ffcb-4267-9639-81357c29430d-kube-api-access-svzl2\") pod \"payload-processing-e2e-shared-b-w6-7eaf74-d47bd994b-tww2z\" (UID: \"72d82ec7-ffcb-4267-9639-81357c29430d\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w6-7eaf74-d47bd994b-tww2z" Oct 30 21:09:25.046979 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:09:25.046888 5378 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w6-7eaf74-6b776cb9f-rqvlh"] Oct 30 21:09:25.053054 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:09:25.053031 5378 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w6-7eaf74-6b776cb9f-rqvlh"] Oct 30 21:09:25.053304 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:09:25.053266 5378 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-7eaf74-6b776cb9f-rqvlh" Oct 30 21:09:25.089230 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:09:25.089203 5378 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8w6p7\" (UniqueName: \"kubernetes.io/projected/f8a30c85-5cd7-4961-ae05-e7c587f48573-kube-api-access-8w6p7\") pod \"payload-pre-processing-e2e-shared-b-w6-7eaf74-6b776cb9f-rqvlh\" (UID: \"f8a30c85-5cd7-4961-ae05-e7c587f48573\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-7eaf74-6b776cb9f-rqvlh" Oct 30 21:09:25.089359 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:09:25.089342 5378 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f8a30c85-5cd7-4961-ae05-e7c587f48573-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w6-7eaf74-6b776cb9f-rqvlh\" (UID: \"f8a30c85-5cd7-4961-ae05-e7c587f48573\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-7eaf74-6b776cb9f-rqvlh" Oct 30 21:09:25.089403 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:09:25.089394 5378 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/72d82ec7-ffcb-4267-9639-81357c29430d-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w6-7eaf74-d47bd994b-tww2z\" (UID: \"72d82ec7-ffcb-4267-9639-81357c29430d\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w6-7eaf74-d47bd994b-tww2z" Oct 30 21:09:25.089455 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:09:25.089420 5378 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-svzl2\" (UniqueName: \"kubernetes.io/projected/72d82ec7-ffcb-4267-9639-81357c29430d-kube-api-access-svzl2\") pod \"payload-processing-e2e-shared-b-w6-7eaf74-d47bd994b-tww2z\" (UID: \"72d82ec7-ffcb-4267-9639-81357c29430d\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w6-7eaf74-d47bd994b-tww2z" Oct 30 21:09:25.090028 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:09:25.090007 5378 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/72d82ec7-ffcb-4267-9639-81357c29430d-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w6-7eaf74-d47bd994b-tww2z\" (UID: \"72d82ec7-ffcb-4267-9639-81357c29430d\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w6-7eaf74-d47bd994b-tww2z" Oct 30 21:09:25.148207 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:09:25.148176 5378 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-svzl2\" (UniqueName: \"kubernetes.io/projected/72d82ec7-ffcb-4267-9639-81357c29430d-kube-api-access-svzl2\") pod \"payload-processing-e2e-shared-b-w6-7eaf74-d47bd994b-tww2z\" (UID: \"72d82ec7-ffcb-4267-9639-81357c29430d\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w6-7eaf74-d47bd994b-tww2z" Oct 30 21:09:25.190498 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:09:25.190470 5378 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f8a30c85-5cd7-4961-ae05-e7c587f48573-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w6-7eaf74-6b776cb9f-rqvlh\" (UID: \"f8a30c85-5cd7-4961-ae05-e7c587f48573\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-7eaf74-6b776cb9f-rqvlh" Oct 30 21:09:25.190688 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:09:25.190675 5378 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8w6p7\" (UniqueName: \"kubernetes.io/projected/f8a30c85-5cd7-4961-ae05-e7c587f48573-kube-api-access-8w6p7\") pod \"payload-pre-processing-e2e-shared-b-w6-7eaf74-6b776cb9f-rqvlh\" (UID: \"f8a30c85-5cd7-4961-ae05-e7c587f48573\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-7eaf74-6b776cb9f-rqvlh" Oct 30 21:09:25.191182 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:09:25.191163 5378 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f8a30c85-5cd7-4961-ae05-e7c587f48573-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w6-7eaf74-6b776cb9f-rqvlh\" (UID: \"f8a30c85-5cd7-4961-ae05-e7c587f48573\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-7eaf74-6b776cb9f-rqvlh" Oct 30 21:09:25.248511 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:09:25.248480 5378 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8w6p7\" (UniqueName: \"kubernetes.io/projected/f8a30c85-5cd7-4961-ae05-e7c587f48573-kube-api-access-8w6p7\") pod \"payload-pre-processing-e2e-shared-b-w6-7eaf74-6b776cb9f-rqvlh\" (UID: \"f8a30c85-5cd7-4961-ae05-e7c587f48573\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-7eaf74-6b776cb9f-rqvlh" Oct 30 21:09:25.279634 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:09:25.279609 5378 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w6-7eaf74-d47bd994b-tww2z" Oct 30 21:09:25.371306 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:09:25.371274 5378 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-7eaf74-6b776cb9f-rqvlh" Oct 30 21:09:25.657541 ip-10-0-138-149 kubenswrapper[5378]: W1030 21:09:25.657511 5378 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod72d82ec7_ffcb_4267_9639_81357c29430d.slice/crio-b1bb32cc48b8e972032ef99d8cb5a22de4160084e3bc1730f3aae7cdabea776e WatchSource:0}: Error finding container b1bb32cc48b8e972032ef99d8cb5a22de4160084e3bc1730f3aae7cdabea776e: Status 404 returned error can't find the container with id b1bb32cc48b8e972032ef99d8cb5a22de4160084e3bc1730f3aae7cdabea776e Oct 30 21:09:25.658170 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:09:25.658059 5378 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w6-7eaf74-d47bd994b-tww2z"] Oct 30 21:09:25.766319 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:09:25.766287 5378 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w6-7eaf74-6b776cb9f-rqvlh"] Oct 30 21:09:25.769970 ip-10-0-138-149 kubenswrapper[5378]: W1030 21:09:25.769943 5378 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf8a30c85_5cd7_4961_ae05_e7c587f48573.slice/crio-27ded346fb41fea2a6bdcec473c11c73429283df72e12f9978c7eb27c027be4d WatchSource:0}: Error finding container 27ded346fb41fea2a6bdcec473c11c73429283df72e12f9978c7eb27c027be4d: Status 404 returned error can't find the container with id 27ded346fb41fea2a6bdcec473c11c73429283df72e12f9978c7eb27c027be4d Oct 30 21:09:26.094459 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:09:26.094368 5378 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-7eaf74-6b776cb9f-rqvlh" event={"ID":"f8a30c85-5cd7-4961-ae05-e7c587f48573","Type":"ContainerStarted","Data":"442677e71a348c1221d49dedce080d924ce1b3de0ea5408090a1f681d7428439"} Oct 30 21:09:26.094459 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:09:26.094429 5378 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-7eaf74-6b776cb9f-rqvlh" event={"ID":"f8a30c85-5cd7-4961-ae05-e7c587f48573","Type":"ContainerStarted","Data":"27ded346fb41fea2a6bdcec473c11c73429283df72e12f9978c7eb27c027be4d"} Oct 30 21:09:26.094744 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:09:26.094581 5378 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-7eaf74-6b776cb9f-rqvlh" Oct 30 21:09:26.096145 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:09:26.096118 5378 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w6-7eaf74-d47bd994b-tww2z" event={"ID":"72d82ec7-ffcb-4267-9639-81357c29430d","Type":"ContainerStarted","Data":"4c3b1462b6a5d0d56598d8e9368d10be76bd32e259eaaa402b5adefe2af05e56"} Oct 30 21:09:26.096279 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:09:26.096165 5378 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w6-7eaf74-d47bd994b-tww2z" event={"ID":"72d82ec7-ffcb-4267-9639-81357c29430d","Type":"ContainerStarted","Data":"b1bb32cc48b8e972032ef99d8cb5a22de4160084e3bc1730f3aae7cdabea776e"} Oct 30 21:09:26.096279 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:09:26.096200 5378 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w6-7eaf74-d47bd994b-tww2z" Oct 30 21:09:26.211290 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:09:26.211243 5378 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-7eaf74-6b776cb9f-rqvlh" podStartSLOduration=2.211229814 podStartE2EDuration="2.211229814s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:09:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:09:26.209394572 +0000 UTC m=+1856.637508690" watchObservedRunningTime="2026-10-30 21:09:26.211229814 +0000 UTC m=+1856.639343733" Oct 30 21:09:26.346785 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:09:26.346132 5378 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-b-w6-7eaf74-d47bd994b-tww2z" podStartSLOduration=2.346112152 podStartE2EDuration="2.346112152s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:09:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:09:26.336999175 +0000 UTC m=+1856.765113096" watchObservedRunningTime="2026-10-30 21:09:26.346112152 +0000 UTC m=+1856.774226072" Oct 30 21:09:26.975721 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:09:26.975689 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:09:27.172749 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:09:27.172713 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:09:27.307307 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:09:27.307231 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:09:28.878652 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:09:28.878611 5378 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-r9g59"] Oct 30 21:09:29.056619 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:09:29.056549 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:09:32.999009 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:09:32.998968 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:09:34.124022 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:09:34.123974 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:09:34.285136 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:09:34.285101 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:09:34.461639 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:09:34.461015 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:09:34.630584 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:09:34.630529 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:09:34.785339 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:09:34.785259 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:09:34.786299 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:09:34.786278 5378 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w1-677b45-78946fddc-s5lpc"] Oct 30 21:09:35.172175 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:09:35.153802 5378 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w1-677b45-78946fddc-s5lpc"] Oct 30 21:09:35.172175 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:09:35.153954 5378 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-677b45-78946fddc-s5lpc" Oct 30 21:09:35.245592 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:09:35.245535 5378 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7ad59559-fd63-4973-9037-43c23ce150db-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w1-677b45-78946fddc-s5lpc\" (UID: \"7ad59559-fd63-4973-9037-43c23ce150db\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-677b45-78946fddc-s5lpc" Oct 30 21:09:35.245952 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:09:35.245926 5378 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b4lgs\" (UniqueName: \"kubernetes.io/projected/7ad59559-fd63-4973-9037-43c23ce150db-kube-api-access-b4lgs\") pod \"payload-pre-processing-e2e-shared-b-w1-677b45-78946fddc-s5lpc\" (UID: \"7ad59559-fd63-4973-9037-43c23ce150db\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-677b45-78946fddc-s5lpc" Oct 30 21:09:35.271340 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:09:35.271293 5378 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w1-677b45\"" Oct 30 21:09:35.293611 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:09:35.276984 5378 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w1-677b45-dockercfg-nmln7\"" Oct 30 21:09:35.370696 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:09:35.354929 5378 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b4lgs\" (UniqueName: \"kubernetes.io/projected/7ad59559-fd63-4973-9037-43c23ce150db-kube-api-access-b4lgs\") pod \"payload-pre-processing-e2e-shared-b-w1-677b45-78946fddc-s5lpc\" (UID: \"7ad59559-fd63-4973-9037-43c23ce150db\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-677b45-78946fddc-s5lpc" Oct 30 21:09:35.371355 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:09:35.371319 5378 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7ad59559-fd63-4973-9037-43c23ce150db-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w1-677b45-78946fddc-s5lpc\" (UID: \"7ad59559-fd63-4973-9037-43c23ce150db\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-677b45-78946fddc-s5lpc" Oct 30 21:09:35.372557 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:09:35.372531 5378 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7ad59559-fd63-4973-9037-43c23ce150db-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w1-677b45-78946fddc-s5lpc\" (UID: \"7ad59559-fd63-4973-9037-43c23ce150db\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-677b45-78946fddc-s5lpc" Oct 30 21:09:35.421588 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:09:35.406127 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:09:35.452709 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:09:35.452624 5378 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-f3b172-openshift-default-d64ffcd45-k78ls"] Oct 30 21:09:35.465489 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:09:35.465459 5378 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-f3b172-openshift-default-d64ffcd45-k78ls" Oct 30 21:09:35.552374 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:09:35.544172 5378 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-f3b172-openshift-default-dockercfg-lmvqr\"" Oct 30 21:09:35.572734 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:09:35.553948 5378 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b4lgs\" (UniqueName: \"kubernetes.io/projected/7ad59559-fd63-4973-9037-43c23ce150db-kube-api-access-b4lgs\") pod \"payload-pre-processing-e2e-shared-b-w1-677b45-78946fddc-s5lpc\" (UID: \"7ad59559-fd63-4973-9037-43c23ce150db\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-677b45-78946fddc-s5lpc" Oct 30 21:09:35.580296 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:09:35.580213 5378 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-f3b172-openshift-default-d64ffcd45-k78ls"] Oct 30 21:09:35.618679 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:09:35.617886 5378 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/34b71d39-551a-4d2e-bf08-8df0cf45fc9e-istiod-ca-cert\") pod \"e2e-inf-a-f3b172-openshift-default-d64ffcd45-k78ls\" (UID: \"34b71d39-551a-4d2e-bf08-8df0cf45fc9e\") " pod="openshift-ingress/e2e-inf-a-f3b172-openshift-default-d64ffcd45-k78ls" Oct 30 21:09:35.618679 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:09:35.617954 5378 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/34b71d39-551a-4d2e-bf08-8df0cf45fc9e-istio-data\") pod \"e2e-inf-a-f3b172-openshift-default-d64ffcd45-k78ls\" (UID: \"34b71d39-551a-4d2e-bf08-8df0cf45fc9e\") " pod="openshift-ingress/e2e-inf-a-f3b172-openshift-default-d64ffcd45-k78ls" Oct 30 21:09:35.618679 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:09:35.617977 5378 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/34b71d39-551a-4d2e-bf08-8df0cf45fc9e-istio-token\") pod \"e2e-inf-a-f3b172-openshift-default-d64ffcd45-k78ls\" (UID: \"34b71d39-551a-4d2e-bf08-8df0cf45fc9e\") " pod="openshift-ingress/e2e-inf-a-f3b172-openshift-default-d64ffcd45-k78ls" Oct 30 21:09:35.618679 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:09:35.618033 5378 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tkd7f\" (UniqueName: \"kubernetes.io/projected/34b71d39-551a-4d2e-bf08-8df0cf45fc9e-kube-api-access-tkd7f\") pod \"e2e-inf-a-f3b172-openshift-default-d64ffcd45-k78ls\" (UID: \"34b71d39-551a-4d2e-bf08-8df0cf45fc9e\") " pod="openshift-ingress/e2e-inf-a-f3b172-openshift-default-d64ffcd45-k78ls" Oct 30 21:09:35.618679 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:09:35.618161 5378 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/34b71d39-551a-4d2e-bf08-8df0cf45fc9e-workload-certs\") pod \"e2e-inf-a-f3b172-openshift-default-d64ffcd45-k78ls\" (UID: \"34b71d39-551a-4d2e-bf08-8df0cf45fc9e\") " pod="openshift-ingress/e2e-inf-a-f3b172-openshift-default-d64ffcd45-k78ls" Oct 30 21:09:35.618679 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:09:35.618212 5378 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/34b71d39-551a-4d2e-bf08-8df0cf45fc9e-istio-podinfo\") pod \"e2e-inf-a-f3b172-openshift-default-d64ffcd45-k78ls\" (UID: \"34b71d39-551a-4d2e-bf08-8df0cf45fc9e\") " pod="openshift-ingress/e2e-inf-a-f3b172-openshift-default-d64ffcd45-k78ls" Oct 30 21:09:35.618679 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:09:35.618237 5378 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/34b71d39-551a-4d2e-bf08-8df0cf45fc9e-istio-envoy\") pod \"e2e-inf-a-f3b172-openshift-default-d64ffcd45-k78ls\" (UID: \"34b71d39-551a-4d2e-bf08-8df0cf45fc9e\") " pod="openshift-ingress/e2e-inf-a-f3b172-openshift-default-d64ffcd45-k78ls" Oct 30 21:09:35.618679 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:09:35.618287 5378 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/34b71d39-551a-4d2e-bf08-8df0cf45fc9e-credential-socket\") pod \"e2e-inf-a-f3b172-openshift-default-d64ffcd45-k78ls\" (UID: \"34b71d39-551a-4d2e-bf08-8df0cf45fc9e\") " pod="openshift-ingress/e2e-inf-a-f3b172-openshift-default-d64ffcd45-k78ls" Oct 30 21:09:35.618679 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:09:35.618433 5378 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/34b71d39-551a-4d2e-bf08-8df0cf45fc9e-workload-socket\") pod \"e2e-inf-a-f3b172-openshift-default-d64ffcd45-k78ls\" (UID: \"34b71d39-551a-4d2e-bf08-8df0cf45fc9e\") " pod="openshift-ingress/e2e-inf-a-f3b172-openshift-default-d64ffcd45-k78ls" Oct 30 21:09:35.720214 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:09:35.720118 5378 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/34b71d39-551a-4d2e-bf08-8df0cf45fc9e-credential-socket\") pod \"e2e-inf-a-f3b172-openshift-default-d64ffcd45-k78ls\" (UID: \"34b71d39-551a-4d2e-bf08-8df0cf45fc9e\") " pod="openshift-ingress/e2e-inf-a-f3b172-openshift-default-d64ffcd45-k78ls" Oct 30 21:09:35.720394 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:09:35.720267 5378 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/34b71d39-551a-4d2e-bf08-8df0cf45fc9e-workload-socket\") pod \"e2e-inf-a-f3b172-openshift-default-d64ffcd45-k78ls\" (UID: \"34b71d39-551a-4d2e-bf08-8df0cf45fc9e\") " pod="openshift-ingress/e2e-inf-a-f3b172-openshift-default-d64ffcd45-k78ls" Oct 30 21:09:35.720394 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:09:35.720354 5378 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/34b71d39-551a-4d2e-bf08-8df0cf45fc9e-istiod-ca-cert\") pod \"e2e-inf-a-f3b172-openshift-default-d64ffcd45-k78ls\" (UID: \"34b71d39-551a-4d2e-bf08-8df0cf45fc9e\") " pod="openshift-ingress/e2e-inf-a-f3b172-openshift-default-d64ffcd45-k78ls" Oct 30 21:09:35.720513 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:09:35.720394 5378 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/34b71d39-551a-4d2e-bf08-8df0cf45fc9e-istio-data\") pod \"e2e-inf-a-f3b172-openshift-default-d64ffcd45-k78ls\" (UID: \"34b71d39-551a-4d2e-bf08-8df0cf45fc9e\") " pod="openshift-ingress/e2e-inf-a-f3b172-openshift-default-d64ffcd45-k78ls" Oct 30 21:09:35.720513 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:09:35.720419 5378 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/34b71d39-551a-4d2e-bf08-8df0cf45fc9e-istio-token\") pod \"e2e-inf-a-f3b172-openshift-default-d64ffcd45-k78ls\" (UID: \"34b71d39-551a-4d2e-bf08-8df0cf45fc9e\") " pod="openshift-ingress/e2e-inf-a-f3b172-openshift-default-d64ffcd45-k78ls" Oct 30 21:09:35.720513 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:09:35.720482 5378 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tkd7f\" (UniqueName: \"kubernetes.io/projected/34b71d39-551a-4d2e-bf08-8df0cf45fc9e-kube-api-access-tkd7f\") pod \"e2e-inf-a-f3b172-openshift-default-d64ffcd45-k78ls\" (UID: \"34b71d39-551a-4d2e-bf08-8df0cf45fc9e\") " pod="openshift-ingress/e2e-inf-a-f3b172-openshift-default-d64ffcd45-k78ls" Oct 30 21:09:35.720756 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:09:35.720629 5378 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/34b71d39-551a-4d2e-bf08-8df0cf45fc9e-workload-certs\") pod \"e2e-inf-a-f3b172-openshift-default-d64ffcd45-k78ls\" (UID: \"34b71d39-551a-4d2e-bf08-8df0cf45fc9e\") " pod="openshift-ingress/e2e-inf-a-f3b172-openshift-default-d64ffcd45-k78ls" Oct 30 21:09:35.720756 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:09:35.720695 5378 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/34b71d39-551a-4d2e-bf08-8df0cf45fc9e-istio-podinfo\") pod \"e2e-inf-a-f3b172-openshift-default-d64ffcd45-k78ls\" (UID: \"34b71d39-551a-4d2e-bf08-8df0cf45fc9e\") " pod="openshift-ingress/e2e-inf-a-f3b172-openshift-default-d64ffcd45-k78ls" Oct 30 21:09:35.720756 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:09:35.720717 5378 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/34b71d39-551a-4d2e-bf08-8df0cf45fc9e-credential-socket\") pod \"e2e-inf-a-f3b172-openshift-default-d64ffcd45-k78ls\" (UID: \"34b71d39-551a-4d2e-bf08-8df0cf45fc9e\") " pod="openshift-ingress/e2e-inf-a-f3b172-openshift-default-d64ffcd45-k78ls" Oct 30 21:09:35.720756 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:09:35.720740 5378 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/34b71d39-551a-4d2e-bf08-8df0cf45fc9e-istio-envoy\") pod \"e2e-inf-a-f3b172-openshift-default-d64ffcd45-k78ls\" (UID: \"34b71d39-551a-4d2e-bf08-8df0cf45fc9e\") " pod="openshift-ingress/e2e-inf-a-f3b172-openshift-default-d64ffcd45-k78ls" Oct 30 21:09:35.720990 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:09:35.720957 5378 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/34b71d39-551a-4d2e-bf08-8df0cf45fc9e-workload-socket\") pod \"e2e-inf-a-f3b172-openshift-default-d64ffcd45-k78ls\" (UID: \"34b71d39-551a-4d2e-bf08-8df0cf45fc9e\") " pod="openshift-ingress/e2e-inf-a-f3b172-openshift-default-d64ffcd45-k78ls" Oct 30 21:09:35.721327 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:09:35.721305 5378 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/34b71d39-551a-4d2e-bf08-8df0cf45fc9e-istio-data\") pod \"e2e-inf-a-f3b172-openshift-default-d64ffcd45-k78ls\" (UID: \"34b71d39-551a-4d2e-bf08-8df0cf45fc9e\") " pod="openshift-ingress/e2e-inf-a-f3b172-openshift-default-d64ffcd45-k78ls" Oct 30 21:09:35.721437 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:09:35.721366 5378 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/34b71d39-551a-4d2e-bf08-8df0cf45fc9e-workload-certs\") pod \"e2e-inf-a-f3b172-openshift-default-d64ffcd45-k78ls\" (UID: \"34b71d39-551a-4d2e-bf08-8df0cf45fc9e\") " pod="openshift-ingress/e2e-inf-a-f3b172-openshift-default-d64ffcd45-k78ls" Oct 30 21:09:35.721609 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:09:35.721591 5378 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/34b71d39-551a-4d2e-bf08-8df0cf45fc9e-istiod-ca-cert\") pod \"e2e-inf-a-f3b172-openshift-default-d64ffcd45-k78ls\" (UID: \"34b71d39-551a-4d2e-bf08-8df0cf45fc9e\") " pod="openshift-ingress/e2e-inf-a-f3b172-openshift-default-d64ffcd45-k78ls" Oct 30 21:09:35.723085 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:09:35.723065 5378 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/34b71d39-551a-4d2e-bf08-8df0cf45fc9e-istio-envoy\") pod \"e2e-inf-a-f3b172-openshift-default-d64ffcd45-k78ls\" (UID: \"34b71d39-551a-4d2e-bf08-8df0cf45fc9e\") " pod="openshift-ingress/e2e-inf-a-f3b172-openshift-default-d64ffcd45-k78ls" Oct 30 21:09:35.723303 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:09:35.723285 5378 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/34b71d39-551a-4d2e-bf08-8df0cf45fc9e-istio-podinfo\") pod \"e2e-inf-a-f3b172-openshift-default-d64ffcd45-k78ls\" (UID: \"34b71d39-551a-4d2e-bf08-8df0cf45fc9e\") " pod="openshift-ingress/e2e-inf-a-f3b172-openshift-default-d64ffcd45-k78ls" Oct 30 21:09:35.778961 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:09:35.778925 5378 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/34b71d39-551a-4d2e-bf08-8df0cf45fc9e-istio-token\") pod \"e2e-inf-a-f3b172-openshift-default-d64ffcd45-k78ls\" (UID: \"34b71d39-551a-4d2e-bf08-8df0cf45fc9e\") " pod="openshift-ingress/e2e-inf-a-f3b172-openshift-default-d64ffcd45-k78ls" Oct 30 21:09:35.779086 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:09:35.779036 5378 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tkd7f\" (UniqueName: \"kubernetes.io/projected/34b71d39-551a-4d2e-bf08-8df0cf45fc9e-kube-api-access-tkd7f\") pod \"e2e-inf-a-f3b172-openshift-default-d64ffcd45-k78ls\" (UID: \"34b71d39-551a-4d2e-bf08-8df0cf45fc9e\") " pod="openshift-ingress/e2e-inf-a-f3b172-openshift-default-d64ffcd45-k78ls" Oct 30 21:09:35.815537 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:09:35.815510 5378 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-677b45-78946fddc-s5lpc" Oct 30 21:09:35.858470 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:09:35.858438 5378 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-f3b172-openshift-default-d64ffcd45-k78ls" Oct 30 21:09:35.940434 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:09:35.940360 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:09:36.199670 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:09:36.199638 5378 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w1-677b45-78946fddc-s5lpc"] Oct 30 21:09:36.200257 ip-10-0-138-149 kubenswrapper[5378]: W1030 21:09:36.200233 5378 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7ad59559_fd63_4973_9037_43c23ce150db.slice/crio-8171e2e8ab4acfe77b47fe63cbc39ca332c6133a24a74ff57257b19b41229b0e WatchSource:0}: Error finding container 8171e2e8ab4acfe77b47fe63cbc39ca332c6133a24a74ff57257b19b41229b0e: Status 404 returned error can't find the container with id 8171e2e8ab4acfe77b47fe63cbc39ca332c6133a24a74ff57257b19b41229b0e Oct 30 21:09:36.270473 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:09:36.270444 5378 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-677b45-78946fddc-s5lpc" event={"ID":"7ad59559-fd63-4973-9037-43c23ce150db","Type":"ContainerStarted","Data":"8171e2e8ab4acfe77b47fe63cbc39ca332c6133a24a74ff57257b19b41229b0e"} Oct 30 21:09:36.313111 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:09:36.313084 5378 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-f3b172-openshift-default-d64ffcd45-k78ls"] Oct 30 21:09:36.313419 ip-10-0-138-149 kubenswrapper[5378]: W1030 21:09:36.313389 5378 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod34b71d39_551a_4d2e_bf08_8df0cf45fc9e.slice/crio-454ae9fd6c93d5b4243d38292fe626e2ff0e57b0d8767abbe78fcd4d73a04e4f WatchSource:0}: Error finding container 454ae9fd6c93d5b4243d38292fe626e2ff0e57b0d8767abbe78fcd4d73a04e4f: Status 404 returned error can't find the container with id 454ae9fd6c93d5b4243d38292fe626e2ff0e57b0d8767abbe78fcd4d73a04e4f Oct 30 21:09:36.316848 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:09:36.316811 5378 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 21:09:36.316942 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:09:36.316889 5378 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 21:09:36.317077 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:09:36.316939 5378 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 21:09:37.107910 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:09:37.107881 5378 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-7eaf74-6b776cb9f-rqvlh" Oct 30 21:09:37.108085 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:09:37.107947 5378 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w6-7eaf74-d47bd994b-tww2z" Oct 30 21:09:37.348099 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:09:37.348064 5378 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-f3b172-openshift-default-d64ffcd45-k78ls" event={"ID":"34b71d39-551a-4d2e-bf08-8df0cf45fc9e","Type":"ContainerStarted","Data":"62c77678f268f015b05bf2bc59bc3683ebf9d7164731bcbf27258a0c9b82483d"} Oct 30 21:09:37.348513 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:09:37.348117 5378 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-f3b172-openshift-default-d64ffcd45-k78ls" event={"ID":"34b71d39-551a-4d2e-bf08-8df0cf45fc9e","Type":"ContainerStarted","Data":"454ae9fd6c93d5b4243d38292fe626e2ff0e57b0d8767abbe78fcd4d73a04e4f"} Oct 30 21:09:37.350883 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:09:37.350852 5378 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-677b45-78946fddc-s5lpc" event={"ID":"7ad59559-fd63-4973-9037-43c23ce150db","Type":"ContainerStarted","Data":"1af25aba1bb88880623f1c6caf9a8ff2772ff41ec6acf3b30ff542fb59e119cd"} Oct 30 21:09:37.351005 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:09:37.350907 5378 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-677b45-78946fddc-s5lpc" Oct 30 21:09:37.573269 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:09:37.573166 5378 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-677b45-78946fddc-s5lpc" podStartSLOduration=3.573150678 podStartE2EDuration="3.573150678s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:09:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:09:37.567483619 +0000 UTC m=+1867.995597539" watchObservedRunningTime="2026-10-30 21:09:37.573150678 +0000 UTC m=+1868.001264597" Oct 30 21:09:37.865969 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:09:37.862616 5378 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-f3b172-openshift-default-d64ffcd45-k78ls" Oct 30 21:09:37.915070 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:09:37.894365 5378 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-f3b172-openshift-default-d64ffcd45-k78ls" Oct 30 21:09:38.023220 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:09:38.023168 5378 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-f3b172-openshift-default-d64ffcd45-k78ls" podStartSLOduration=3.023154965 podStartE2EDuration="3.023154965s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:09:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:09:37.715917832 +0000 UTC m=+1868.144031751" watchObservedRunningTime="2026-10-30 21:09:38.023154965 +0000 UTC m=+1868.451268947" Oct 30 21:09:38.357187 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:09:38.357158 5378 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-f3b172-openshift-default-d64ffcd45-k78ls" Oct 30 21:09:38.358167 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:09:38.358148 5378 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-f3b172-openshift-default-d64ffcd45-k78ls" Oct 30 21:09:39.517438 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:09:39.517402 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:09:41.037359 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:09:41.037325 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:09:41.255278 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:09:41.255246 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:09:41.388336 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:09:41.388305 5378 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-a-b888d0-openshift-default-65d785fbfc-7lvf5"] Oct 30 21:09:41.394068 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:09:41.394045 5378 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-b888d0-openshift-default-65d785fbfc-7lvf5" Oct 30 21:09:41.449899 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:09:41.449867 5378 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-a-b888d0-openshift-default-dockercfg-rnlkd\"" Oct 30 21:09:41.451640 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:09:41.451610 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:09:41.485613 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:09:41.485558 5378 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dce080ad-1859-4657-aea5-9a2749ac1866-istio-data\") pod \"e2e-api-a-b888d0-openshift-default-65d785fbfc-7lvf5\" (UID: \"dce080ad-1859-4657-aea5-9a2749ac1866\") " pod="openshift-ingress/e2e-api-a-b888d0-openshift-default-65d785fbfc-7lvf5" Oct 30 21:09:41.485791 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:09:41.485675 5378 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dce080ad-1859-4657-aea5-9a2749ac1866-credential-socket\") pod \"e2e-api-a-b888d0-openshift-default-65d785fbfc-7lvf5\" (UID: \"dce080ad-1859-4657-aea5-9a2749ac1866\") " pod="openshift-ingress/e2e-api-a-b888d0-openshift-default-65d785fbfc-7lvf5" Oct 30 21:09:41.485791 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:09:41.485732 5378 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dce080ad-1859-4657-aea5-9a2749ac1866-istio-token\") pod \"e2e-api-a-b888d0-openshift-default-65d785fbfc-7lvf5\" (UID: \"dce080ad-1859-4657-aea5-9a2749ac1866\") " pod="openshift-ingress/e2e-api-a-b888d0-openshift-default-65d785fbfc-7lvf5" Oct 30 21:09:41.485926 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:09:41.485858 5378 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dce080ad-1859-4657-aea5-9a2749ac1866-workload-socket\") pod \"e2e-api-a-b888d0-openshift-default-65d785fbfc-7lvf5\" (UID: \"dce080ad-1859-4657-aea5-9a2749ac1866\") " pod="openshift-ingress/e2e-api-a-b888d0-openshift-default-65d785fbfc-7lvf5" Oct 30 21:09:41.485926 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:09:41.485890 5378 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dce080ad-1859-4657-aea5-9a2749ac1866-workload-certs\") pod \"e2e-api-a-b888d0-openshift-default-65d785fbfc-7lvf5\" (UID: \"dce080ad-1859-4657-aea5-9a2749ac1866\") " pod="openshift-ingress/e2e-api-a-b888d0-openshift-default-65d785fbfc-7lvf5" Oct 30 21:09:41.486002 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:09:41.485923 5378 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dce080ad-1859-4657-aea5-9a2749ac1866-istiod-ca-cert\") pod \"e2e-api-a-b888d0-openshift-default-65d785fbfc-7lvf5\" (UID: \"dce080ad-1859-4657-aea5-9a2749ac1866\") " pod="openshift-ingress/e2e-api-a-b888d0-openshift-default-65d785fbfc-7lvf5" Oct 30 21:09:41.486081 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:09:41.486060 5378 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p958m\" (UniqueName: \"kubernetes.io/projected/dce080ad-1859-4657-aea5-9a2749ac1866-kube-api-access-p958m\") pod \"e2e-api-a-b888d0-openshift-default-65d785fbfc-7lvf5\" (UID: \"dce080ad-1859-4657-aea5-9a2749ac1866\") " pod="openshift-ingress/e2e-api-a-b888d0-openshift-default-65d785fbfc-7lvf5" Oct 30 21:09:41.486258 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:09:41.486242 5378 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dce080ad-1859-4657-aea5-9a2749ac1866-istio-podinfo\") pod \"e2e-api-a-b888d0-openshift-default-65d785fbfc-7lvf5\" (UID: \"dce080ad-1859-4657-aea5-9a2749ac1866\") " pod="openshift-ingress/e2e-api-a-b888d0-openshift-default-65d785fbfc-7lvf5" Oct 30 21:09:41.486352 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:09:41.486334 5378 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dce080ad-1859-4657-aea5-9a2749ac1866-istio-envoy\") pod \"e2e-api-a-b888d0-openshift-default-65d785fbfc-7lvf5\" (UID: \"dce080ad-1859-4657-aea5-9a2749ac1866\") " pod="openshift-ingress/e2e-api-a-b888d0-openshift-default-65d785fbfc-7lvf5" Oct 30 21:09:41.505320 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:09:41.505292 5378 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-b888d0-openshift-default-65d785fbfc-7lvf5"] Oct 30 21:09:41.589919 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:09:41.589881 5378 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dce080ad-1859-4657-aea5-9a2749ac1866-istio-data\") pod \"e2e-api-a-b888d0-openshift-default-65d785fbfc-7lvf5\" (UID: \"dce080ad-1859-4657-aea5-9a2749ac1866\") " pod="openshift-ingress/e2e-api-a-b888d0-openshift-default-65d785fbfc-7lvf5" Oct 30 21:09:41.590167 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:09:41.589945 5378 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dce080ad-1859-4657-aea5-9a2749ac1866-credential-socket\") pod \"e2e-api-a-b888d0-openshift-default-65d785fbfc-7lvf5\" (UID: \"dce080ad-1859-4657-aea5-9a2749ac1866\") " pod="openshift-ingress/e2e-api-a-b888d0-openshift-default-65d785fbfc-7lvf5" Oct 30 21:09:41.590167 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:09:41.589975 5378 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dce080ad-1859-4657-aea5-9a2749ac1866-istio-token\") pod \"e2e-api-a-b888d0-openshift-default-65d785fbfc-7lvf5\" (UID: \"dce080ad-1859-4657-aea5-9a2749ac1866\") " pod="openshift-ingress/e2e-api-a-b888d0-openshift-default-65d785fbfc-7lvf5" Oct 30 21:09:41.590167 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:09:41.590066 5378 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dce080ad-1859-4657-aea5-9a2749ac1866-workload-socket\") pod \"e2e-api-a-b888d0-openshift-default-65d785fbfc-7lvf5\" (UID: \"dce080ad-1859-4657-aea5-9a2749ac1866\") " pod="openshift-ingress/e2e-api-a-b888d0-openshift-default-65d785fbfc-7lvf5" Oct 30 21:09:41.590167 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:09:41.590095 5378 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dce080ad-1859-4657-aea5-9a2749ac1866-workload-certs\") pod \"e2e-api-a-b888d0-openshift-default-65d785fbfc-7lvf5\" (UID: \"dce080ad-1859-4657-aea5-9a2749ac1866\") " pod="openshift-ingress/e2e-api-a-b888d0-openshift-default-65d785fbfc-7lvf5" Oct 30 21:09:41.590167 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:09:41.590125 5378 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dce080ad-1859-4657-aea5-9a2749ac1866-istiod-ca-cert\") pod \"e2e-api-a-b888d0-openshift-default-65d785fbfc-7lvf5\" (UID: \"dce080ad-1859-4657-aea5-9a2749ac1866\") " pod="openshift-ingress/e2e-api-a-b888d0-openshift-default-65d785fbfc-7lvf5" Oct 30 21:09:41.590628 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:09:41.590225 5378 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p958m\" (UniqueName: \"kubernetes.io/projected/dce080ad-1859-4657-aea5-9a2749ac1866-kube-api-access-p958m\") pod \"e2e-api-a-b888d0-openshift-default-65d785fbfc-7lvf5\" (UID: \"dce080ad-1859-4657-aea5-9a2749ac1866\") " pod="openshift-ingress/e2e-api-a-b888d0-openshift-default-65d785fbfc-7lvf5" Oct 30 21:09:41.590628 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:09:41.590389 5378 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dce080ad-1859-4657-aea5-9a2749ac1866-istio-data\") pod \"e2e-api-a-b888d0-openshift-default-65d785fbfc-7lvf5\" (UID: \"dce080ad-1859-4657-aea5-9a2749ac1866\") " pod="openshift-ingress/e2e-api-a-b888d0-openshift-default-65d785fbfc-7lvf5" Oct 30 21:09:41.590628 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:09:41.590435 5378 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dce080ad-1859-4657-aea5-9a2749ac1866-istio-podinfo\") pod \"e2e-api-a-b888d0-openshift-default-65d785fbfc-7lvf5\" (UID: \"dce080ad-1859-4657-aea5-9a2749ac1866\") " pod="openshift-ingress/e2e-api-a-b888d0-openshift-default-65d785fbfc-7lvf5" Oct 30 21:09:41.590628 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:09:41.590527 5378 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dce080ad-1859-4657-aea5-9a2749ac1866-istio-envoy\") pod \"e2e-api-a-b888d0-openshift-default-65d785fbfc-7lvf5\" (UID: \"dce080ad-1859-4657-aea5-9a2749ac1866\") " pod="openshift-ingress/e2e-api-a-b888d0-openshift-default-65d785fbfc-7lvf5" Oct 30 21:09:41.591018 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:09:41.590725 5378 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dce080ad-1859-4657-aea5-9a2749ac1866-workload-socket\") pod \"e2e-api-a-b888d0-openshift-default-65d785fbfc-7lvf5\" (UID: \"dce080ad-1859-4657-aea5-9a2749ac1866\") " pod="openshift-ingress/e2e-api-a-b888d0-openshift-default-65d785fbfc-7lvf5" Oct 30 21:09:41.591018 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:09:41.590983 5378 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dce080ad-1859-4657-aea5-9a2749ac1866-credential-socket\") pod \"e2e-api-a-b888d0-openshift-default-65d785fbfc-7lvf5\" (UID: \"dce080ad-1859-4657-aea5-9a2749ac1866\") " pod="openshift-ingress/e2e-api-a-b888d0-openshift-default-65d785fbfc-7lvf5" Oct 30 21:09:41.591604 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:09:41.591540 5378 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dce080ad-1859-4657-aea5-9a2749ac1866-workload-certs\") pod \"e2e-api-a-b888d0-openshift-default-65d785fbfc-7lvf5\" (UID: \"dce080ad-1859-4657-aea5-9a2749ac1866\") " pod="openshift-ingress/e2e-api-a-b888d0-openshift-default-65d785fbfc-7lvf5" Oct 30 21:09:41.591720 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:09:41.591666 5378 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dce080ad-1859-4657-aea5-9a2749ac1866-istiod-ca-cert\") pod \"e2e-api-a-b888d0-openshift-default-65d785fbfc-7lvf5\" (UID: \"dce080ad-1859-4657-aea5-9a2749ac1866\") " pod="openshift-ingress/e2e-api-a-b888d0-openshift-default-65d785fbfc-7lvf5" Oct 30 21:09:41.598116 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:09:41.598091 5378 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dce080ad-1859-4657-aea5-9a2749ac1866-istio-envoy\") pod \"e2e-api-a-b888d0-openshift-default-65d785fbfc-7lvf5\" (UID: \"dce080ad-1859-4657-aea5-9a2749ac1866\") " pod="openshift-ingress/e2e-api-a-b888d0-openshift-default-65d785fbfc-7lvf5" Oct 30 21:09:41.600122 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:09:41.600095 5378 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dce080ad-1859-4657-aea5-9a2749ac1866-istio-podinfo\") pod \"e2e-api-a-b888d0-openshift-default-65d785fbfc-7lvf5\" (UID: \"dce080ad-1859-4657-aea5-9a2749ac1866\") " pod="openshift-ingress/e2e-api-a-b888d0-openshift-default-65d785fbfc-7lvf5" Oct 30 21:09:41.657221 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:09:41.657155 5378 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p958m\" (UniqueName: \"kubernetes.io/projected/dce080ad-1859-4657-aea5-9a2749ac1866-kube-api-access-p958m\") pod \"e2e-api-a-b888d0-openshift-default-65d785fbfc-7lvf5\" (UID: \"dce080ad-1859-4657-aea5-9a2749ac1866\") " pod="openshift-ingress/e2e-api-a-b888d0-openshift-default-65d785fbfc-7lvf5" Oct 30 21:09:41.658064 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:09:41.658042 5378 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dce080ad-1859-4657-aea5-9a2749ac1866-istio-token\") pod \"e2e-api-a-b888d0-openshift-default-65d785fbfc-7lvf5\" (UID: \"dce080ad-1859-4657-aea5-9a2749ac1866\") " pod="openshift-ingress/e2e-api-a-b888d0-openshift-default-65d785fbfc-7lvf5" Oct 30 21:09:41.710394 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:09:41.710363 5378 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-b888d0-openshift-default-65d785fbfc-7lvf5" Oct 30 21:09:42.162972 ip-10-0-138-149 kubenswrapper[5378]: W1030 21:09:42.162940 5378 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddce080ad_1859_4657_aea5_9a2749ac1866.slice/crio-bdb03cc247502914f9875ae64eec845a373a1d4f5d1864214d4e59dc646879eb WatchSource:0}: Error finding container bdb03cc247502914f9875ae64eec845a373a1d4f5d1864214d4e59dc646879eb: Status 404 returned error can't find the container with id bdb03cc247502914f9875ae64eec845a373a1d4f5d1864214d4e59dc646879eb Oct 30 21:09:42.163327 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:09:42.163247 5378 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-b888d0-openshift-default-65d785fbfc-7lvf5"] Oct 30 21:09:42.165890 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:09:42.165857 5378 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 21:09:42.165996 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:09:42.165923 5378 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 21:09:42.165996 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:09:42.165954 5378 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 21:09:42.389194 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:09:42.389155 5378 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-b888d0-openshift-default-65d785fbfc-7lvf5" event={"ID":"dce080ad-1859-4657-aea5-9a2749ac1866","Type":"ContainerStarted","Data":"87b4a6c4032af7c847dd052c0abd6e5eeb843593014c5658c6c85db0df22db28"} Oct 30 21:09:42.389397 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:09:42.389222 5378 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-b888d0-openshift-default-65d785fbfc-7lvf5" event={"ID":"dce080ad-1859-4657-aea5-9a2749ac1866","Type":"ContainerStarted","Data":"bdb03cc247502914f9875ae64eec845a373a1d4f5d1864214d4e59dc646879eb"} Oct 30 21:09:42.519185 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:09:42.519091 5378 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-a-b888d0-openshift-default-65d785fbfc-7lvf5" podStartSLOduration=1.519076128 podStartE2EDuration="1.519076128s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:09:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:09:42.516065818 +0000 UTC m=+1872.944179739" watchObservedRunningTime="2026-10-30 21:09:42.519076128 +0000 UTC m=+1872.947190047" Oct 30 21:09:42.710715 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:09:42.710663 5378 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-b888d0-openshift-default-65d785fbfc-7lvf5" Oct 30 21:09:43.713042 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:09:43.713006 5378 patch_prober.go:29] interesting pod/e2e-api-a-b888d0-openshift-default-65d785fbfc-7lvf5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.140:15021/healthz/ready\": dial tcp 10.133.0.140:15021: connect: connection refused" start-of-body= Oct 30 21:09:43.713433 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:09:43.713062 5378 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-b888d0-openshift-default-65d785fbfc-7lvf5" podUID="dce080ad-1859-4657-aea5-9a2749ac1866" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.140:15021/healthz/ready\": dial tcp 10.133.0.140:15021: connect: connection refused" Oct 30 21:09:44.711618 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:09:44.711581 5378 patch_prober.go:29] interesting pod/e2e-api-a-b888d0-openshift-default-65d785fbfc-7lvf5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.140:15021/healthz/ready\": dial tcp 10.133.0.140:15021: connect: connection refused" start-of-body= Oct 30 21:09:44.711775 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:09:44.711653 5378 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-b888d0-openshift-default-65d785fbfc-7lvf5" podUID="dce080ad-1859-4657-aea5-9a2749ac1866" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.140:15021/healthz/ready\": dial tcp 10.133.0.140:15021: connect: connection refused" Oct 30 21:09:45.177645 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:09:45.172682 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:09:45.711060 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:09:45.711019 5378 patch_prober.go:29] interesting pod/e2e-api-a-b888d0-openshift-default-65d785fbfc-7lvf5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.140:15021/healthz/ready\": dial tcp 10.133.0.140:15021: connect: connection refused" start-of-body= Oct 30 21:09:45.711241 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:09:45.711078 5378 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-b888d0-openshift-default-65d785fbfc-7lvf5" podUID="dce080ad-1859-4657-aea5-9a2749ac1866" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.140:15021/healthz/ready\": dial tcp 10.133.0.140:15021: connect: connection refused" Oct 30 21:09:46.678681 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:09:46.678634 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:09:46.711142 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:09:46.711114 5378 patch_prober.go:29] interesting pod/e2e-api-a-b888d0-openshift-default-65d785fbfc-7lvf5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.140:15021/healthz/ready\": dial tcp 10.133.0.140:15021: connect: connection refused" start-of-body= Oct 30 21:09:46.711288 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:09:46.711179 5378 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-b888d0-openshift-default-65d785fbfc-7lvf5" podUID="dce080ad-1859-4657-aea5-9a2749ac1866" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.140:15021/healthz/ready\": dial tcp 10.133.0.140:15021: connect: connection refused" Oct 30 21:09:46.871964 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:09:46.871893 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:09:47.187396 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:09:47.187345 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:09:47.735373 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:09:47.735317 5378 patch_prober.go:29] interesting pod/e2e-api-a-b888d0-openshift-default-65d785fbfc-7lvf5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.140:15021/healthz/ready\": dial tcp 10.133.0.140:15021: connect: connection refused" start-of-body= Oct 30 21:09:47.736120 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:09:47.736087 5378 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-b888d0-openshift-default-65d785fbfc-7lvf5" podUID="dce080ad-1859-4657-aea5-9a2749ac1866" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.140:15021/healthz/ready\": dial tcp 10.133.0.140:15021: connect: connection refused" Oct 30 21:09:48.360511 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:09:48.360479 5378 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-677b45-78946fddc-s5lpc" Oct 30 21:09:48.716191 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:09:48.716108 5378 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-b888d0-openshift-default-65d785fbfc-7lvf5" Oct 30 21:09:48.716503 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:09:48.716482 5378 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-b888d0-openshift-default-65d785fbfc-7lvf5" Oct 30 21:09:48.717038 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:09:48.717022 5378 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-b888d0-openshift-default-65d785fbfc-7lvf5" Oct 30 21:09:48.838330 ip-10-0-138-149 kubenswrapper[5378]: E1030 21:09:48.838293 5378 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q6mx7" containerName="extractor" Oct 30 21:09:50.548245 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:09:50.548211 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:09:55.379416 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:09:55.379371 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:10:00.167640 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:10:00.167601 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:10:04.950791 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:10:04.950761 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:10:09.975328 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:10:09.975238 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:10:10.352288 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:10:10.352259 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:10:10.495209 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:10:10.495166 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:10:14.868437 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:10:14.868402 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:10:18.838518 ip-10-0-138-149 kubenswrapper[5378]: E1030 21:10:18.838477 5378 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q6mx7" containerName="extractor" Oct 30 21:10:19.659874 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:10:19.659833 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:10:21.254991 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:10:21.254956 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:10:21.577671 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:10:21.577592 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:10:24.427736 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:10:24.427680 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:10:26.718239 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:10:26.718208 5378 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-a-b888d0-openshift-default-65d785fbfc-vjfvl"] Oct 30 21:10:26.723149 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:10:26.723126 5378 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-b888d0-openshift-default-65d785fbfc-vjfvl" Oct 30 21:10:26.768577 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:10:26.768542 5378 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-a-b888d0-openshift-default-65d785fbfc-m9ld8"] Oct 30 21:10:26.773385 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:10:26.773363 5378 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-b888d0-openshift-default-65d785fbfc-m9ld8" Oct 30 21:10:26.821187 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:10:26.821161 5378 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-b888d0-openshift-default-65d785fbfc-vjfvl"] Oct 30 21:10:26.849933 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:10:26.847414 5378 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/072e9c4b-6360-4cdd-8594-517158d18132-istiod-ca-cert\") pod \"e2e-api-a-b888d0-openshift-default-65d785fbfc-vjfvl\" (UID: \"072e9c4b-6360-4cdd-8594-517158d18132\") " pod="openshift-ingress/e2e-api-a-b888d0-openshift-default-65d785fbfc-vjfvl" Oct 30 21:10:26.849933 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:10:26.847452 5378 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/072e9c4b-6360-4cdd-8594-517158d18132-istio-token\") pod \"e2e-api-a-b888d0-openshift-default-65d785fbfc-vjfvl\" (UID: \"072e9c4b-6360-4cdd-8594-517158d18132\") " pod="openshift-ingress/e2e-api-a-b888d0-openshift-default-65d785fbfc-vjfvl" Oct 30 21:10:26.849933 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:10:26.847472 5378 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/072e9c4b-6360-4cdd-8594-517158d18132-workload-socket\") pod \"e2e-api-a-b888d0-openshift-default-65d785fbfc-vjfvl\" (UID: \"072e9c4b-6360-4cdd-8594-517158d18132\") " pod="openshift-ingress/e2e-api-a-b888d0-openshift-default-65d785fbfc-vjfvl" Oct 30 21:10:26.849933 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:10:26.847520 5378 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/072e9c4b-6360-4cdd-8594-517158d18132-credential-socket\") pod \"e2e-api-a-b888d0-openshift-default-65d785fbfc-vjfvl\" (UID: \"072e9c4b-6360-4cdd-8594-517158d18132\") " pod="openshift-ingress/e2e-api-a-b888d0-openshift-default-65d785fbfc-vjfvl" Oct 30 21:10:26.849933 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:10:26.847596 5378 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/072e9c4b-6360-4cdd-8594-517158d18132-workload-certs\") pod \"e2e-api-a-b888d0-openshift-default-65d785fbfc-vjfvl\" (UID: \"072e9c4b-6360-4cdd-8594-517158d18132\") " pod="openshift-ingress/e2e-api-a-b888d0-openshift-default-65d785fbfc-vjfvl" Oct 30 21:10:26.849933 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:10:26.847626 5378 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/072e9c4b-6360-4cdd-8594-517158d18132-istio-envoy\") pod \"e2e-api-a-b888d0-openshift-default-65d785fbfc-vjfvl\" (UID: \"072e9c4b-6360-4cdd-8594-517158d18132\") " pod="openshift-ingress/e2e-api-a-b888d0-openshift-default-65d785fbfc-vjfvl" Oct 30 21:10:26.849933 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:10:26.847676 5378 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/072e9c4b-6360-4cdd-8594-517158d18132-istio-data\") pod \"e2e-api-a-b888d0-openshift-default-65d785fbfc-vjfvl\" (UID: \"072e9c4b-6360-4cdd-8594-517158d18132\") " pod="openshift-ingress/e2e-api-a-b888d0-openshift-default-65d785fbfc-vjfvl" Oct 30 21:10:26.849933 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:10:26.847800 5378 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/072e9c4b-6360-4cdd-8594-517158d18132-istio-podinfo\") pod \"e2e-api-a-b888d0-openshift-default-65d785fbfc-vjfvl\" (UID: \"072e9c4b-6360-4cdd-8594-517158d18132\") " pod="openshift-ingress/e2e-api-a-b888d0-openshift-default-65d785fbfc-vjfvl" Oct 30 21:10:26.849933 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:10:26.847836 5378 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jd4hw\" (UniqueName: \"kubernetes.io/projected/072e9c4b-6360-4cdd-8594-517158d18132-kube-api-access-jd4hw\") pod \"e2e-api-a-b888d0-openshift-default-65d785fbfc-vjfvl\" (UID: \"072e9c4b-6360-4cdd-8594-517158d18132\") " pod="openshift-ingress/e2e-api-a-b888d0-openshift-default-65d785fbfc-vjfvl" Oct 30 21:10:26.877355 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:10:26.877322 5378 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-b888d0-openshift-default-65d785fbfc-m9ld8"] Oct 30 21:10:26.949219 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:10:26.949184 5378 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/072e9c4b-6360-4cdd-8594-517158d18132-istiod-ca-cert\") pod \"e2e-api-a-b888d0-openshift-default-65d785fbfc-vjfvl\" (UID: \"072e9c4b-6360-4cdd-8594-517158d18132\") " pod="openshift-ingress/e2e-api-a-b888d0-openshift-default-65d785fbfc-vjfvl" Oct 30 21:10:26.950357 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:10:26.950329 5378 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/072e9c4b-6360-4cdd-8594-517158d18132-istio-token\") pod \"e2e-api-a-b888d0-openshift-default-65d785fbfc-vjfvl\" (UID: \"072e9c4b-6360-4cdd-8594-517158d18132\") " pod="openshift-ingress/e2e-api-a-b888d0-openshift-default-65d785fbfc-vjfvl" Oct 30 21:10:26.950357 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:10:26.950226 5378 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/072e9c4b-6360-4cdd-8594-517158d18132-istiod-ca-cert\") pod \"e2e-api-a-b888d0-openshift-default-65d785fbfc-vjfvl\" (UID: \"072e9c4b-6360-4cdd-8594-517158d18132\") " pod="openshift-ingress/e2e-api-a-b888d0-openshift-default-65d785fbfc-vjfvl" Oct 30 21:10:26.950588 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:10:26.950368 5378 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/072e9c4b-6360-4cdd-8594-517158d18132-workload-socket\") pod \"e2e-api-a-b888d0-openshift-default-65d785fbfc-vjfvl\" (UID: \"072e9c4b-6360-4cdd-8594-517158d18132\") " pod="openshift-ingress/e2e-api-a-b888d0-openshift-default-65d785fbfc-vjfvl" Oct 30 21:10:26.950588 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:10:26.950459 5378 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ffcf4858-801d-4c04-be59-2c2492f56947-istiod-ca-cert\") pod \"e2e-api-a-b888d0-openshift-default-65d785fbfc-m9ld8\" (UID: \"ffcf4858-801d-4c04-be59-2c2492f56947\") " pod="openshift-ingress/e2e-api-a-b888d0-openshift-default-65d785fbfc-m9ld8" Oct 30 21:10:26.950588 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:10:26.950518 5378 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/072e9c4b-6360-4cdd-8594-517158d18132-credential-socket\") pod \"e2e-api-a-b888d0-openshift-default-65d785fbfc-vjfvl\" (UID: \"072e9c4b-6360-4cdd-8594-517158d18132\") " pod="openshift-ingress/e2e-api-a-b888d0-openshift-default-65d785fbfc-vjfvl" Oct 30 21:10:26.950760 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:10:26.950641 5378 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ffcf4858-801d-4c04-be59-2c2492f56947-istio-data\") pod \"e2e-api-a-b888d0-openshift-default-65d785fbfc-m9ld8\" (UID: \"ffcf4858-801d-4c04-be59-2c2492f56947\") " pod="openshift-ingress/e2e-api-a-b888d0-openshift-default-65d785fbfc-m9ld8" Oct 30 21:10:26.950760 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:10:26.950677 5378 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/072e9c4b-6360-4cdd-8594-517158d18132-workload-certs\") pod \"e2e-api-a-b888d0-openshift-default-65d785fbfc-vjfvl\" (UID: \"072e9c4b-6360-4cdd-8594-517158d18132\") " pod="openshift-ingress/e2e-api-a-b888d0-openshift-default-65d785fbfc-vjfvl" Oct 30 21:10:26.950760 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:10:26.950693 5378 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/072e9c4b-6360-4cdd-8594-517158d18132-workload-socket\") pod \"e2e-api-a-b888d0-openshift-default-65d785fbfc-vjfvl\" (UID: \"072e9c4b-6360-4cdd-8594-517158d18132\") " pod="openshift-ingress/e2e-api-a-b888d0-openshift-default-65d785fbfc-vjfvl" Oct 30 21:10:26.950920 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:10:26.950780 5378 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/072e9c4b-6360-4cdd-8594-517158d18132-istio-envoy\") pod \"e2e-api-a-b888d0-openshift-default-65d785fbfc-vjfvl\" (UID: \"072e9c4b-6360-4cdd-8594-517158d18132\") " pod="openshift-ingress/e2e-api-a-b888d0-openshift-default-65d785fbfc-vjfvl" Oct 30 21:10:26.950920 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:10:26.950853 5378 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ffcf4858-801d-4c04-be59-2c2492f56947-workload-socket\") pod \"e2e-api-a-b888d0-openshift-default-65d785fbfc-m9ld8\" (UID: \"ffcf4858-801d-4c04-be59-2c2492f56947\") " pod="openshift-ingress/e2e-api-a-b888d0-openshift-default-65d785fbfc-m9ld8" Oct 30 21:10:26.950920 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:10:26.950901 5378 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ffcf4858-801d-4c04-be59-2c2492f56947-istio-envoy\") pod \"e2e-api-a-b888d0-openshift-default-65d785fbfc-m9ld8\" (UID: \"ffcf4858-801d-4c04-be59-2c2492f56947\") " pod="openshift-ingress/e2e-api-a-b888d0-openshift-default-65d785fbfc-m9ld8" Oct 30 21:10:26.951076 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:10:26.950943 5378 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/072e9c4b-6360-4cdd-8594-517158d18132-istio-data\") pod \"e2e-api-a-b888d0-openshift-default-65d785fbfc-vjfvl\" (UID: \"072e9c4b-6360-4cdd-8594-517158d18132\") " pod="openshift-ingress/e2e-api-a-b888d0-openshift-default-65d785fbfc-vjfvl" Oct 30 21:10:26.951076 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:10:26.950982 5378 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ffcf4858-801d-4c04-be59-2c2492f56947-credential-socket\") pod \"e2e-api-a-b888d0-openshift-default-65d785fbfc-m9ld8\" (UID: \"ffcf4858-801d-4c04-be59-2c2492f56947\") " pod="openshift-ingress/e2e-api-a-b888d0-openshift-default-65d785fbfc-m9ld8" Oct 30 21:10:26.951076 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:10:26.951015 5378 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ffcf4858-801d-4c04-be59-2c2492f56947-workload-certs\") pod \"e2e-api-a-b888d0-openshift-default-65d785fbfc-m9ld8\" (UID: \"ffcf4858-801d-4c04-be59-2c2492f56947\") " pod="openshift-ingress/e2e-api-a-b888d0-openshift-default-65d785fbfc-m9ld8" Oct 30 21:10:26.951076 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:10:26.951044 5378 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/072e9c4b-6360-4cdd-8594-517158d18132-credential-socket\") pod \"e2e-api-a-b888d0-openshift-default-65d785fbfc-vjfvl\" (UID: \"072e9c4b-6360-4cdd-8594-517158d18132\") " pod="openshift-ingress/e2e-api-a-b888d0-openshift-default-65d785fbfc-vjfvl" Oct 30 21:10:26.952675 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:10:26.951693 5378 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/072e9c4b-6360-4cdd-8594-517158d18132-istio-data\") pod \"e2e-api-a-b888d0-openshift-default-65d785fbfc-vjfvl\" (UID: \"072e9c4b-6360-4cdd-8594-517158d18132\") " pod="openshift-ingress/e2e-api-a-b888d0-openshift-default-65d785fbfc-vjfvl" Oct 30 21:10:26.952675 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:10:26.951961 5378 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/072e9c4b-6360-4cdd-8594-517158d18132-workload-certs\") pod \"e2e-api-a-b888d0-openshift-default-65d785fbfc-vjfvl\" (UID: \"072e9c4b-6360-4cdd-8594-517158d18132\") " pod="openshift-ingress/e2e-api-a-b888d0-openshift-default-65d785fbfc-vjfvl" Oct 30 21:10:26.952675 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:10:26.952125 5378 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bn4v4\" (UniqueName: \"kubernetes.io/projected/ffcf4858-801d-4c04-be59-2c2492f56947-kube-api-access-bn4v4\") pod \"e2e-api-a-b888d0-openshift-default-65d785fbfc-m9ld8\" (UID: \"ffcf4858-801d-4c04-be59-2c2492f56947\") " pod="openshift-ingress/e2e-api-a-b888d0-openshift-default-65d785fbfc-m9ld8" Oct 30 21:10:26.952675 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:10:26.952171 5378 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ffcf4858-801d-4c04-be59-2c2492f56947-istio-podinfo\") pod \"e2e-api-a-b888d0-openshift-default-65d785fbfc-m9ld8\" (UID: \"ffcf4858-801d-4c04-be59-2c2492f56947\") " pod="openshift-ingress/e2e-api-a-b888d0-openshift-default-65d785fbfc-m9ld8" Oct 30 21:10:26.952675 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:10:26.952237 5378 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/072e9c4b-6360-4cdd-8594-517158d18132-istio-podinfo\") pod \"e2e-api-a-b888d0-openshift-default-65d785fbfc-vjfvl\" (UID: \"072e9c4b-6360-4cdd-8594-517158d18132\") " pod="openshift-ingress/e2e-api-a-b888d0-openshift-default-65d785fbfc-vjfvl" Oct 30 21:10:26.952675 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:10:26.952286 5378 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jd4hw\" (UniqueName: \"kubernetes.io/projected/072e9c4b-6360-4cdd-8594-517158d18132-kube-api-access-jd4hw\") pod \"e2e-api-a-b888d0-openshift-default-65d785fbfc-vjfvl\" (UID: \"072e9c4b-6360-4cdd-8594-517158d18132\") " pod="openshift-ingress/e2e-api-a-b888d0-openshift-default-65d785fbfc-vjfvl" Oct 30 21:10:26.952675 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:10:26.952312 5378 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ffcf4858-801d-4c04-be59-2c2492f56947-istio-token\") pod \"e2e-api-a-b888d0-openshift-default-65d785fbfc-m9ld8\" (UID: \"ffcf4858-801d-4c04-be59-2c2492f56947\") " pod="openshift-ingress/e2e-api-a-b888d0-openshift-default-65d785fbfc-m9ld8" Oct 30 21:10:26.954984 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:10:26.954961 5378 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/072e9c4b-6360-4cdd-8594-517158d18132-istio-envoy\") pod \"e2e-api-a-b888d0-openshift-default-65d785fbfc-vjfvl\" (UID: \"072e9c4b-6360-4cdd-8594-517158d18132\") " pod="openshift-ingress/e2e-api-a-b888d0-openshift-default-65d785fbfc-vjfvl" Oct 30 21:10:26.956071 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:10:26.956042 5378 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/072e9c4b-6360-4cdd-8594-517158d18132-istio-podinfo\") pod \"e2e-api-a-b888d0-openshift-default-65d785fbfc-vjfvl\" (UID: \"072e9c4b-6360-4cdd-8594-517158d18132\") " pod="openshift-ingress/e2e-api-a-b888d0-openshift-default-65d785fbfc-vjfvl" Oct 30 21:10:27.009016 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:10:27.008940 5378 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/072e9c4b-6360-4cdd-8594-517158d18132-istio-token\") pod \"e2e-api-a-b888d0-openshift-default-65d785fbfc-vjfvl\" (UID: \"072e9c4b-6360-4cdd-8594-517158d18132\") " pod="openshift-ingress/e2e-api-a-b888d0-openshift-default-65d785fbfc-vjfvl" Oct 30 21:10:27.009879 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:10:27.009852 5378 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jd4hw\" (UniqueName: \"kubernetes.io/projected/072e9c4b-6360-4cdd-8594-517158d18132-kube-api-access-jd4hw\") pod \"e2e-api-a-b888d0-openshift-default-65d785fbfc-vjfvl\" (UID: \"072e9c4b-6360-4cdd-8594-517158d18132\") " pod="openshift-ingress/e2e-api-a-b888d0-openshift-default-65d785fbfc-vjfvl" Oct 30 21:10:27.048045 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:10:27.048017 5378 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-b888d0-openshift-default-65d785fbfc-vjfvl" Oct 30 21:10:27.053976 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:10:27.053955 5378 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ffcf4858-801d-4c04-be59-2c2492f56947-istiod-ca-cert\") pod \"e2e-api-a-b888d0-openshift-default-65d785fbfc-m9ld8\" (UID: \"ffcf4858-801d-4c04-be59-2c2492f56947\") " pod="openshift-ingress/e2e-api-a-b888d0-openshift-default-65d785fbfc-m9ld8" Oct 30 21:10:27.054059 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:10:27.054025 5378 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ffcf4858-801d-4c04-be59-2c2492f56947-istio-data\") pod \"e2e-api-a-b888d0-openshift-default-65d785fbfc-m9ld8\" (UID: \"ffcf4858-801d-4c04-be59-2c2492f56947\") " pod="openshift-ingress/e2e-api-a-b888d0-openshift-default-65d785fbfc-m9ld8" Oct 30 21:10:27.054115 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:10:27.054071 5378 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ffcf4858-801d-4c04-be59-2c2492f56947-workload-socket\") pod \"e2e-api-a-b888d0-openshift-default-65d785fbfc-m9ld8\" (UID: \"ffcf4858-801d-4c04-be59-2c2492f56947\") " pod="openshift-ingress/e2e-api-a-b888d0-openshift-default-65d785fbfc-m9ld8" Oct 30 21:10:27.054115 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:10:27.054094 5378 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ffcf4858-801d-4c04-be59-2c2492f56947-istio-envoy\") pod \"e2e-api-a-b888d0-openshift-default-65d785fbfc-m9ld8\" (UID: \"ffcf4858-801d-4c04-be59-2c2492f56947\") " pod="openshift-ingress/e2e-api-a-b888d0-openshift-default-65d785fbfc-m9ld8" Oct 30 21:10:27.054218 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:10:27.054125 5378 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ffcf4858-801d-4c04-be59-2c2492f56947-credential-socket\") pod \"e2e-api-a-b888d0-openshift-default-65d785fbfc-m9ld8\" (UID: \"ffcf4858-801d-4c04-be59-2c2492f56947\") " pod="openshift-ingress/e2e-api-a-b888d0-openshift-default-65d785fbfc-m9ld8" Oct 30 21:10:27.054218 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:10:27.054150 5378 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ffcf4858-801d-4c04-be59-2c2492f56947-workload-certs\") pod \"e2e-api-a-b888d0-openshift-default-65d785fbfc-m9ld8\" (UID: \"ffcf4858-801d-4c04-be59-2c2492f56947\") " pod="openshift-ingress/e2e-api-a-b888d0-openshift-default-65d785fbfc-m9ld8" Oct 30 21:10:27.054328 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:10:27.054242 5378 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bn4v4\" (UniqueName: \"kubernetes.io/projected/ffcf4858-801d-4c04-be59-2c2492f56947-kube-api-access-bn4v4\") pod \"e2e-api-a-b888d0-openshift-default-65d785fbfc-m9ld8\" (UID: \"ffcf4858-801d-4c04-be59-2c2492f56947\") " pod="openshift-ingress/e2e-api-a-b888d0-openshift-default-65d785fbfc-m9ld8" Oct 30 21:10:27.054328 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:10:27.054277 5378 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ffcf4858-801d-4c04-be59-2c2492f56947-istio-podinfo\") pod \"e2e-api-a-b888d0-openshift-default-65d785fbfc-m9ld8\" (UID: \"ffcf4858-801d-4c04-be59-2c2492f56947\") " pod="openshift-ingress/e2e-api-a-b888d0-openshift-default-65d785fbfc-m9ld8" Oct 30 21:10:27.054435 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:10:27.054358 5378 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ffcf4858-801d-4c04-be59-2c2492f56947-istio-token\") pod \"e2e-api-a-b888d0-openshift-default-65d785fbfc-m9ld8\" (UID: \"ffcf4858-801d-4c04-be59-2c2492f56947\") " pod="openshift-ingress/e2e-api-a-b888d0-openshift-default-65d785fbfc-m9ld8" Oct 30 21:10:27.054491 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:10:27.054438 5378 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ffcf4858-801d-4c04-be59-2c2492f56947-istio-data\") pod \"e2e-api-a-b888d0-openshift-default-65d785fbfc-m9ld8\" (UID: \"ffcf4858-801d-4c04-be59-2c2492f56947\") " pod="openshift-ingress/e2e-api-a-b888d0-openshift-default-65d785fbfc-m9ld8" Oct 30 21:10:27.054590 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:10:27.054554 5378 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ffcf4858-801d-4c04-be59-2c2492f56947-workload-socket\") pod \"e2e-api-a-b888d0-openshift-default-65d785fbfc-m9ld8\" (UID: \"ffcf4858-801d-4c04-be59-2c2492f56947\") " pod="openshift-ingress/e2e-api-a-b888d0-openshift-default-65d785fbfc-m9ld8" Oct 30 21:10:27.054824 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:10:27.054802 5378 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ffcf4858-801d-4c04-be59-2c2492f56947-workload-certs\") pod \"e2e-api-a-b888d0-openshift-default-65d785fbfc-m9ld8\" (UID: \"ffcf4858-801d-4c04-be59-2c2492f56947\") " pod="openshift-ingress/e2e-api-a-b888d0-openshift-default-65d785fbfc-m9ld8" Oct 30 21:10:27.054978 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:10:27.054956 5378 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ffcf4858-801d-4c04-be59-2c2492f56947-credential-socket\") pod \"e2e-api-a-b888d0-openshift-default-65d785fbfc-m9ld8\" (UID: \"ffcf4858-801d-4c04-be59-2c2492f56947\") " pod="openshift-ingress/e2e-api-a-b888d0-openshift-default-65d785fbfc-m9ld8" Oct 30 21:10:27.055379 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:10:27.055355 5378 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ffcf4858-801d-4c04-be59-2c2492f56947-istiod-ca-cert\") pod \"e2e-api-a-b888d0-openshift-default-65d785fbfc-m9ld8\" (UID: \"ffcf4858-801d-4c04-be59-2c2492f56947\") " pod="openshift-ingress/e2e-api-a-b888d0-openshift-default-65d785fbfc-m9ld8" Oct 30 21:10:27.056773 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:10:27.056746 5378 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ffcf4858-801d-4c04-be59-2c2492f56947-istio-envoy\") pod \"e2e-api-a-b888d0-openshift-default-65d785fbfc-m9ld8\" (UID: \"ffcf4858-801d-4c04-be59-2c2492f56947\") " pod="openshift-ingress/e2e-api-a-b888d0-openshift-default-65d785fbfc-m9ld8" Oct 30 21:10:27.057071 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:10:27.057051 5378 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ffcf4858-801d-4c04-be59-2c2492f56947-istio-podinfo\") pod \"e2e-api-a-b888d0-openshift-default-65d785fbfc-m9ld8\" (UID: \"ffcf4858-801d-4c04-be59-2c2492f56947\") " pod="openshift-ingress/e2e-api-a-b888d0-openshift-default-65d785fbfc-m9ld8" Oct 30 21:10:27.112681 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:10:27.112639 5378 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bn4v4\" (UniqueName: \"kubernetes.io/projected/ffcf4858-801d-4c04-be59-2c2492f56947-kube-api-access-bn4v4\") pod \"e2e-api-a-b888d0-openshift-default-65d785fbfc-m9ld8\" (UID: \"ffcf4858-801d-4c04-be59-2c2492f56947\") " pod="openshift-ingress/e2e-api-a-b888d0-openshift-default-65d785fbfc-m9ld8" Oct 30 21:10:27.112829 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:10:27.112780 5378 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ffcf4858-801d-4c04-be59-2c2492f56947-istio-token\") pod \"e2e-api-a-b888d0-openshift-default-65d785fbfc-m9ld8\" (UID: \"ffcf4858-801d-4c04-be59-2c2492f56947\") " pod="openshift-ingress/e2e-api-a-b888d0-openshift-default-65d785fbfc-m9ld8" Oct 30 21:10:27.388063 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:10:27.388027 5378 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-b888d0-openshift-default-65d785fbfc-m9ld8" Oct 30 21:10:27.493492 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:10:27.493462 5378 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-b888d0-openshift-default-65d785fbfc-vjfvl"] Oct 30 21:10:27.493656 ip-10-0-138-149 kubenswrapper[5378]: W1030 21:10:27.493511 5378 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod072e9c4b_6360_4cdd_8594_517158d18132.slice/crio-9f764b2dc467c8542cfeef6131960aea2a7730887c7d89458185c4becde22aaa WatchSource:0}: Error finding container 9f764b2dc467c8542cfeef6131960aea2a7730887c7d89458185c4becde22aaa: Status 404 returned error can't find the container with id 9f764b2dc467c8542cfeef6131960aea2a7730887c7d89458185c4becde22aaa Oct 30 21:10:27.497639 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:10:27.497599 5378 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 21:10:27.497747 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:10:27.497686 5378 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 21:10:27.497747 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:10:27.497733 5378 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 21:10:27.773191 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:10:27.773162 5378 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-b888d0-openshift-default-65d785fbfc-m9ld8"] Oct 30 21:10:27.773888 ip-10-0-138-149 kubenswrapper[5378]: W1030 21:10:27.773859 5378 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podffcf4858_801d_4c04_be59_2c2492f56947.slice/crio-686ef4e51b8c165372eff5d0336caa439034e1be38ed795ecfbc00b3fa4cb974 WatchSource:0}: Error finding container 686ef4e51b8c165372eff5d0336caa439034e1be38ed795ecfbc00b3fa4cb974: Status 404 returned error can't find the container with id 686ef4e51b8c165372eff5d0336caa439034e1be38ed795ecfbc00b3fa4cb974 Oct 30 21:10:27.777059 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:10:27.777019 5378 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 21:10:27.777138 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:10:27.777114 5378 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 21:10:27.777186 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:10:27.777160 5378 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 21:10:27.888969 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:10:27.888929 5378 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-b888d0-openshift-default-65d785fbfc-m9ld8" event={"ID":"ffcf4858-801d-4c04-be59-2c2492f56947","Type":"ContainerStarted","Data":"bedb5cdbeaa14b70dc1b81bc8a66c6dc91881456249eb23ac3104f92190d47ee"} Oct 30 21:10:27.889152 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:10:27.888994 5378 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-b888d0-openshift-default-65d785fbfc-m9ld8" event={"ID":"ffcf4858-801d-4c04-be59-2c2492f56947","Type":"ContainerStarted","Data":"686ef4e51b8c165372eff5d0336caa439034e1be38ed795ecfbc00b3fa4cb974"} Oct 30 21:10:27.891138 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:10:27.891108 5378 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-b888d0-openshift-default-65d785fbfc-vjfvl" event={"ID":"072e9c4b-6360-4cdd-8594-517158d18132","Type":"ContainerStarted","Data":"bd617042ca38094008c7ae1180bcd1226f139d12c879f98fe53eede7f73a6f85"} Oct 30 21:10:27.891285 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:10:27.891158 5378 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-b888d0-openshift-default-65d785fbfc-vjfvl" event={"ID":"072e9c4b-6360-4cdd-8594-517158d18132","Type":"ContainerStarted","Data":"9f764b2dc467c8542cfeef6131960aea2a7730887c7d89458185c4becde22aaa"} Oct 30 21:10:28.012028 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:10:28.011973 5378 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-a-b888d0-openshift-default-65d785fbfc-m9ld8" podStartSLOduration=2.011958536 podStartE2EDuration="2.011958536s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:10:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:10:28.008557179 +0000 UTC m=+1918.436671091" watchObservedRunningTime="2026-10-30 21:10:28.011958536 +0000 UTC m=+1918.440072454" Oct 30 21:10:28.048864 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:10:28.048776 5378 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-b888d0-openshift-default-65d785fbfc-vjfvl" Oct 30 21:10:28.050870 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:10:28.050839 5378 patch_prober.go:29] interesting pod/e2e-api-a-b888d0-openshift-default-65d785fbfc-vjfvl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.141:15021/healthz/ready\": dial tcp 10.133.0.141:15021: connect: connection refused" start-of-body= Oct 30 21:10:28.051039 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:10:28.050897 5378 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-b888d0-openshift-default-65d785fbfc-vjfvl" podUID="072e9c4b-6360-4cdd-8594-517158d18132" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.141:15021/healthz/ready\": dial tcp 10.133.0.141:15021: connect: connection refused" Oct 30 21:10:28.128012 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:10:28.127957 5378 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-a-b888d0-openshift-default-65d785fbfc-vjfvl" podStartSLOduration=2.127942216 podStartE2EDuration="2.127942216s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:10:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:10:28.127176041 +0000 UTC m=+1918.555289961" watchObservedRunningTime="2026-10-30 21:10:28.127942216 +0000 UTC m=+1918.556056135" Oct 30 21:10:28.388674 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:10:28.388631 5378 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-b888d0-openshift-default-65d785fbfc-m9ld8" Oct 30 21:10:28.390534 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:10:28.390492 5378 patch_prober.go:29] interesting pod/e2e-api-a-b888d0-openshift-default-65d785fbfc-m9ld8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.142:15021/healthz/ready\": dial tcp 10.133.0.142:15021: connect: connection refused" start-of-body= Oct 30 21:10:28.390688 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:10:28.390544 5378 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-b888d0-openshift-default-65d785fbfc-m9ld8" podUID="ffcf4858-801d-4c04-be59-2c2492f56947" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.142:15021/healthz/ready\": dial tcp 10.133.0.142:15021: connect: connection refused" Oct 30 21:10:28.631590 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:10:28.629295 5378 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-r9g59"] Oct 30 21:10:29.051117 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:10:29.048945 5378 patch_prober.go:29] interesting pod/e2e-api-a-b888d0-openshift-default-65d785fbfc-vjfvl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.141:15021/healthz/ready\": dial tcp 10.133.0.141:15021: connect: connection refused" start-of-body= Oct 30 21:10:29.051117 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:10:29.049003 5378 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-b888d0-openshift-default-65d785fbfc-vjfvl" podUID="072e9c4b-6360-4cdd-8594-517158d18132" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.141:15021/healthz/ready\": dial tcp 10.133.0.141:15021: connect: connection refused" Oct 30 21:10:29.394590 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:10:29.393979 5378 patch_prober.go:29] interesting pod/e2e-api-a-b888d0-openshift-default-65d785fbfc-m9ld8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.142:15021/healthz/ready\": dial tcp 10.133.0.142:15021: connect: connection refused" start-of-body= Oct 30 21:10:29.394590 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:10:29.394040 5378 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-b888d0-openshift-default-65d785fbfc-m9ld8" podUID="ffcf4858-801d-4c04-be59-2c2492f56947" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.142:15021/healthz/ready\": dial tcp 10.133.0.142:15021: connect: connection refused" Oct 30 21:10:29.697633 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:10:29.697538 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:10:30.048636 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:10:30.048525 5378 patch_prober.go:29] interesting pod/e2e-api-a-b888d0-openshift-default-65d785fbfc-vjfvl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.141:15021/healthz/ready\": dial tcp 10.133.0.141:15021: connect: connection refused" start-of-body= Oct 30 21:10:30.048798 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:10:30.048636 5378 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-b888d0-openshift-default-65d785fbfc-vjfvl" podUID="072e9c4b-6360-4cdd-8594-517158d18132" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.141:15021/healthz/ready\": dial tcp 10.133.0.141:15021: connect: connection refused" Oct 30 21:10:30.399589 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:10:30.391682 5378 patch_prober.go:29] interesting pod/e2e-api-a-b888d0-openshift-default-65d785fbfc-m9ld8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.142:15021/healthz/ready\": dial tcp 10.133.0.142:15021: connect: connection refused" start-of-body= Oct 30 21:10:30.399589 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:10:30.391743 5378 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-b888d0-openshift-default-65d785fbfc-m9ld8" podUID="ffcf4858-801d-4c04-be59-2c2492f56947" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.142:15021/healthz/ready\": dial tcp 10.133.0.142:15021: connect: connection refused" Oct 30 21:10:31.048479 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:10:31.048444 5378 patch_prober.go:29] interesting pod/e2e-api-a-b888d0-openshift-default-65d785fbfc-vjfvl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.141:15021/healthz/ready\": dial tcp 10.133.0.141:15021: connect: connection refused" start-of-body= Oct 30 21:10:31.048689 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:10:31.048503 5378 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-b888d0-openshift-default-65d785fbfc-vjfvl" podUID="072e9c4b-6360-4cdd-8594-517158d18132" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.141:15021/healthz/ready\": dial tcp 10.133.0.141:15021: connect: connection refused" Oct 30 21:10:31.392589 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:10:31.391953 5378 patch_prober.go:29] interesting pod/e2e-api-a-b888d0-openshift-default-65d785fbfc-m9ld8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.142:15021/healthz/ready\": dial tcp 10.133.0.142:15021: connect: connection refused" start-of-body= Oct 30 21:10:31.392589 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:10:31.392015 5378 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-b888d0-openshift-default-65d785fbfc-m9ld8" podUID="ffcf4858-801d-4c04-be59-2c2492f56947" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.142:15021/healthz/ready\": dial tcp 10.133.0.142:15021: connect: connection refused" Oct 30 21:10:32.048910 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:10:32.048870 5378 patch_prober.go:29] interesting pod/e2e-api-a-b888d0-openshift-default-65d785fbfc-vjfvl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.141:15021/healthz/ready\": dial tcp 10.133.0.141:15021: connect: connection refused" start-of-body= Oct 30 21:10:32.049309 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:10:32.048947 5378 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-b888d0-openshift-default-65d785fbfc-vjfvl" podUID="072e9c4b-6360-4cdd-8594-517158d18132" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.141:15021/healthz/ready\": dial tcp 10.133.0.141:15021: connect: connection refused" Oct 30 21:10:32.390590 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:10:32.389894 5378 patch_prober.go:29] interesting pod/e2e-api-a-b888d0-openshift-default-65d785fbfc-m9ld8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.142:15021/healthz/ready\": dial tcp 10.133.0.142:15021: connect: connection refused" start-of-body= Oct 30 21:10:32.390590 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:10:32.389956 5378 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-b888d0-openshift-default-65d785fbfc-m9ld8" podUID="ffcf4858-801d-4c04-be59-2c2492f56947" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.142:15021/healthz/ready\": dial tcp 10.133.0.142:15021: connect: connection refused" Oct 30 21:10:33.049012 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:10:33.048970 5378 patch_prober.go:29] interesting pod/e2e-api-a-b888d0-openshift-default-65d785fbfc-vjfvl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.141:15021/healthz/ready\": dial tcp 10.133.0.141:15021: connect: connection refused" start-of-body= Oct 30 21:10:33.049423 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:10:33.049041 5378 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-b888d0-openshift-default-65d785fbfc-vjfvl" podUID="072e9c4b-6360-4cdd-8594-517158d18132" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.141:15021/healthz/ready\": dial tcp 10.133.0.141:15021: connect: connection refused" Oct 30 21:10:33.392590 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:10:33.391933 5378 patch_prober.go:29] interesting pod/e2e-api-a-b888d0-openshift-default-65d785fbfc-m9ld8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.142:15021/healthz/ready\": dial tcp 10.133.0.142:15021: connect: connection refused" start-of-body= Oct 30 21:10:33.392590 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:10:33.392008 5378 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-b888d0-openshift-default-65d785fbfc-m9ld8" podUID="ffcf4858-801d-4c04-be59-2c2492f56947" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.142:15021/healthz/ready\": dial tcp 10.133.0.142:15021: connect: connection refused" Oct 30 21:10:34.022721 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:10:34.022684 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:10:34.060585 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:10:34.057739 5378 patch_prober.go:29] interesting pod/e2e-api-a-b888d0-openshift-default-65d785fbfc-vjfvl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.141:15021/healthz/ready\": dial tcp 10.133.0.141:15021: connect: connection refused" start-of-body= Oct 30 21:10:34.060585 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:10:34.057819 5378 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-b888d0-openshift-default-65d785fbfc-vjfvl" podUID="072e9c4b-6360-4cdd-8594-517158d18132" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.141:15021/healthz/ready\": dial tcp 10.133.0.141:15021: connect: connection refused" Oct 30 21:10:34.599901 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:10:34.599854 5378 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-b888d0-openshift-default-65d785fbfc-m9ld8" Oct 30 21:10:34.600440 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:10:34.600410 5378 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-b888d0-openshift-default-65d785fbfc-m9ld8" Oct 30 21:10:34.601339 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:10:34.601316 5378 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-b888d0-openshift-default-65d785fbfc-m9ld8" Oct 30 21:10:34.826765 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:10:34.826722 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:10:35.055617 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:10:35.055519 5378 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-b888d0-openshift-default-65d785fbfc-vjfvl" Oct 30 21:10:35.057118 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:10:35.057089 5378 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-b888d0-openshift-default-65d785fbfc-vjfvl" Oct 30 21:10:35.062938 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:10:35.062911 5378 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-b888d0-openshift-default-65d785fbfc-vjfvl" Oct 30 21:10:35.161591 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:10:35.161542 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:10:35.344221 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:10:35.341576 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:10:35.530221 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:10:35.521431 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:10:35.883782 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:10:35.883747 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:10:36.197048 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:10:36.195634 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:10:40.042701 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:10:40.042662 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:10:40.245522 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:10:40.245491 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:10:40.532974 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:10:40.532940 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:10:40.765766 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:10:40.765724 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:10:41.000642 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:10:41.000605 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:10:43.066000 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:10:43.065957 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:10:46.276606 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:10:46.276552 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:10:46.404439 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:10:46.404406 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:10:46.561673 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:10:46.560708 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:10:47.085696 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:10:47.064598 5378 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-cb6404-5f8b4fc56b-twpl5"] Oct 30 21:10:47.085696 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:10:47.076797 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:10:47.085696 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:10:47.078330 5378 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-cb6404-5f8b4fc56b-twpl5" Oct 30 21:10:47.141606 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:10:47.113349 5378 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-cb6404-5f8b4fc56b-twpl5"] Oct 30 21:10:47.207597 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:10:47.203865 5378 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-b-cb6404-dockercfg-xhgrz\"" Oct 30 21:10:47.214822 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:10:47.204065 5378 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-b-cb6404\"" Oct 30 21:10:47.277648 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:10:47.277612 5378 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rdqh9\" (UniqueName: \"kubernetes.io/projected/3f616067-e750-4f5c-8fd4-d5a9cf5cf9ec-kube-api-access-rdqh9\") pod \"payload-processing-e2e-api-b-cb6404-5f8b4fc56b-twpl5\" (UID: \"3f616067-e750-4f5c-8fd4-d5a9cf5cf9ec\") " pod="openshift-ingress/payload-processing-e2e-api-b-cb6404-5f8b4fc56b-twpl5" Oct 30 21:10:47.277648 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:10:47.277656 5378 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3f616067-e750-4f5c-8fd4-d5a9cf5cf9ec-plugins-config-volume\") pod \"payload-processing-e2e-api-b-cb6404-5f8b4fc56b-twpl5\" (UID: \"3f616067-e750-4f5c-8fd4-d5a9cf5cf9ec\") " pod="openshift-ingress/payload-processing-e2e-api-b-cb6404-5f8b4fc56b-twpl5" Oct 30 21:10:47.379066 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:10:47.379039 5378 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rdqh9\" (UniqueName: \"kubernetes.io/projected/3f616067-e750-4f5c-8fd4-d5a9cf5cf9ec-kube-api-access-rdqh9\") pod \"payload-processing-e2e-api-b-cb6404-5f8b4fc56b-twpl5\" (UID: \"3f616067-e750-4f5c-8fd4-d5a9cf5cf9ec\") " pod="openshift-ingress/payload-processing-e2e-api-b-cb6404-5f8b4fc56b-twpl5" Oct 30 21:10:47.379256 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:10:47.379073 5378 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3f616067-e750-4f5c-8fd4-d5a9cf5cf9ec-plugins-config-volume\") pod \"payload-processing-e2e-api-b-cb6404-5f8b4fc56b-twpl5\" (UID: \"3f616067-e750-4f5c-8fd4-d5a9cf5cf9ec\") " pod="openshift-ingress/payload-processing-e2e-api-b-cb6404-5f8b4fc56b-twpl5" Oct 30 21:10:47.379859 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:10:47.379838 5378 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3f616067-e750-4f5c-8fd4-d5a9cf5cf9ec-plugins-config-volume\") pod \"payload-processing-e2e-api-b-cb6404-5f8b4fc56b-twpl5\" (UID: \"3f616067-e750-4f5c-8fd4-d5a9cf5cf9ec\") " pod="openshift-ingress/payload-processing-e2e-api-b-cb6404-5f8b4fc56b-twpl5" Oct 30 21:10:47.436553 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:10:47.436516 5378 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rdqh9\" (UniqueName: \"kubernetes.io/projected/3f616067-e750-4f5c-8fd4-d5a9cf5cf9ec-kube-api-access-rdqh9\") pod \"payload-processing-e2e-api-b-cb6404-5f8b4fc56b-twpl5\" (UID: \"3f616067-e750-4f5c-8fd4-d5a9cf5cf9ec\") " pod="openshift-ingress/payload-processing-e2e-api-b-cb6404-5f8b4fc56b-twpl5" Oct 30 21:10:47.711271 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:10:47.711196 5378 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-cb6404-5f8b4fc56b-twpl5" Oct 30 21:10:48.099619 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:10:48.099587 5378 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-cb6404-5f8b4fc56b-twpl5"] Oct 30 21:10:48.100073 ip-10-0-138-149 kubenswrapper[5378]: W1030 21:10:48.100050 5378 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3f616067_e750_4f5c_8fd4_d5a9cf5cf9ec.slice/crio-1966d90e947d46dbf86e4762183c5f682ecccbc99e621d0ee0f27421b4bb89c0 WatchSource:0}: Error finding container 1966d90e947d46dbf86e4762183c5f682ecccbc99e621d0ee0f27421b4bb89c0: Status 404 returned error can't find the container with id 1966d90e947d46dbf86e4762183c5f682ecccbc99e621d0ee0f27421b4bb89c0 Oct 30 21:10:48.215654 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:10:48.215607 5378 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-cb6404-5f8b4fc56b-twpl5" event={"ID":"3f616067-e750-4f5c-8fd4-d5a9cf5cf9ec","Type":"ContainerStarted","Data":"b34c7e2ac1472859f0876d7263bbf4bd775388913a492eaeda5fb814a4dfc1a5"} Oct 30 21:10:48.215791 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:10:48.215669 5378 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-cb6404-5f8b4fc56b-twpl5" event={"ID":"3f616067-e750-4f5c-8fd4-d5a9cf5cf9ec","Type":"ContainerStarted","Data":"1966d90e947d46dbf86e4762183c5f682ecccbc99e621d0ee0f27421b4bb89c0"} Oct 30 21:10:48.215791 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:10:48.215716 5378 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-b-cb6404-5f8b4fc56b-twpl5" Oct 30 21:10:48.333030 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:10:48.332968 5378 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-b-cb6404-5f8b4fc56b-twpl5" podStartSLOduration=2.3329498859999998 podStartE2EDuration="2.332949886s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:10:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:10:48.33034677 +0000 UTC m=+1938.758460690" watchObservedRunningTime="2026-10-30 21:10:48.332949886 +0000 UTC m=+1938.761063807" Oct 30 21:10:48.838816 ip-10-0-138-149 kubenswrapper[5378]: E1030 21:10:48.838775 5378 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q6mx7" containerName="extractor" Oct 30 21:10:49.377573 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:10:49.377541 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:10:51.176925 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:10:51.176889 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:10:55.993910 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:10:55.993880 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:10:59.227522 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:10:59.227492 5378 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-b-cb6404-5f8b4fc56b-twpl5" Oct 30 21:11:01.135118 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:11:01.135084 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:11:01.386643 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:11:01.373465 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:11:01.572700 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:11:01.572666 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:11:06.604832 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:11:06.604788 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:11:09.001373 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:11:09.001331 5378 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w6-7eaf74-openshift-default-69c8f48765-phhsn"] Oct 30 21:11:09.025436 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:11:09.025412 5378 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w6-7eaf74-openshift-default-69c8f48765-phhsn" Oct 30 21:11:09.133692 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:11:09.130620 5378 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4521c8f6-928b-47a2-8457-de63ee943c09-istio-envoy\") pod \"e2e-shared-b-w6-7eaf74-openshift-default-69c8f48765-phhsn\" (UID: \"4521c8f6-928b-47a2-8457-de63ee943c09\") " pod="openshift-ingress/e2e-shared-b-w6-7eaf74-openshift-default-69c8f48765-phhsn" Oct 30 21:11:09.133692 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:11:09.130751 5378 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4521c8f6-928b-47a2-8457-de63ee943c09-istiod-ca-cert\") pod \"e2e-shared-b-w6-7eaf74-openshift-default-69c8f48765-phhsn\" (UID: \"4521c8f6-928b-47a2-8457-de63ee943c09\") " pod="openshift-ingress/e2e-shared-b-w6-7eaf74-openshift-default-69c8f48765-phhsn" Oct 30 21:11:09.133692 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:11:09.130796 5378 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4521c8f6-928b-47a2-8457-de63ee943c09-istio-data\") pod \"e2e-shared-b-w6-7eaf74-openshift-default-69c8f48765-phhsn\" (UID: \"4521c8f6-928b-47a2-8457-de63ee943c09\") " pod="openshift-ingress/e2e-shared-b-w6-7eaf74-openshift-default-69c8f48765-phhsn" Oct 30 21:11:09.133692 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:11:09.130866 5378 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4521c8f6-928b-47a2-8457-de63ee943c09-credential-socket\") pod \"e2e-shared-b-w6-7eaf74-openshift-default-69c8f48765-phhsn\" (UID: \"4521c8f6-928b-47a2-8457-de63ee943c09\") " pod="openshift-ingress/e2e-shared-b-w6-7eaf74-openshift-default-69c8f48765-phhsn" Oct 30 21:11:09.133692 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:11:09.130943 5378 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4521c8f6-928b-47a2-8457-de63ee943c09-istio-token\") pod \"e2e-shared-b-w6-7eaf74-openshift-default-69c8f48765-phhsn\" (UID: \"4521c8f6-928b-47a2-8457-de63ee943c09\") " pod="openshift-ingress/e2e-shared-b-w6-7eaf74-openshift-default-69c8f48765-phhsn" Oct 30 21:11:09.133692 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:11:09.130992 5378 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4521c8f6-928b-47a2-8457-de63ee943c09-istio-podinfo\") pod \"e2e-shared-b-w6-7eaf74-openshift-default-69c8f48765-phhsn\" (UID: \"4521c8f6-928b-47a2-8457-de63ee943c09\") " pod="openshift-ingress/e2e-shared-b-w6-7eaf74-openshift-default-69c8f48765-phhsn" Oct 30 21:11:09.133692 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:11:09.131055 5378 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4521c8f6-928b-47a2-8457-de63ee943c09-workload-certs\") pod \"e2e-shared-b-w6-7eaf74-openshift-default-69c8f48765-phhsn\" (UID: \"4521c8f6-928b-47a2-8457-de63ee943c09\") " pod="openshift-ingress/e2e-shared-b-w6-7eaf74-openshift-default-69c8f48765-phhsn" Oct 30 21:11:09.133692 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:11:09.131088 5378 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vwffv\" (UniqueName: \"kubernetes.io/projected/4521c8f6-928b-47a2-8457-de63ee943c09-kube-api-access-vwffv\") pod \"e2e-shared-b-w6-7eaf74-openshift-default-69c8f48765-phhsn\" (UID: \"4521c8f6-928b-47a2-8457-de63ee943c09\") " pod="openshift-ingress/e2e-shared-b-w6-7eaf74-openshift-default-69c8f48765-phhsn" Oct 30 21:11:09.133692 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:11:09.131157 5378 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4521c8f6-928b-47a2-8457-de63ee943c09-workload-socket\") pod \"e2e-shared-b-w6-7eaf74-openshift-default-69c8f48765-phhsn\" (UID: \"4521c8f6-928b-47a2-8457-de63ee943c09\") " pod="openshift-ingress/e2e-shared-b-w6-7eaf74-openshift-default-69c8f48765-phhsn" Oct 30 21:11:09.140754 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:11:09.139273 5378 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w6-7eaf74-openshift-default-69c8f48765-phhsn"] Oct 30 21:11:09.233465 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:11:09.233436 5378 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4521c8f6-928b-47a2-8457-de63ee943c09-workload-certs\") pod \"e2e-shared-b-w6-7eaf74-openshift-default-69c8f48765-phhsn\" (UID: \"4521c8f6-928b-47a2-8457-de63ee943c09\") " pod="openshift-ingress/e2e-shared-b-w6-7eaf74-openshift-default-69c8f48765-phhsn" Oct 30 21:11:09.233668 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:11:09.233491 5378 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vwffv\" (UniqueName: \"kubernetes.io/projected/4521c8f6-928b-47a2-8457-de63ee943c09-kube-api-access-vwffv\") pod \"e2e-shared-b-w6-7eaf74-openshift-default-69c8f48765-phhsn\" (UID: \"4521c8f6-928b-47a2-8457-de63ee943c09\") " pod="openshift-ingress/e2e-shared-b-w6-7eaf74-openshift-default-69c8f48765-phhsn" Oct 30 21:11:09.233668 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:11:09.233593 5378 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4521c8f6-928b-47a2-8457-de63ee943c09-workload-socket\") pod \"e2e-shared-b-w6-7eaf74-openshift-default-69c8f48765-phhsn\" (UID: \"4521c8f6-928b-47a2-8457-de63ee943c09\") " pod="openshift-ingress/e2e-shared-b-w6-7eaf74-openshift-default-69c8f48765-phhsn" Oct 30 21:11:09.233783 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:11:09.233765 5378 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4521c8f6-928b-47a2-8457-de63ee943c09-istio-envoy\") pod \"e2e-shared-b-w6-7eaf74-openshift-default-69c8f48765-phhsn\" (UID: \"4521c8f6-928b-47a2-8457-de63ee943c09\") " pod="openshift-ingress/e2e-shared-b-w6-7eaf74-openshift-default-69c8f48765-phhsn" Oct 30 21:11:09.233897 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:11:09.233882 5378 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4521c8f6-928b-47a2-8457-de63ee943c09-istiod-ca-cert\") pod \"e2e-shared-b-w6-7eaf74-openshift-default-69c8f48765-phhsn\" (UID: \"4521c8f6-928b-47a2-8457-de63ee943c09\") " pod="openshift-ingress/e2e-shared-b-w6-7eaf74-openshift-default-69c8f48765-phhsn" Oct 30 21:11:09.233939 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:11:09.233922 5378 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4521c8f6-928b-47a2-8457-de63ee943c09-istio-data\") pod \"e2e-shared-b-w6-7eaf74-openshift-default-69c8f48765-phhsn\" (UID: \"4521c8f6-928b-47a2-8457-de63ee943c09\") " pod="openshift-ingress/e2e-shared-b-w6-7eaf74-openshift-default-69c8f48765-phhsn" Oct 30 21:11:09.234007 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:11:09.233993 5378 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4521c8f6-928b-47a2-8457-de63ee943c09-credential-socket\") pod \"e2e-shared-b-w6-7eaf74-openshift-default-69c8f48765-phhsn\" (UID: \"4521c8f6-928b-47a2-8457-de63ee943c09\") " pod="openshift-ingress/e2e-shared-b-w6-7eaf74-openshift-default-69c8f48765-phhsn" Oct 30 21:11:09.234041 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:11:09.234022 5378 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4521c8f6-928b-47a2-8457-de63ee943c09-workload-socket\") pod \"e2e-shared-b-w6-7eaf74-openshift-default-69c8f48765-phhsn\" (UID: \"4521c8f6-928b-47a2-8457-de63ee943c09\") " pod="openshift-ingress/e2e-shared-b-w6-7eaf74-openshift-default-69c8f48765-phhsn" Oct 30 21:11:09.234113 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:11:09.234097 5378 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4521c8f6-928b-47a2-8457-de63ee943c09-istio-token\") pod \"e2e-shared-b-w6-7eaf74-openshift-default-69c8f48765-phhsn\" (UID: \"4521c8f6-928b-47a2-8457-de63ee943c09\") " pod="openshift-ingress/e2e-shared-b-w6-7eaf74-openshift-default-69c8f48765-phhsn" Oct 30 21:11:09.234171 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:11:09.234157 5378 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4521c8f6-928b-47a2-8457-de63ee943c09-istio-podinfo\") pod \"e2e-shared-b-w6-7eaf74-openshift-default-69c8f48765-phhsn\" (UID: \"4521c8f6-928b-47a2-8457-de63ee943c09\") " pod="openshift-ingress/e2e-shared-b-w6-7eaf74-openshift-default-69c8f48765-phhsn" Oct 30 21:11:09.234367 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:11:09.234342 5378 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4521c8f6-928b-47a2-8457-de63ee943c09-credential-socket\") pod \"e2e-shared-b-w6-7eaf74-openshift-default-69c8f48765-phhsn\" (UID: \"4521c8f6-928b-47a2-8457-de63ee943c09\") " pod="openshift-ingress/e2e-shared-b-w6-7eaf74-openshift-default-69c8f48765-phhsn" Oct 30 21:11:09.234943 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:11:09.234921 5378 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4521c8f6-928b-47a2-8457-de63ee943c09-istio-data\") pod \"e2e-shared-b-w6-7eaf74-openshift-default-69c8f48765-phhsn\" (UID: \"4521c8f6-928b-47a2-8457-de63ee943c09\") " pod="openshift-ingress/e2e-shared-b-w6-7eaf74-openshift-default-69c8f48765-phhsn" Oct 30 21:11:09.235439 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:11:09.235307 5378 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4521c8f6-928b-47a2-8457-de63ee943c09-istiod-ca-cert\") pod \"e2e-shared-b-w6-7eaf74-openshift-default-69c8f48765-phhsn\" (UID: \"4521c8f6-928b-47a2-8457-de63ee943c09\") " pod="openshift-ingress/e2e-shared-b-w6-7eaf74-openshift-default-69c8f48765-phhsn" Oct 30 21:11:09.236032 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:11:09.236008 5378 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4521c8f6-928b-47a2-8457-de63ee943c09-workload-certs\") pod \"e2e-shared-b-w6-7eaf74-openshift-default-69c8f48765-phhsn\" (UID: \"4521c8f6-928b-47a2-8457-de63ee943c09\") " pod="openshift-ingress/e2e-shared-b-w6-7eaf74-openshift-default-69c8f48765-phhsn" Oct 30 21:11:09.237043 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:11:09.237019 5378 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4521c8f6-928b-47a2-8457-de63ee943c09-istio-envoy\") pod \"e2e-shared-b-w6-7eaf74-openshift-default-69c8f48765-phhsn\" (UID: \"4521c8f6-928b-47a2-8457-de63ee943c09\") " pod="openshift-ingress/e2e-shared-b-w6-7eaf74-openshift-default-69c8f48765-phhsn" Oct 30 21:11:09.237338 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:11:09.237318 5378 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4521c8f6-928b-47a2-8457-de63ee943c09-istio-podinfo\") pod \"e2e-shared-b-w6-7eaf74-openshift-default-69c8f48765-phhsn\" (UID: \"4521c8f6-928b-47a2-8457-de63ee943c09\") " pod="openshift-ingress/e2e-shared-b-w6-7eaf74-openshift-default-69c8f48765-phhsn" Oct 30 21:11:09.299063 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:11:09.298989 5378 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vwffv\" (UniqueName: \"kubernetes.io/projected/4521c8f6-928b-47a2-8457-de63ee943c09-kube-api-access-vwffv\") pod \"e2e-shared-b-w6-7eaf74-openshift-default-69c8f48765-phhsn\" (UID: \"4521c8f6-928b-47a2-8457-de63ee943c09\") " pod="openshift-ingress/e2e-shared-b-w6-7eaf74-openshift-default-69c8f48765-phhsn" Oct 30 21:11:09.308032 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:11:09.308001 5378 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4521c8f6-928b-47a2-8457-de63ee943c09-istio-token\") pod \"e2e-shared-b-w6-7eaf74-openshift-default-69c8f48765-phhsn\" (UID: \"4521c8f6-928b-47a2-8457-de63ee943c09\") " pod="openshift-ingress/e2e-shared-b-w6-7eaf74-openshift-default-69c8f48765-phhsn" Oct 30 21:11:09.354959 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:11:09.354925 5378 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w6-7eaf74-openshift-default-69c8f48765-phhsn" Oct 30 21:11:09.563516 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:11:09.563438 5378 generic.go:356] "Generic (PLEG): container finished" podID="a5f9447f-85f1-40c0-b794-3a39315da3cc" containerID="e6437a86d114df1af144051ae1f9213ff0b6b1a6c558b343f17d3a8cecda7675" exitCode=0 Oct 30 21:11:09.563702 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:11:09.563518 5378 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-ef7e2d-openshift-default-6b6dbc765f-679n5" event={"ID":"a5f9447f-85f1-40c0-b794-3a39315da3cc","Type":"ContainerDied","Data":"e6437a86d114df1af144051ae1f9213ff0b6b1a6c558b343f17d3a8cecda7675"} Oct 30 21:11:09.564057 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:11:09.564038 5378 scope.go:122] "RemoveContainer" containerID="e6437a86d114df1af144051ae1f9213ff0b6b1a6c558b343f17d3a8cecda7675" Oct 30 21:11:09.565317 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:11:09.565292 5378 generic.go:356] "Generic (PLEG): container finished" podID="3559349f-6be4-4ef6-8061-59c7d1a8b678" containerID="7f6ec1fe4760ca648091b49d5fd6b282dd26bfb546798a789e4dc64db5bb8a6b" exitCode=0 Oct 30 21:11:09.565425 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:11:09.565367 5378 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w6-7eaf74-openshift-default-69c8f48765-tmsjl" event={"ID":"3559349f-6be4-4ef6-8061-59c7d1a8b678","Type":"ContainerDied","Data":"7f6ec1fe4760ca648091b49d5fd6b282dd26bfb546798a789e4dc64db5bb8a6b"} Oct 30 21:11:09.565763 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:11:09.565748 5378 scope.go:122] "RemoveContainer" containerID="7f6ec1fe4760ca648091b49d5fd6b282dd26bfb546798a789e4dc64db5bb8a6b" Oct 30 21:11:09.566050 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:11:09.566024 5378 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 21:11:09.566134 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:11:09.566116 5378 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 21:11:09.566192 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:11:09.566169 5378 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 21:11:09.567309 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:11:09.567287 5378 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 21:11:09.567377 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:11:09.567352 5378 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 21:11:09.567413 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:11:09.567384 5378 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 21:11:09.819972 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:11:09.819933 5378 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 21:11:09.820098 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:11:09.820028 5378 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 21:11:09.820167 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:11:09.820116 5378 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 21:11:09.824233 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:11:09.824213 5378 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w6-7eaf74-openshift-default-69c8f48765-phhsn"] Oct 30 21:11:10.606204 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:11:10.604979 5378 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-ef7e2d-openshift-default-6b6dbc765f-679n5" event={"ID":"a5f9447f-85f1-40c0-b794-3a39315da3cc","Type":"ContainerStarted","Data":"0df859899b8cb42a90723ced69e4a6a0adbc2676b84d2ab26af9ceb4b80b81a6"} Oct 30 21:11:10.606926 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:11:10.606907 5378 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-ef7e2d-openshift-default-6b6dbc765f-679n5" Oct 30 21:11:10.608334 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:11:10.608050 5378 patch_prober.go:29] interesting pod/e2e-api-a-ef7e2d-openshift-default-6b6dbc765f-679n5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.135:15021/healthz/ready\": dial tcp 10.133.0.135:15021: connect: connection refused" start-of-body= Oct 30 21:11:10.608334 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:11:10.608098 5378 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-ef7e2d-openshift-default-6b6dbc765f-679n5" podUID="a5f9447f-85f1-40c0-b794-3a39315da3cc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.135:15021/healthz/ready\": dial tcp 10.133.0.135:15021: connect: connection refused" Oct 30 21:11:10.610697 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:11:10.610670 5378 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w6-7eaf74-openshift-default-69c8f48765-phhsn" event={"ID":"4521c8f6-928b-47a2-8457-de63ee943c09","Type":"ContainerStarted","Data":"0d9e73bb769d3c124c9bbae906f5dad2f2c95bce22b8e82a51fb49142ce3aede"} Oct 30 21:11:10.610813 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:11:10.610722 5378 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w6-7eaf74-openshift-default-69c8f48765-phhsn" event={"ID":"4521c8f6-928b-47a2-8457-de63ee943c09","Type":"ContainerStarted","Data":"768609ea3f54dd446802b5a8bbed89cdb068d346679eac4e4d45487239b9599c"} Oct 30 21:11:10.612675 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:11:10.612651 5378 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w6-7eaf74-openshift-default-69c8f48765-tmsjl" event={"ID":"3559349f-6be4-4ef6-8061-59c7d1a8b678","Type":"ContainerStarted","Data":"549031df7c36d7fbdf19c5af343bfe734c786b67dcbc348285d854daaf3e1b52"} Oct 30 21:11:10.947458 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:11:10.947382 5378 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-ef7e2d-openshift-default-6b6dbc765f-679n5" Oct 30 21:11:10.947717 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:11:10.947704 5378 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w6-7eaf74-openshift-default-69c8f48765-tmsjl" Oct 30 21:11:11.047705 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:11:11.039493 5378 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w6-7eaf74-openshift-default-69c8f48765-phhsn" podStartSLOduration=3.039470225 podStartE2EDuration="3.039470225s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:11:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:11:10.852413108 +0000 UTC m=+1961.280527028" watchObservedRunningTime="2026-10-30 21:11:11.039470225 +0000 UTC m=+1961.467584149" Oct 30 21:11:11.357987 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:11:11.357955 5378 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w6-7eaf74-openshift-default-69c8f48765-phhsn" Oct 30 21:11:11.362721 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:11:11.362685 5378 patch_prober.go:29] interesting pod/e2e-shared-b-w6-7eaf74-openshift-default-69c8f48765-phhsn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.144:15021/healthz/ready\": dial tcp 10.133.0.144:15021: connect: connection refused" start-of-body= Oct 30 21:11:11.362997 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:11:11.362963 5378 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w6-7eaf74-openshift-default-69c8f48765-phhsn" podUID="4521c8f6-928b-47a2-8457-de63ee943c09" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.144:15021/healthz/ready\": dial tcp 10.133.0.144:15021: connect: connection refused" Oct 30 21:11:11.530773 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:11:11.530733 5378 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w6-7eaf74-openshift-default-69c8f48765-tmsjl" Oct 30 21:11:11.538586 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:11:11.538525 5378 patch_prober.go:29] interesting pod/e2e-shared-b-w6-7eaf74-openshift-default-69c8f48765-tmsjl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.134:15021/healthz/ready\": dial tcp 10.133.0.134:15021: connect: connection refused" start-of-body= Oct 30 21:11:11.538745 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:11:11.538599 5378 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w6-7eaf74-openshift-default-69c8f48765-tmsjl" podUID="3559349f-6be4-4ef6-8061-59c7d1a8b678" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.134:15021/healthz/ready\": dial tcp 10.133.0.134:15021: connect: connection refused" Oct 30 21:11:11.629829 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:11:11.629737 5378 patch_prober.go:29] interesting pod/e2e-api-a-ef7e2d-openshift-default-6b6dbc765f-679n5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.135:15021/healthz/ready\": dial tcp 10.133.0.135:15021: connect: connection refused" start-of-body= Oct 30 21:11:11.630400 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:11:11.630365 5378 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-ef7e2d-openshift-default-6b6dbc765f-679n5" podUID="a5f9447f-85f1-40c0-b794-3a39315da3cc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.135:15021/healthz/ready\": dial tcp 10.133.0.135:15021: connect: connection refused" Oct 30 21:11:11.783251 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:11:11.783216 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:11:12.376904 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:11:12.376863 5378 patch_prober.go:29] interesting pod/e2e-shared-b-w6-7eaf74-openshift-default-69c8f48765-phhsn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.144:15021/healthz/ready\": dial tcp 10.133.0.144:15021: connect: connection refused" start-of-body= Oct 30 21:11:12.377189 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:11:12.377156 5378 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w6-7eaf74-openshift-default-69c8f48765-phhsn" podUID="4521c8f6-928b-47a2-8457-de63ee943c09" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.144:15021/healthz/ready\": dial tcp 10.133.0.144:15021: connect: connection refused" Oct 30 21:11:12.530855 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:11:12.529953 5378 patch_prober.go:29] interesting pod/e2e-shared-b-w6-7eaf74-openshift-default-69c8f48765-tmsjl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.134:15021/healthz/ready\": dial tcp 10.133.0.134:15021: connect: connection refused" start-of-body= Oct 30 21:11:12.530855 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:11:12.530023 5378 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w6-7eaf74-openshift-default-69c8f48765-tmsjl" podUID="3559349f-6be4-4ef6-8061-59c7d1a8b678" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.134:15021/healthz/ready\": dial tcp 10.133.0.134:15021: connect: connection refused" Oct 30 21:11:12.619589 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:11:12.607010 5378 patch_prober.go:29] interesting pod/e2e-api-a-ef7e2d-openshift-default-6b6dbc765f-679n5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.135:15021/healthz/ready\": dial tcp 10.133.0.135:15021: connect: connection refused" start-of-body= Oct 30 21:11:12.619589 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:11:12.607071 5378 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-ef7e2d-openshift-default-6b6dbc765f-679n5" podUID="a5f9447f-85f1-40c0-b794-3a39315da3cc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.135:15021/healthz/ready\": dial tcp 10.133.0.135:15021: connect: connection refused" Oct 30 21:11:13.363591 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:11:13.356976 5378 patch_prober.go:29] interesting pod/e2e-shared-b-w6-7eaf74-openshift-default-69c8f48765-phhsn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.144:15021/healthz/ready\": dial tcp 10.133.0.144:15021: connect: connection refused" start-of-body= Oct 30 21:11:13.363591 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:11:13.357040 5378 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w6-7eaf74-openshift-default-69c8f48765-phhsn" podUID="4521c8f6-928b-47a2-8457-de63ee943c09" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.144:15021/healthz/ready\": dial tcp 10.133.0.144:15021: connect: connection refused" Oct 30 21:11:13.528587 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:11:13.527978 5378 patch_prober.go:29] interesting pod/e2e-shared-b-w6-7eaf74-openshift-default-69c8f48765-tmsjl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.134:15021/healthz/ready\": dial tcp 10.133.0.134:15021: connect: connection refused" start-of-body= Oct 30 21:11:13.528587 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:11:13.528060 5378 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w6-7eaf74-openshift-default-69c8f48765-tmsjl" podUID="3559349f-6be4-4ef6-8061-59c7d1a8b678" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.134:15021/healthz/ready\": dial tcp 10.133.0.134:15021: connect: connection refused" Oct 30 21:11:13.607624 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:11:13.607307 5378 patch_prober.go:29] interesting pod/e2e-api-a-ef7e2d-openshift-default-6b6dbc765f-679n5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.135:15021/healthz/ready\": dial tcp 10.133.0.135:15021: connect: connection refused" start-of-body= Oct 30 21:11:13.607624 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:11:13.607368 5378 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-ef7e2d-openshift-default-6b6dbc765f-679n5" podUID="a5f9447f-85f1-40c0-b794-3a39315da3cc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.135:15021/healthz/ready\": dial tcp 10.133.0.135:15021: connect: connection refused" Oct 30 21:11:14.357590 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:11:14.356976 5378 patch_prober.go:29] interesting pod/e2e-shared-b-w6-7eaf74-openshift-default-69c8f48765-phhsn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.144:15021/healthz/ready\": dial tcp 10.133.0.144:15021: connect: connection refused" start-of-body= Oct 30 21:11:14.357590 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:11:14.357060 5378 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w6-7eaf74-openshift-default-69c8f48765-phhsn" podUID="4521c8f6-928b-47a2-8457-de63ee943c09" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.144:15021/healthz/ready\": dial tcp 10.133.0.144:15021: connect: connection refused" Oct 30 21:11:14.524587 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:11:14.523187 5378 patch_prober.go:29] interesting pod/e2e-shared-b-w6-7eaf74-openshift-default-69c8f48765-tmsjl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.134:15021/healthz/ready\": dial tcp 10.133.0.134:15021: connect: connection refused" start-of-body= Oct 30 21:11:14.524587 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:11:14.523248 5378 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w6-7eaf74-openshift-default-69c8f48765-tmsjl" podUID="3559349f-6be4-4ef6-8061-59c7d1a8b678" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.134:15021/healthz/ready\": dial tcp 10.133.0.134:15021: connect: connection refused" Oct 30 21:11:14.612117 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:11:14.612024 5378 patch_prober.go:29] interesting pod/e2e-api-a-ef7e2d-openshift-default-6b6dbc765f-679n5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.135:15021/healthz/ready\": dial tcp 10.133.0.135:15021: connect: connection refused" start-of-body= Oct 30 21:11:14.612351 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:11:14.612322 5378 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-ef7e2d-openshift-default-6b6dbc765f-679n5" podUID="a5f9447f-85f1-40c0-b794-3a39315da3cc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.135:15021/healthz/ready\": dial tcp 10.133.0.135:15021: connect: connection refused" Oct 30 21:11:15.356591 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:11:15.355920 5378 patch_prober.go:29] interesting pod/e2e-shared-b-w6-7eaf74-openshift-default-69c8f48765-phhsn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.144:15021/healthz/ready\": dial tcp 10.133.0.144:15021: connect: connection refused" start-of-body= Oct 30 21:11:15.356591 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:11:15.355975 5378 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w6-7eaf74-openshift-default-69c8f48765-phhsn" podUID="4521c8f6-928b-47a2-8457-de63ee943c09" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.144:15021/healthz/ready\": dial tcp 10.133.0.144:15021: connect: connection refused" Oct 30 21:11:15.523011 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:11:15.522973 5378 patch_prober.go:29] interesting pod/e2e-shared-b-w6-7eaf74-openshift-default-69c8f48765-tmsjl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.134:15021/healthz/ready\": dial tcp 10.133.0.134:15021: connect: connection refused" start-of-body= Oct 30 21:11:15.523288 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:11:15.523257 5378 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w6-7eaf74-openshift-default-69c8f48765-tmsjl" podUID="3559349f-6be4-4ef6-8061-59c7d1a8b678" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.134:15021/healthz/ready\": dial tcp 10.133.0.134:15021: connect: connection refused" Oct 30 21:11:15.603965 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:11:15.603927 5378 patch_prober.go:29] interesting pod/e2e-api-a-ef7e2d-openshift-default-6b6dbc765f-679n5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.135:15021/healthz/ready\": dial tcp 10.133.0.135:15021: connect: connection refused" start-of-body= Oct 30 21:11:15.604592 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:11:15.604536 5378 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-ef7e2d-openshift-default-6b6dbc765f-679n5" podUID="a5f9447f-85f1-40c0-b794-3a39315da3cc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.135:15021/healthz/ready\": dial tcp 10.133.0.135:15021: connect: connection refused" Oct 30 21:11:16.367614 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:11:16.364071 5378 patch_prober.go:29] interesting pod/e2e-shared-b-w6-7eaf74-openshift-default-69c8f48765-phhsn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.144:15021/healthz/ready\": dial tcp 10.133.0.144:15021: connect: connection refused" start-of-body= Oct 30 21:11:16.367614 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:11:16.364148 5378 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w6-7eaf74-openshift-default-69c8f48765-phhsn" podUID="4521c8f6-928b-47a2-8457-de63ee943c09" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.144:15021/healthz/ready\": dial tcp 10.133.0.144:15021: connect: connection refused" Oct 30 21:11:16.644642 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:11:16.637939 5378 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w6-7eaf74-openshift-default-69c8f48765-tmsjl" Oct 30 21:11:16.652641 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:11:16.647777 5378 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w6-7eaf74-openshift-default-69c8f48765-tmsjl" Oct 30 21:11:16.680586 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:11:16.680227 5378 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-ef7e2d-openshift-default-6b6dbc765f-679n5" Oct 30 21:11:16.687583 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:11:16.684763 5378 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-ef7e2d-openshift-default-6b6dbc765f-679n5" Oct 30 21:11:17.255066 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:11:17.255024 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:11:17.360512 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:11:17.360481 5378 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w6-7eaf74-openshift-default-69c8f48765-phhsn" Oct 30 21:11:17.360963 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:11:17.360944 5378 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w6-7eaf74-openshift-default-69c8f48765-phhsn" Oct 30 21:11:17.361528 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:11:17.361507 5378 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w6-7eaf74-openshift-default-69c8f48765-phhsn" Oct 30 21:11:17.424827 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:11:17.424799 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:11:17.567766 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:11:17.567668 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:11:18.838487 ip-10-0-138-149 kubenswrapper[5378]: E1030 21:11:18.838448 5378 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q6mx7" containerName="extractor" Oct 30 21:11:22.272147 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:11:22.272107 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:11:27.162269 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:11:27.162239 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:11:28.697319 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:11:28.697288 5378 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-9926f7-57bb5db5fb-b224f"] Oct 30 21:11:28.803514 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:11:28.803478 5378 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-9926f7-646b87d4d7-pgzdg"] Oct 30 21:11:29.333587 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:11:29.329177 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:11:29.471842 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:11:29.471811 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:11:30.768296 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:11:30.768264 5378 scope.go:122] "RemoveContainer" containerID="36a0351f1605358413b4ed86dc714b151758b0961124deaed1360602b7804c2a" Oct 30 21:11:32.117116 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:11:32.117078 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:11:33.697689 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:11:33.697646 5378 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-a-9926f7-57bb5db5fb-b224f" podUID="6d33dc48-4424-477b-ae64-bfc39c7faaa3" containerName="payload-processing" containerID="cri-o://4a9df14a558134ad49179773491f61f140605eab1ce581abcfc1ef8509dc20ce" gracePeriod=25 Oct 30 21:11:33.804593 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:11:33.804533 5378 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-a-9926f7-646b87d4d7-pgzdg" podUID="2fd814bb-f2a6-4ae3-a53c-6108f380f607" containerName="payload-pre-processing" containerID="cri-o://16c7b014c1e6335bf5a943d7aafdc9279ae20b9a34da594caf26aef8ed563753" gracePeriod=25 Oct 30 21:11:33.934084 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:11:33.934030 5378 generic.go:356] "Generic (PLEG): container finished" podID="6d33dc48-4424-477b-ae64-bfc39c7faaa3" containerID="4a9df14a558134ad49179773491f61f140605eab1ce581abcfc1ef8509dc20ce" exitCode=0 Oct 30 21:11:33.934211 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:11:33.934120 5378 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-9926f7-57bb5db5fb-b224f" event={"ID":"6d33dc48-4424-477b-ae64-bfc39c7faaa3","Type":"ContainerDied","Data":"4a9df14a558134ad49179773491f61f140605eab1ce581abcfc1ef8509dc20ce"} Oct 30 21:11:34.035287 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:11:34.035256 5378 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-9926f7-57bb5db5fb-b224f" Oct 30 21:11:34.158224 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:11:34.156558 5378 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6d33dc48-4424-477b-ae64-bfc39c7faaa3-kube-api-access-tphpj\" (UniqueName: \"kubernetes.io/projected/6d33dc48-4424-477b-ae64-bfc39c7faaa3-kube-api-access-tphpj\") pod \"6d33dc48-4424-477b-ae64-bfc39c7faaa3\" (UID: \"6d33dc48-4424-477b-ae64-bfc39c7faaa3\") " Oct 30 21:11:34.158224 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:11:34.156910 5378 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6d33dc48-4424-477b-ae64-bfc39c7faaa3-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6d33dc48-4424-477b-ae64-bfc39c7faaa3-plugins-config-volume\") pod \"6d33dc48-4424-477b-ae64-bfc39c7faaa3\" (UID: \"6d33dc48-4424-477b-ae64-bfc39c7faaa3\") " Oct 30 21:11:34.161825 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:11:34.161789 5378 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6d33dc48-4424-477b-ae64-bfc39c7faaa3-plugins-config-volume" pod "6d33dc48-4424-477b-ae64-bfc39c7faaa3" (UID: "6d33dc48-4424-477b-ae64-bfc39c7faaa3"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:11:34.164688 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:11:34.164649 5378 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6d33dc48-4424-477b-ae64-bfc39c7faaa3-kube-api-access-tphpj" pod "6d33dc48-4424-477b-ae64-bfc39c7faaa3" (UID: "6d33dc48-4424-477b-ae64-bfc39c7faaa3"). InnerVolumeSpecName "kube-api-access-tphpj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:11:34.165283 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:11:34.164980 5378 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-9926f7-646b87d4d7-pgzdg" Oct 30 21:11:34.259470 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:11:34.259370 5378 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2fd814bb-f2a6-4ae3-a53c-6108f380f607-kube-api-access-lfs2k\" (UniqueName: \"kubernetes.io/projected/2fd814bb-f2a6-4ae3-a53c-6108f380f607-kube-api-access-lfs2k\") pod \"2fd814bb-f2a6-4ae3-a53c-6108f380f607\" (UID: \"2fd814bb-f2a6-4ae3-a53c-6108f380f607\") " Oct 30 21:11:34.259829 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:11:34.259803 5378 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2fd814bb-f2a6-4ae3-a53c-6108f380f607-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2fd814bb-f2a6-4ae3-a53c-6108f380f607-plugins-config-volume\") pod \"2fd814bb-f2a6-4ae3-a53c-6108f380f607\" (UID: \"2fd814bb-f2a6-4ae3-a53c-6108f380f607\") " Oct 30 21:11:34.260313 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:11:34.260287 5378 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2fd814bb-f2a6-4ae3-a53c-6108f380f607-plugins-config-volume" pod "2fd814bb-f2a6-4ae3-a53c-6108f380f607" (UID: "2fd814bb-f2a6-4ae3-a53c-6108f380f607"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:11:34.260608 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:11:34.260587 5378 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2fd814bb-f2a6-4ae3-a53c-6108f380f607-plugins-config-volume\") on node \"ip-10-0-138-149.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:11:34.260685 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:11:34.260612 5378 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tphpj\" (UniqueName: \"kubernetes.io/projected/6d33dc48-4424-477b-ae64-bfc39c7faaa3-kube-api-access-tphpj\") on node \"ip-10-0-138-149.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:11:34.260685 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:11:34.260626 5378 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6d33dc48-4424-477b-ae64-bfc39c7faaa3-plugins-config-volume\") on node \"ip-10-0-138-149.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:11:34.262206 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:11:34.262185 5378 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2fd814bb-f2a6-4ae3-a53c-6108f380f607-kube-api-access-lfs2k" pod "2fd814bb-f2a6-4ae3-a53c-6108f380f607" (UID: "2fd814bb-f2a6-4ae3-a53c-6108f380f607"). InnerVolumeSpecName "kube-api-access-lfs2k". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:11:34.361954 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:11:34.361914 5378 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lfs2k\" (UniqueName: \"kubernetes.io/projected/2fd814bb-f2a6-4ae3-a53c-6108f380f607-kube-api-access-lfs2k\") on node \"ip-10-0-138-149.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:11:34.943718 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:11:34.943682 5378 generic.go:356] "Generic (PLEG): container finished" podID="2fd814bb-f2a6-4ae3-a53c-6108f380f607" containerID="16c7b014c1e6335bf5a943d7aafdc9279ae20b9a34da594caf26aef8ed563753" exitCode=0 Oct 30 21:11:34.944149 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:11:34.943743 5378 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-9926f7-646b87d4d7-pgzdg" Oct 30 21:11:34.944149 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:11:34.943768 5378 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-9926f7-646b87d4d7-pgzdg" event={"ID":"2fd814bb-f2a6-4ae3-a53c-6108f380f607","Type":"ContainerDied","Data":"16c7b014c1e6335bf5a943d7aafdc9279ae20b9a34da594caf26aef8ed563753"} Oct 30 21:11:34.944149 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:11:34.943824 5378 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-9926f7-646b87d4d7-pgzdg" event={"ID":"2fd814bb-f2a6-4ae3-a53c-6108f380f607","Type":"ContainerDied","Data":"220b40cabe0ea327951f88952db811071c0b1a215d521f42240af84958e98669"} Oct 30 21:11:34.944149 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:11:34.943850 5378 scope.go:122] "RemoveContainer" containerID="16c7b014c1e6335bf5a943d7aafdc9279ae20b9a34da594caf26aef8ed563753" Oct 30 21:11:34.945470 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:11:34.945421 5378 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-9926f7-57bb5db5fb-b224f" event={"ID":"6d33dc48-4424-477b-ae64-bfc39c7faaa3","Type":"ContainerDied","Data":"fddc5f3afee6ab0bfefeedfa7f5b416b8704dbd139e76c61ccf84ed308bb9abb"} Oct 30 21:11:34.945470 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:11:34.945467 5378 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-9926f7-57bb5db5fb-b224f" Oct 30 21:11:34.955312 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:11:34.955294 5378 scope.go:122] "RemoveContainer" containerID="16c7b014c1e6335bf5a943d7aafdc9279ae20b9a34da594caf26aef8ed563753" Oct 30 21:11:34.955692 ip-10-0-138-149 kubenswrapper[5378]: E1030 21:11:34.955652 5378 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"16c7b014c1e6335bf5a943d7aafdc9279ae20b9a34da594caf26aef8ed563753\": container with ID starting with 16c7b014c1e6335bf5a943d7aafdc9279ae20b9a34da594caf26aef8ed563753 not found: ID does not exist" containerID="16c7b014c1e6335bf5a943d7aafdc9279ae20b9a34da594caf26aef8ed563753" Oct 30 21:11:34.955814 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:11:34.955702 5378 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"16c7b014c1e6335bf5a943d7aafdc9279ae20b9a34da594caf26aef8ed563753"} err="failed to get container status \"16c7b014c1e6335bf5a943d7aafdc9279ae20b9a34da594caf26aef8ed563753\": rpc error: code = NotFound desc = could not find container \"16c7b014c1e6335bf5a943d7aafdc9279ae20b9a34da594caf26aef8ed563753\": container with ID starting with 16c7b014c1e6335bf5a943d7aafdc9279ae20b9a34da594caf26aef8ed563753 not found: ID does not exist" Oct 30 21:11:34.955814 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:11:34.955728 5378 scope.go:122] "RemoveContainer" containerID="4a9df14a558134ad49179773491f61f140605eab1ce581abcfc1ef8509dc20ce" Oct 30 21:11:35.110902 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:11:35.110874 5378 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-9926f7-646b87d4d7-pgzdg"] Oct 30 21:11:35.161917 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:11:35.161888 5378 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-9926f7-646b87d4d7-pgzdg"] Oct 30 21:11:35.235492 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:11:35.235458 5378 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-9926f7-57bb5db5fb-b224f"] Oct 30 21:11:35.247897 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:11:35.247872 5378 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-9926f7-57bb5db5fb-b224f"] Oct 30 21:11:35.745302 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:11:35.745272 5378 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2fd814bb-f2a6-4ae3-a53c-6108f380f607" path="/var/lib/kubelet/pods/2fd814bb-f2a6-4ae3-a53c-6108f380f607/volumes" Oct 30 21:11:35.745785 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:11:35.745770 5378 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6d33dc48-4424-477b-ae64-bfc39c7faaa3" path="/var/lib/kubelet/pods/6d33dc48-4424-477b-ae64-bfc39c7faaa3/volumes" Oct 30 21:11:36.902498 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:11:36.902464 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:11:39.226176 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:11:39.226101 5378 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w6-7eaf74-openshift-default-69c8f48765-r2cmt"] Oct 30 21:11:39.226925 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:11:39.226907 5378 cpu_manager.go:414] "containerMap: removing container" podUID="2fd814bb-f2a6-4ae3-a53c-6108f380f607" containerName="payload-pre-processing" Oct 30 21:11:39.226990 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:11:39.226926 5378 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2fd814bb-f2a6-4ae3-a53c-6108f380f607" containerName="payload-pre-processing" Oct 30 21:11:39.226990 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:11:39.226964 5378 cpu_manager.go:414] "containerMap: removing container" podUID="6d33dc48-4424-477b-ae64-bfc39c7faaa3" containerName="payload-processing" Oct 30 21:11:39.226990 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:11:39.226970 5378 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6d33dc48-4424-477b-ae64-bfc39c7faaa3" containerName="payload-processing" Oct 30 21:11:39.227113 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:11:39.227103 5378 memory_manager.go:365] "RemoveStaleState removing state" podUID="2fd814bb-f2a6-4ae3-a53c-6108f380f607" containerName="payload-pre-processing" Oct 30 21:11:39.227148 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:11:39.227120 5378 memory_manager.go:365] "RemoveStaleState removing state" podUID="6d33dc48-4424-477b-ae64-bfc39c7faaa3" containerName="payload-processing" Oct 30 21:11:39.230539 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:11:39.230520 5378 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w6-7eaf74-openshift-default-69c8f48765-r2cmt" Oct 30 21:11:39.334057 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:11:39.334025 5378 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wvvfh\" (UniqueName: \"kubernetes.io/projected/1eb6161d-5e96-4dd5-bbe1-68300df5c425-kube-api-access-wvvfh\") pod \"e2e-shared-b-w6-7eaf74-openshift-default-69c8f48765-r2cmt\" (UID: \"1eb6161d-5e96-4dd5-bbe1-68300df5c425\") " pod="openshift-ingress/e2e-shared-b-w6-7eaf74-openshift-default-69c8f48765-r2cmt" Oct 30 21:11:39.334302 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:11:39.334200 5378 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1eb6161d-5e96-4dd5-bbe1-68300df5c425-istio-token\") pod \"e2e-shared-b-w6-7eaf74-openshift-default-69c8f48765-r2cmt\" (UID: \"1eb6161d-5e96-4dd5-bbe1-68300df5c425\") " pod="openshift-ingress/e2e-shared-b-w6-7eaf74-openshift-default-69c8f48765-r2cmt" Oct 30 21:11:39.334302 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:11:39.334255 5378 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1eb6161d-5e96-4dd5-bbe1-68300df5c425-credential-socket\") pod \"e2e-shared-b-w6-7eaf74-openshift-default-69c8f48765-r2cmt\" (UID: \"1eb6161d-5e96-4dd5-bbe1-68300df5c425\") " pod="openshift-ingress/e2e-shared-b-w6-7eaf74-openshift-default-69c8f48765-r2cmt" Oct 30 21:11:39.334401 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:11:39.334356 5378 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1eb6161d-5e96-4dd5-bbe1-68300df5c425-workload-certs\") pod \"e2e-shared-b-w6-7eaf74-openshift-default-69c8f48765-r2cmt\" (UID: \"1eb6161d-5e96-4dd5-bbe1-68300df5c425\") " pod="openshift-ingress/e2e-shared-b-w6-7eaf74-openshift-default-69c8f48765-r2cmt" Oct 30 21:11:39.334476 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:11:39.334462 5378 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1eb6161d-5e96-4dd5-bbe1-68300df5c425-istio-envoy\") pod \"e2e-shared-b-w6-7eaf74-openshift-default-69c8f48765-r2cmt\" (UID: \"1eb6161d-5e96-4dd5-bbe1-68300df5c425\") " pod="openshift-ingress/e2e-shared-b-w6-7eaf74-openshift-default-69c8f48765-r2cmt" Oct 30 21:11:39.334549 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:11:39.334535 5378 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1eb6161d-5e96-4dd5-bbe1-68300df5c425-istio-data\") pod \"e2e-shared-b-w6-7eaf74-openshift-default-69c8f48765-r2cmt\" (UID: \"1eb6161d-5e96-4dd5-bbe1-68300df5c425\") " pod="openshift-ingress/e2e-shared-b-w6-7eaf74-openshift-default-69c8f48765-r2cmt" Oct 30 21:11:39.334649 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:11:39.334624 5378 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1eb6161d-5e96-4dd5-bbe1-68300df5c425-istiod-ca-cert\") pod \"e2e-shared-b-w6-7eaf74-openshift-default-69c8f48765-r2cmt\" (UID: \"1eb6161d-5e96-4dd5-bbe1-68300df5c425\") " pod="openshift-ingress/e2e-shared-b-w6-7eaf74-openshift-default-69c8f48765-r2cmt" Oct 30 21:11:39.334649 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:11:39.334644 5378 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1eb6161d-5e96-4dd5-bbe1-68300df5c425-workload-socket\") pod \"e2e-shared-b-w6-7eaf74-openshift-default-69c8f48765-r2cmt\" (UID: \"1eb6161d-5e96-4dd5-bbe1-68300df5c425\") " pod="openshift-ingress/e2e-shared-b-w6-7eaf74-openshift-default-69c8f48765-r2cmt" Oct 30 21:11:39.334729 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:11:39.334665 5378 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1eb6161d-5e96-4dd5-bbe1-68300df5c425-istio-podinfo\") pod \"e2e-shared-b-w6-7eaf74-openshift-default-69c8f48765-r2cmt\" (UID: \"1eb6161d-5e96-4dd5-bbe1-68300df5c425\") " pod="openshift-ingress/e2e-shared-b-w6-7eaf74-openshift-default-69c8f48765-r2cmt" Oct 30 21:11:39.438175 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:11:39.438142 5378 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w6-7eaf74-openshift-default-69c8f48765-r2cmt"] Oct 30 21:11:39.439185 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:11:39.439146 5378 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1eb6161d-5e96-4dd5-bbe1-68300df5c425-istio-token\") pod \"e2e-shared-b-w6-7eaf74-openshift-default-69c8f48765-r2cmt\" (UID: \"1eb6161d-5e96-4dd5-bbe1-68300df5c425\") " pod="openshift-ingress/e2e-shared-b-w6-7eaf74-openshift-default-69c8f48765-r2cmt" Oct 30 21:11:39.439363 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:11:39.439237 5378 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1eb6161d-5e96-4dd5-bbe1-68300df5c425-credential-socket\") pod \"e2e-shared-b-w6-7eaf74-openshift-default-69c8f48765-r2cmt\" (UID: \"1eb6161d-5e96-4dd5-bbe1-68300df5c425\") " pod="openshift-ingress/e2e-shared-b-w6-7eaf74-openshift-default-69c8f48765-r2cmt" Oct 30 21:11:39.439363 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:11:39.439280 5378 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1eb6161d-5e96-4dd5-bbe1-68300df5c425-workload-certs\") pod \"e2e-shared-b-w6-7eaf74-openshift-default-69c8f48765-r2cmt\" (UID: \"1eb6161d-5e96-4dd5-bbe1-68300df5c425\") " pod="openshift-ingress/e2e-shared-b-w6-7eaf74-openshift-default-69c8f48765-r2cmt" Oct 30 21:11:39.439491 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:11:39.439373 5378 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1eb6161d-5e96-4dd5-bbe1-68300df5c425-istio-envoy\") pod \"e2e-shared-b-w6-7eaf74-openshift-default-69c8f48765-r2cmt\" (UID: \"1eb6161d-5e96-4dd5-bbe1-68300df5c425\") " pod="openshift-ingress/e2e-shared-b-w6-7eaf74-openshift-default-69c8f48765-r2cmt" Oct 30 21:11:39.439491 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:11:39.439464 5378 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1eb6161d-5e96-4dd5-bbe1-68300df5c425-istio-data\") pod \"e2e-shared-b-w6-7eaf74-openshift-default-69c8f48765-r2cmt\" (UID: \"1eb6161d-5e96-4dd5-bbe1-68300df5c425\") " pod="openshift-ingress/e2e-shared-b-w6-7eaf74-openshift-default-69c8f48765-r2cmt" Oct 30 21:11:39.439651 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:11:39.439521 5378 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1eb6161d-5e96-4dd5-bbe1-68300df5c425-istiod-ca-cert\") pod \"e2e-shared-b-w6-7eaf74-openshift-default-69c8f48765-r2cmt\" (UID: \"1eb6161d-5e96-4dd5-bbe1-68300df5c425\") " pod="openshift-ingress/e2e-shared-b-w6-7eaf74-openshift-default-69c8f48765-r2cmt" Oct 30 21:11:39.439651 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:11:39.439551 5378 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1eb6161d-5e96-4dd5-bbe1-68300df5c425-workload-socket\") pod \"e2e-shared-b-w6-7eaf74-openshift-default-69c8f48765-r2cmt\" (UID: \"1eb6161d-5e96-4dd5-bbe1-68300df5c425\") " pod="openshift-ingress/e2e-shared-b-w6-7eaf74-openshift-default-69c8f48765-r2cmt" Oct 30 21:11:39.439651 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:11:39.439608 5378 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1eb6161d-5e96-4dd5-bbe1-68300df5c425-istio-podinfo\") pod \"e2e-shared-b-w6-7eaf74-openshift-default-69c8f48765-r2cmt\" (UID: \"1eb6161d-5e96-4dd5-bbe1-68300df5c425\") " pod="openshift-ingress/e2e-shared-b-w6-7eaf74-openshift-default-69c8f48765-r2cmt" Oct 30 21:11:39.439824 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:11:39.439704 5378 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wvvfh\" (UniqueName: \"kubernetes.io/projected/1eb6161d-5e96-4dd5-bbe1-68300df5c425-kube-api-access-wvvfh\") pod \"e2e-shared-b-w6-7eaf74-openshift-default-69c8f48765-r2cmt\" (UID: \"1eb6161d-5e96-4dd5-bbe1-68300df5c425\") " pod="openshift-ingress/e2e-shared-b-w6-7eaf74-openshift-default-69c8f48765-r2cmt" Oct 30 21:11:39.439905 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:11:39.439883 5378 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1eb6161d-5e96-4dd5-bbe1-68300df5c425-credential-socket\") pod \"e2e-shared-b-w6-7eaf74-openshift-default-69c8f48765-r2cmt\" (UID: \"1eb6161d-5e96-4dd5-bbe1-68300df5c425\") " pod="openshift-ingress/e2e-shared-b-w6-7eaf74-openshift-default-69c8f48765-r2cmt" Oct 30 21:11:39.440540 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:11:39.440511 5378 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1eb6161d-5e96-4dd5-bbe1-68300df5c425-workload-socket\") pod \"e2e-shared-b-w6-7eaf74-openshift-default-69c8f48765-r2cmt\" (UID: \"1eb6161d-5e96-4dd5-bbe1-68300df5c425\") " pod="openshift-ingress/e2e-shared-b-w6-7eaf74-openshift-default-69c8f48765-r2cmt" Oct 30 21:11:39.440699 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:11:39.440542 5378 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1eb6161d-5e96-4dd5-bbe1-68300df5c425-istio-data\") pod \"e2e-shared-b-w6-7eaf74-openshift-default-69c8f48765-r2cmt\" (UID: \"1eb6161d-5e96-4dd5-bbe1-68300df5c425\") " pod="openshift-ingress/e2e-shared-b-w6-7eaf74-openshift-default-69c8f48765-r2cmt" Oct 30 21:11:39.440836 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:11:39.440814 5378 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1eb6161d-5e96-4dd5-bbe1-68300df5c425-workload-certs\") pod \"e2e-shared-b-w6-7eaf74-openshift-default-69c8f48765-r2cmt\" (UID: \"1eb6161d-5e96-4dd5-bbe1-68300df5c425\") " pod="openshift-ingress/e2e-shared-b-w6-7eaf74-openshift-default-69c8f48765-r2cmt" Oct 30 21:11:39.441638 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:11:39.441614 5378 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1eb6161d-5e96-4dd5-bbe1-68300df5c425-istiod-ca-cert\") pod \"e2e-shared-b-w6-7eaf74-openshift-default-69c8f48765-r2cmt\" (UID: \"1eb6161d-5e96-4dd5-bbe1-68300df5c425\") " pod="openshift-ingress/e2e-shared-b-w6-7eaf74-openshift-default-69c8f48765-r2cmt" Oct 30 21:11:39.453802 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:11:39.444091 5378 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1eb6161d-5e96-4dd5-bbe1-68300df5c425-istio-podinfo\") pod \"e2e-shared-b-w6-7eaf74-openshift-default-69c8f48765-r2cmt\" (UID: \"1eb6161d-5e96-4dd5-bbe1-68300df5c425\") " pod="openshift-ingress/e2e-shared-b-w6-7eaf74-openshift-default-69c8f48765-r2cmt" Oct 30 21:11:39.453802 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:11:39.444300 5378 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1eb6161d-5e96-4dd5-bbe1-68300df5c425-istio-envoy\") pod \"e2e-shared-b-w6-7eaf74-openshift-default-69c8f48765-r2cmt\" (UID: \"1eb6161d-5e96-4dd5-bbe1-68300df5c425\") " pod="openshift-ingress/e2e-shared-b-w6-7eaf74-openshift-default-69c8f48765-r2cmt" Oct 30 21:11:39.496725 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:11:39.496657 5378 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1eb6161d-5e96-4dd5-bbe1-68300df5c425-istio-token\") pod \"e2e-shared-b-w6-7eaf74-openshift-default-69c8f48765-r2cmt\" (UID: \"1eb6161d-5e96-4dd5-bbe1-68300df5c425\") " pod="openshift-ingress/e2e-shared-b-w6-7eaf74-openshift-default-69c8f48765-r2cmt" Oct 30 21:11:39.498020 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:11:39.497992 5378 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wvvfh\" (UniqueName: \"kubernetes.io/projected/1eb6161d-5e96-4dd5-bbe1-68300df5c425-kube-api-access-wvvfh\") pod \"e2e-shared-b-w6-7eaf74-openshift-default-69c8f48765-r2cmt\" (UID: \"1eb6161d-5e96-4dd5-bbe1-68300df5c425\") " pod="openshift-ingress/e2e-shared-b-w6-7eaf74-openshift-default-69c8f48765-r2cmt" Oct 30 21:11:39.544191 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:11:39.544161 5378 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w6-7eaf74-openshift-default-69c8f48765-r2cmt" Oct 30 21:11:39.981485 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:11:39.981456 5378 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w6-7eaf74-openshift-default-69c8f48765-r2cmt"] Oct 30 21:11:39.982291 ip-10-0-138-149 kubenswrapper[5378]: W1030 21:11:39.982260 5378 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1eb6161d_5e96_4dd5_bbe1_68300df5c425.slice/crio-36a78e7c35f2196cb956ce7ef0236bce7990c83a169cdc4d729130c95f149dba WatchSource:0}: Error finding container 36a78e7c35f2196cb956ce7ef0236bce7990c83a169cdc4d729130c95f149dba: Status 404 returned error can't find the container with id 36a78e7c35f2196cb956ce7ef0236bce7990c83a169cdc4d729130c95f149dba Oct 30 21:11:39.985129 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:11:39.985087 5378 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 21:11:39.985232 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:11:39.985178 5378 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 21:11:39.985232 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:11:39.985221 5378 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 21:11:40.012893 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:11:40.012868 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:11:40.075023 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:11:40.074996 5378 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w6-7eaf74-openshift-default-69c8f48765-r2cmt" event={"ID":"1eb6161d-5e96-4dd5-bbe1-68300df5c425","Type":"ContainerStarted","Data":"353294d3f04e0c51f2a7fc80a1531f6733ae2e83281940b801e73fa00431c4df"} Oct 30 21:11:40.075151 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:11:40.075050 5378 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w6-7eaf74-openshift-default-69c8f48765-r2cmt" event={"ID":"1eb6161d-5e96-4dd5-bbe1-68300df5c425","Type":"ContainerStarted","Data":"36a78e7c35f2196cb956ce7ef0236bce7990c83a169cdc4d729130c95f149dba"} Oct 30 21:11:40.166193 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:11:40.166155 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:11:40.194190 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:11:40.194134 5378 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w6-7eaf74-openshift-default-69c8f48765-r2cmt" podStartSLOduration=1.194115028 podStartE2EDuration="1.194115028s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:11:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:11:40.193326111 +0000 UTC m=+1990.621440040" watchObservedRunningTime="2026-10-30 21:11:40.194115028 +0000 UTC m=+1990.622228948" Oct 30 21:11:40.544299 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:11:40.544265 5378 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w6-7eaf74-openshift-default-69c8f48765-r2cmt" Oct 30 21:11:41.555587 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:11:41.552691 5378 patch_prober.go:29] interesting pod/e2e-shared-b-w6-7eaf74-openshift-default-69c8f48765-r2cmt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.145:15021/healthz/ready\": dial tcp 10.133.0.145:15021: connect: connection refused" start-of-body= Oct 30 21:11:41.555587 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:11:41.552760 5378 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w6-7eaf74-openshift-default-69c8f48765-r2cmt" podUID="1eb6161d-5e96-4dd5-bbe1-68300df5c425" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.145:15021/healthz/ready\": dial tcp 10.133.0.145:15021: connect: connection refused" Oct 30 21:11:41.714599 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:11:41.713694 5378 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-r9g59"] Oct 30 21:11:41.930615 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:11:41.930582 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:11:42.176165 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:11:42.176086 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:11:42.320003 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:11:42.319920 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:11:42.551985 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:11:42.551954 5378 patch_prober.go:29] interesting pod/e2e-shared-b-w6-7eaf74-openshift-default-69c8f48765-r2cmt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.145:15021/healthz/ready\": dial tcp 10.133.0.145:15021: connect: connection refused" start-of-body= Oct 30 21:11:42.552255 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:11:42.552229 5378 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w6-7eaf74-openshift-default-69c8f48765-r2cmt" podUID="1eb6161d-5e96-4dd5-bbe1-68300df5c425" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.145:15021/healthz/ready\": dial tcp 10.133.0.145:15021: connect: connection refused" Oct 30 21:11:42.656591 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:11:42.655707 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:11:43.545102 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:11:43.545070 5378 patch_prober.go:29] interesting pod/e2e-shared-b-w6-7eaf74-openshift-default-69c8f48765-r2cmt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.145:15021/healthz/ready\": dial tcp 10.133.0.145:15021: connect: connection refused" start-of-body= Oct 30 21:11:43.545291 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:11:43.545125 5378 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w6-7eaf74-openshift-default-69c8f48765-r2cmt" podUID="1eb6161d-5e96-4dd5-bbe1-68300df5c425" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.145:15021/healthz/ready\": dial tcp 10.133.0.145:15021: connect: connection refused" Oct 30 21:11:44.544694 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:11:44.544655 5378 patch_prober.go:29] interesting pod/e2e-shared-b-w6-7eaf74-openshift-default-69c8f48765-r2cmt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.145:15021/healthz/ready\": dial tcp 10.133.0.145:15021: connect: connection refused" start-of-body= Oct 30 21:11:44.545090 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:11:44.544723 5378 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w6-7eaf74-openshift-default-69c8f48765-r2cmt" podUID="1eb6161d-5e96-4dd5-bbe1-68300df5c425" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.145:15021/healthz/ready\": dial tcp 10.133.0.145:15021: connect: connection refused" Oct 30 21:11:45.544578 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:11:45.544528 5378 patch_prober.go:29] interesting pod/e2e-shared-b-w6-7eaf74-openshift-default-69c8f48765-r2cmt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.145:15021/healthz/ready\": dial tcp 10.133.0.145:15021: connect: connection refused" start-of-body= Oct 30 21:11:45.544777 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:11:45.544620 5378 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w6-7eaf74-openshift-default-69c8f48765-r2cmt" podUID="1eb6161d-5e96-4dd5-bbe1-68300df5c425" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.145:15021/healthz/ready\": dial tcp 10.133.0.145:15021: connect: connection refused" Oct 30 21:11:46.544720 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:11:46.544690 5378 patch_prober.go:29] interesting pod/e2e-shared-b-w6-7eaf74-openshift-default-69c8f48765-r2cmt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.145:15021/healthz/ready\": dial tcp 10.133.0.145:15021: connect: connection refused" start-of-body= Oct 30 21:11:46.544971 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:11:46.544944 5378 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w6-7eaf74-openshift-default-69c8f48765-r2cmt" podUID="1eb6161d-5e96-4dd5-bbe1-68300df5c425" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.145:15021/healthz/ready\": dial tcp 10.133.0.145:15021: connect: connection refused" Oct 30 21:11:47.296114 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:11:47.296057 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:11:47.448594 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:11:47.443065 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:11:47.882003 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:11:47.881969 5378 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w6-7eaf74-openshift-default-69c8f48765-r2cmt" Oct 30 21:11:47.884473 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:11:47.884391 5378 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w6-7eaf74-openshift-default-69c8f48765-r2cmt" Oct 30 21:11:47.888898 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:11:47.888874 5378 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w6-7eaf74-openshift-default-69c8f48765-r2cmt" Oct 30 21:11:48.842581 ip-10-0-138-149 kubenswrapper[5378]: E1030 21:11:48.839113 5378 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q6mx7" containerName="extractor" Oct 30 21:11:52.353033 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:11:52.352985 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:11:53.273013 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:11:53.272977 5378 generic.go:356] "Generic (PLEG): container finished" podID="34b71d39-551a-4d2e-bf08-8df0cf45fc9e" containerID="62c77678f268f015b05bf2bc59bc3683ebf9d7164731bcbf27258a0c9b82483d" exitCode=0 Oct 30 21:11:53.273244 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:11:53.273054 5378 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-f3b172-openshift-default-d64ffcd45-k78ls" event={"ID":"34b71d39-551a-4d2e-bf08-8df0cf45fc9e","Type":"ContainerDied","Data":"62c77678f268f015b05bf2bc59bc3683ebf9d7164731bcbf27258a0c9b82483d"} Oct 30 21:11:53.273497 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:11:53.273483 5378 scope.go:122] "RemoveContainer" containerID="62c77678f268f015b05bf2bc59bc3683ebf9d7164731bcbf27258a0c9b82483d" Oct 30 21:11:53.275224 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:11:53.275197 5378 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 21:11:53.275303 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:11:53.275265 5378 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 21:11:53.275303 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:11:53.275297 5378 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 21:11:53.357534 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:11:53.357505 5378 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-f3b172-openshift-default-d64ffcd45-k78ls" Oct 30 21:11:54.287028 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:11:54.286995 5378 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-f3b172-openshift-default-d64ffcd45-k78ls" event={"ID":"34b71d39-551a-4d2e-bf08-8df0cf45fc9e","Type":"ContainerStarted","Data":"926cb2ae14c55ff798f1a2b06bfbafc9ed278b76db65445d73811db30e54fdec"} Oct 30 21:11:54.867590 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:11:54.865851 5378 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-f3b172-openshift-default-d64ffcd45-k78ls" Oct 30 21:11:54.870461 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:11:54.870141 5378 patch_prober.go:29] interesting pod/e2e-inf-a-f3b172-openshift-default-d64ffcd45-k78ls container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.139:15021/healthz/ready\": dial tcp 10.133.0.139:15021: connect: connection refused" start-of-body= Oct 30 21:11:54.870461 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:11:54.870193 5378 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-f3b172-openshift-default-d64ffcd45-k78ls" podUID="34b71d39-551a-4d2e-bf08-8df0cf45fc9e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.139:15021/healthz/ready\": dial tcp 10.133.0.139:15021: connect: connection refused" Oct 30 21:11:55.877691 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:11:55.877652 5378 patch_prober.go:29] interesting pod/e2e-inf-a-f3b172-openshift-default-d64ffcd45-k78ls container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.139:15021/healthz/ready\": dial tcp 10.133.0.139:15021: connect: connection refused" start-of-body= Oct 30 21:11:55.878194 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:11:55.877720 5378 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-f3b172-openshift-default-d64ffcd45-k78ls" podUID="34b71d39-551a-4d2e-bf08-8df0cf45fc9e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.139:15021/healthz/ready\": dial tcp 10.133.0.139:15021: connect: connection refused" Oct 30 21:11:56.878025 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:11:56.877984 5378 patch_prober.go:29] interesting pod/e2e-inf-a-f3b172-openshift-default-d64ffcd45-k78ls container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.139:15021/healthz/ready\": dial tcp 10.133.0.139:15021: connect: connection refused" start-of-body= Oct 30 21:11:56.878695 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:11:56.878641 5378 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-f3b172-openshift-default-d64ffcd45-k78ls" podUID="34b71d39-551a-4d2e-bf08-8df0cf45fc9e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.139:15021/healthz/ready\": dial tcp 10.133.0.139:15021: connect: connection refused" Oct 30 21:11:57.206889 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:11:57.206786 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:11:57.867903 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:11:57.867865 5378 patch_prober.go:29] interesting pod/e2e-inf-a-f3b172-openshift-default-d64ffcd45-k78ls container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.139:15021/healthz/ready\": dial tcp 10.133.0.139:15021: connect: connection refused" start-of-body= Oct 30 21:11:57.868173 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:11:57.868139 5378 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-f3b172-openshift-default-d64ffcd45-k78ls" podUID="34b71d39-551a-4d2e-bf08-8df0cf45fc9e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.139:15021/healthz/ready\": dial tcp 10.133.0.139:15021: connect: connection refused" Oct 30 21:11:58.875004 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:11:58.874551 5378 patch_prober.go:29] interesting pod/e2e-inf-a-f3b172-openshift-default-d64ffcd45-k78ls container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.139:15021/healthz/ready\": dial tcp 10.133.0.139:15021: connect: connection refused" start-of-body= Oct 30 21:11:58.875611 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:11:58.875558 5378 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-f3b172-openshift-default-d64ffcd45-k78ls" podUID="34b71d39-551a-4d2e-bf08-8df0cf45fc9e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.139:15021/healthz/ready\": dial tcp 10.133.0.139:15021: connect: connection refused" Oct 30 21:11:59.864994 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:11:59.864964 5378 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-f3b172-openshift-default-d64ffcd45-k78ls" Oct 30 21:11:59.865508 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:11:59.865483 5378 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-f3b172-openshift-default-d64ffcd45-k78ls" Oct 30 21:11:59.866258 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:11:59.866242 5378 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-f3b172-openshift-default-d64ffcd45-k78ls" Oct 30 21:12:02.001043 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:12:02.001010 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:12:06.779224 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:12:06.779195 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:12:07.350190 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:12:07.350158 5378 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-d061d0-openshift-default-86b87f495-rqk9z"] Oct 30 21:12:07.350414 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:12:07.350393 5378 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-d061d0-openshift-default-86b87f495-rqk9z" podUID="5aec1868-e81e-40f2-b955-29d704ba75b9" containerName="istio-proxy" containerID="cri-o://d802741d82742e989d600f7d722a185de016263597496297fa00dc928f183dff" gracePeriod=30 Oct 30 21:12:07.403012 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:12:07.402974 5378 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-d061d0-openshift-default-86b87f495-j9jnw"] Oct 30 21:12:07.403263 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:12:07.403229 5378 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-d061d0-openshift-default-86b87f495-j9jnw" podUID="81398598-ebe6-417f-91f7-90241e726083" containerName="istio-proxy" containerID="cri-o://a32847f6d47e11c6ad3d564e8628408756fe6a9c4b0467a1c0d45fb609c4b84d" gracePeriod=30 Oct 30 21:12:11.240862 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:12:11.240827 5378 patch_prober.go:29] interesting pod/e2e-worker-w2-d061d0-openshift-default-86b87f495-rqk9z container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 21:12:11.241310 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:12:11.240896 5378 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-d061d0-openshift-default-86b87f495-rqk9z" podUID="5aec1868-e81e-40f2-b955-29d704ba75b9" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 21:12:11.599139 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:12:11.599106 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:12:12.501576 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:12:12.501525 5378 generic.go:356] "Generic (PLEG): container finished" podID="5aec1868-e81e-40f2-b955-29d704ba75b9" containerID="d802741d82742e989d600f7d722a185de016263597496297fa00dc928f183dff" exitCode=0 Oct 30 21:12:12.501941 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:12:12.501594 5378 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-d061d0-openshift-default-86b87f495-rqk9z" event={"ID":"5aec1868-e81e-40f2-b955-29d704ba75b9","Type":"ContainerDied","Data":"d802741d82742e989d600f7d722a185de016263597496297fa00dc928f183dff"} Oct 30 21:12:12.716816 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:12:12.716789 5378 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-d061d0-openshift-default-86b87f495-rqk9z" Oct 30 21:12:12.737240 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:12:12.737215 5378 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-d061d0-openshift-default-86b87f495-j9jnw" Oct 30 21:12:12.805740 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:12:12.805711 5378 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5aec1868-e81e-40f2-b955-29d704ba75b9-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5aec1868-e81e-40f2-b955-29d704ba75b9-workload-certs\") pod \"5aec1868-e81e-40f2-b955-29d704ba75b9\" (UID: \"5aec1868-e81e-40f2-b955-29d704ba75b9\") " Oct 30 21:12:12.805896 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:12:12.805779 5378 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5aec1868-e81e-40f2-b955-29d704ba75b9-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5aec1868-e81e-40f2-b955-29d704ba75b9-istiod-ca-cert\") pod \"5aec1868-e81e-40f2-b955-29d704ba75b9\" (UID: \"5aec1868-e81e-40f2-b955-29d704ba75b9\") " Oct 30 21:12:12.805896 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:12:12.805878 5378 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5aec1868-e81e-40f2-b955-29d704ba75b9-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5aec1868-e81e-40f2-b955-29d704ba75b9-workload-socket\") pod \"5aec1868-e81e-40f2-b955-29d704ba75b9\" (UID: \"5aec1868-e81e-40f2-b955-29d704ba75b9\") " Oct 30 21:12:12.805991 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:12:12.805908 5378 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/5aec1868-e81e-40f2-b955-29d704ba75b9-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5aec1868-e81e-40f2-b955-29d704ba75b9-istio-podinfo\") pod \"5aec1868-e81e-40f2-b955-29d704ba75b9\" (UID: \"5aec1868-e81e-40f2-b955-29d704ba75b9\") " Oct 30 21:12:12.806043 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:12:12.806010 5378 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5aec1868-e81e-40f2-b955-29d704ba75b9-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5aec1868-e81e-40f2-b955-29d704ba75b9-credential-socket\") pod \"5aec1868-e81e-40f2-b955-29d704ba75b9\" (UID: \"5aec1868-e81e-40f2-b955-29d704ba75b9\") " Oct 30 21:12:12.806108 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:12:12.806064 5378 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5aec1868-e81e-40f2-b955-29d704ba75b9-workload-certs" pod "5aec1868-e81e-40f2-b955-29d704ba75b9" (UID: "5aec1868-e81e-40f2-b955-29d704ba75b9"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:12:12.806108 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:12:12.806088 5378 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5aec1868-e81e-40f2-b955-29d704ba75b9-istio-token\" (UniqueName: \"kubernetes.io/projected/5aec1868-e81e-40f2-b955-29d704ba75b9-istio-token\") pod \"5aec1868-e81e-40f2-b955-29d704ba75b9\" (UID: \"5aec1868-e81e-40f2-b955-29d704ba75b9\") " Oct 30 21:12:12.806215 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:12:12.806137 5378 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5aec1868-e81e-40f2-b955-29d704ba75b9-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5aec1868-e81e-40f2-b955-29d704ba75b9-istio-data\") pod \"5aec1868-e81e-40f2-b955-29d704ba75b9\" (UID: \"5aec1868-e81e-40f2-b955-29d704ba75b9\") " Oct 30 21:12:12.806215 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:12:12.806196 5378 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5aec1868-e81e-40f2-b955-29d704ba75b9-workload-socket" pod "5aec1868-e81e-40f2-b955-29d704ba75b9" (UID: "5aec1868-e81e-40f2-b955-29d704ba75b9"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:12:12.806215 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:12:12.806121 5378 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5aec1868-e81e-40f2-b955-29d704ba75b9-istiod-ca-cert" pod "5aec1868-e81e-40f2-b955-29d704ba75b9" (UID: "5aec1868-e81e-40f2-b955-29d704ba75b9"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:12:12.806381 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:12:12.806275 5378 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5aec1868-e81e-40f2-b955-29d704ba75b9-credential-socket" pod "5aec1868-e81e-40f2-b955-29d704ba75b9" (UID: "5aec1868-e81e-40f2-b955-29d704ba75b9"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:12:12.806381 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:12:12.806331 5378 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5aec1868-e81e-40f2-b955-29d704ba75b9-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5aec1868-e81e-40f2-b955-29d704ba75b9-istio-envoy\") pod \"5aec1868-e81e-40f2-b955-29d704ba75b9\" (UID: \"5aec1868-e81e-40f2-b955-29d704ba75b9\") " Oct 30 21:12:12.806381 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:12:12.806366 5378 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5aec1868-e81e-40f2-b955-29d704ba75b9-kube-api-access-dhc57\" (UniqueName: \"kubernetes.io/projected/5aec1868-e81e-40f2-b955-29d704ba75b9-kube-api-access-dhc57\") pod \"5aec1868-e81e-40f2-b955-29d704ba75b9\" (UID: \"5aec1868-e81e-40f2-b955-29d704ba75b9\") " Oct 30 21:12:12.806533 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:12:12.806398 5378 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5aec1868-e81e-40f2-b955-29d704ba75b9-istio-data" pod "5aec1868-e81e-40f2-b955-29d704ba75b9" (UID: "5aec1868-e81e-40f2-b955-29d704ba75b9"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:12:12.807097 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:12:12.807074 5378 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5aec1868-e81e-40f2-b955-29d704ba75b9-credential-socket\") on node \"ip-10-0-138-149.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:12:12.807218 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:12:12.807099 5378 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5aec1868-e81e-40f2-b955-29d704ba75b9-istio-data\") on node \"ip-10-0-138-149.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:12:12.807218 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:12:12.807112 5378 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5aec1868-e81e-40f2-b955-29d704ba75b9-workload-certs\") on node \"ip-10-0-138-149.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:12:12.807218 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:12:12.807125 5378 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5aec1868-e81e-40f2-b955-29d704ba75b9-istiod-ca-cert\") on node \"ip-10-0-138-149.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:12:12.807218 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:12:12.807139 5378 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5aec1868-e81e-40f2-b955-29d704ba75b9-workload-socket\") on node \"ip-10-0-138-149.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:12:12.808494 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:12:12.808474 5378 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5aec1868-e81e-40f2-b955-29d704ba75b9-kube-api-access-dhc57" pod "5aec1868-e81e-40f2-b955-29d704ba75b9" (UID: "5aec1868-e81e-40f2-b955-29d704ba75b9"). InnerVolumeSpecName "kube-api-access-dhc57". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:12:12.808695 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:12:12.808677 5378 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/5aec1868-e81e-40f2-b955-29d704ba75b9-istio-podinfo" pod "5aec1868-e81e-40f2-b955-29d704ba75b9" (UID: "5aec1868-e81e-40f2-b955-29d704ba75b9"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 21:12:12.808815 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:12:12.808796 5378 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5aec1868-e81e-40f2-b955-29d704ba75b9-istio-envoy" pod "5aec1868-e81e-40f2-b955-29d704ba75b9" (UID: "5aec1868-e81e-40f2-b955-29d704ba75b9"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:12:12.808815 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:12:12.808804 5378 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5aec1868-e81e-40f2-b955-29d704ba75b9-istio-token" pod "5aec1868-e81e-40f2-b955-29d704ba75b9" (UID: "5aec1868-e81e-40f2-b955-29d704ba75b9"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:12:12.908202 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:12:12.908165 5378 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/81398598-ebe6-417f-91f7-90241e726083-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/81398598-ebe6-417f-91f7-90241e726083-istio-data\") pod \"81398598-ebe6-417f-91f7-90241e726083\" (UID: \"81398598-ebe6-417f-91f7-90241e726083\") " Oct 30 21:12:12.908202 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:12:12.908199 5378 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/81398598-ebe6-417f-91f7-90241e726083-istio-token\" (UniqueName: \"kubernetes.io/projected/81398598-ebe6-417f-91f7-90241e726083-istio-token\") pod \"81398598-ebe6-417f-91f7-90241e726083\" (UID: \"81398598-ebe6-417f-91f7-90241e726083\") " Oct 30 21:12:12.908430 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:12:12.908235 5378 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/81398598-ebe6-417f-91f7-90241e726083-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/81398598-ebe6-417f-91f7-90241e726083-istiod-ca-cert\") pod \"81398598-ebe6-417f-91f7-90241e726083\" (UID: \"81398598-ebe6-417f-91f7-90241e726083\") " Oct 30 21:12:12.908430 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:12:12.908347 5378 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/81398598-ebe6-417f-91f7-90241e726083-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/81398598-ebe6-417f-91f7-90241e726083-istio-envoy\") pod \"81398598-ebe6-417f-91f7-90241e726083\" (UID: \"81398598-ebe6-417f-91f7-90241e726083\") " Oct 30 21:12:12.908430 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:12:12.908422 5378 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/81398598-ebe6-417f-91f7-90241e726083-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/81398598-ebe6-417f-91f7-90241e726083-credential-socket\") pod \"81398598-ebe6-417f-91f7-90241e726083\" (UID: \"81398598-ebe6-417f-91f7-90241e726083\") " Oct 30 21:12:12.908621 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:12:12.908453 5378 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/81398598-ebe6-417f-91f7-90241e726083-istio-data" pod "81398598-ebe6-417f-91f7-90241e726083" (UID: "81398598-ebe6-417f-91f7-90241e726083"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:12:12.908621 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:12:12.908547 5378 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/81398598-ebe6-417f-91f7-90241e726083-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/81398598-ebe6-417f-91f7-90241e726083-istio-podinfo\") pod \"81398598-ebe6-417f-91f7-90241e726083\" (UID: \"81398598-ebe6-417f-91f7-90241e726083\") " Oct 30 21:12:12.908735 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:12:12.908630 5378 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/81398598-ebe6-417f-91f7-90241e726083-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/81398598-ebe6-417f-91f7-90241e726083-workload-certs\") pod \"81398598-ebe6-417f-91f7-90241e726083\" (UID: \"81398598-ebe6-417f-91f7-90241e726083\") " Oct 30 21:12:12.908735 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:12:12.908664 5378 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/81398598-ebe6-417f-91f7-90241e726083-credential-socket" pod "81398598-ebe6-417f-91f7-90241e726083" (UID: "81398598-ebe6-417f-91f7-90241e726083"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:12:12.908735 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:12:12.908697 5378 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/81398598-ebe6-417f-91f7-90241e726083-kube-api-access-85qn4\" (UniqueName: \"kubernetes.io/projected/81398598-ebe6-417f-91f7-90241e726083-kube-api-access-85qn4\") pod \"81398598-ebe6-417f-91f7-90241e726083\" (UID: \"81398598-ebe6-417f-91f7-90241e726083\") " Oct 30 21:12:12.908897 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:12:12.908757 5378 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/81398598-ebe6-417f-91f7-90241e726083-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/81398598-ebe6-417f-91f7-90241e726083-workload-socket\") pod \"81398598-ebe6-417f-91f7-90241e726083\" (UID: \"81398598-ebe6-417f-91f7-90241e726083\") " Oct 30 21:12:12.908950 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:12:12.908897 5378 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/81398598-ebe6-417f-91f7-90241e726083-istiod-ca-cert" pod "81398598-ebe6-417f-91f7-90241e726083" (UID: "81398598-ebe6-417f-91f7-90241e726083"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:12:12.909185 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:12:12.909163 5378 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/81398598-ebe6-417f-91f7-90241e726083-workload-socket" pod "81398598-ebe6-417f-91f7-90241e726083" (UID: "81398598-ebe6-417f-91f7-90241e726083"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:12:12.909286 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:12:12.909191 5378 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/81398598-ebe6-417f-91f7-90241e726083-workload-certs" pod "81398598-ebe6-417f-91f7-90241e726083" (UID: "81398598-ebe6-417f-91f7-90241e726083"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:12:12.909548 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:12:12.909529 5378 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5aec1868-e81e-40f2-b955-29d704ba75b9-istio-envoy\") on node \"ip-10-0-138-149.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:12:12.909673 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:12:12.909554 5378 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dhc57\" (UniqueName: \"kubernetes.io/projected/5aec1868-e81e-40f2-b955-29d704ba75b9-kube-api-access-dhc57\") on node \"ip-10-0-138-149.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:12:12.909673 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:12:12.909607 5378 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/81398598-ebe6-417f-91f7-90241e726083-credential-socket\") on node \"ip-10-0-138-149.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:12:12.909673 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:12:12.909621 5378 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/81398598-ebe6-417f-91f7-90241e726083-workload-certs\") on node \"ip-10-0-138-149.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:12:12.909673 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:12:12.909634 5378 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/81398598-ebe6-417f-91f7-90241e726083-workload-socket\") on node \"ip-10-0-138-149.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:12:12.909673 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:12:12.909649 5378 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5aec1868-e81e-40f2-b955-29d704ba75b9-istio-podinfo\") on node \"ip-10-0-138-149.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:12:12.909673 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:12:12.909662 5378 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5aec1868-e81e-40f2-b955-29d704ba75b9-istio-token\") on node \"ip-10-0-138-149.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:12:12.909888 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:12:12.909676 5378 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/81398598-ebe6-417f-91f7-90241e726083-istio-data\") on node \"ip-10-0-138-149.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:12:12.909888 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:12:12.909692 5378 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/81398598-ebe6-417f-91f7-90241e726083-istiod-ca-cert\") on node \"ip-10-0-138-149.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:12:12.910833 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:12:12.910810 5378 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/81398598-ebe6-417f-91f7-90241e726083-istio-token" pod "81398598-ebe6-417f-91f7-90241e726083" (UID: "81398598-ebe6-417f-91f7-90241e726083"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:12:12.911038 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:12:12.911004 5378 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/81398598-ebe6-417f-91f7-90241e726083-istio-envoy" pod "81398598-ebe6-417f-91f7-90241e726083" (UID: "81398598-ebe6-417f-91f7-90241e726083"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:12:12.911181 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:12:12.911162 5378 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/81398598-ebe6-417f-91f7-90241e726083-istio-podinfo" pod "81398598-ebe6-417f-91f7-90241e726083" (UID: "81398598-ebe6-417f-91f7-90241e726083"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 21:12:12.911797 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:12:12.911336 5378 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/81398598-ebe6-417f-91f7-90241e726083-kube-api-access-85qn4" pod "81398598-ebe6-417f-91f7-90241e726083" (UID: "81398598-ebe6-417f-91f7-90241e726083"). InnerVolumeSpecName "kube-api-access-85qn4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:12:13.012354 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:12:13.011993 5378 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/81398598-ebe6-417f-91f7-90241e726083-istio-podinfo\") on node \"ip-10-0-138-149.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:12:13.012354 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:12:13.012031 5378 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-85qn4\" (UniqueName: \"kubernetes.io/projected/81398598-ebe6-417f-91f7-90241e726083-kube-api-access-85qn4\") on node \"ip-10-0-138-149.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:12:13.012354 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:12:13.012046 5378 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/81398598-ebe6-417f-91f7-90241e726083-istio-token\") on node \"ip-10-0-138-149.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:12:13.012354 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:12:13.012063 5378 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/81398598-ebe6-417f-91f7-90241e726083-istio-envoy\") on node \"ip-10-0-138-149.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:12:13.510920 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:12:13.510886 5378 generic.go:356] "Generic (PLEG): container finished" podID="81398598-ebe6-417f-91f7-90241e726083" containerID="a32847f6d47e11c6ad3d564e8628408756fe6a9c4b0467a1c0d45fb609c4b84d" exitCode=0 Oct 30 21:12:13.511375 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:12:13.510955 5378 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-d061d0-openshift-default-86b87f495-j9jnw" Oct 30 21:12:13.511375 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:12:13.510965 5378 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-d061d0-openshift-default-86b87f495-j9jnw" event={"ID":"81398598-ebe6-417f-91f7-90241e726083","Type":"ContainerDied","Data":"a32847f6d47e11c6ad3d564e8628408756fe6a9c4b0467a1c0d45fb609c4b84d"} Oct 30 21:12:13.511375 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:12:13.511031 5378 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-d061d0-openshift-default-86b87f495-j9jnw" event={"ID":"81398598-ebe6-417f-91f7-90241e726083","Type":"ContainerDied","Data":"3365e7dde15529f84bb7dab346cf2f0b40eed53428a2d1459c637d2e43a98560"} Oct 30 21:12:13.511375 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:12:13.511061 5378 scope.go:122] "RemoveContainer" containerID="a32847f6d47e11c6ad3d564e8628408756fe6a9c4b0467a1c0d45fb609c4b84d" Oct 30 21:12:13.512981 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:12:13.512946 5378 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-d061d0-openshift-default-86b87f495-rqk9z" Oct 30 21:12:13.513094 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:12:13.512936 5378 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-d061d0-openshift-default-86b87f495-rqk9z" event={"ID":"5aec1868-e81e-40f2-b955-29d704ba75b9","Type":"ContainerDied","Data":"643be13243800a8a1084cbdb7d831abb05933c451fde5442955265f0b1b839bd"} Oct 30 21:12:13.524313 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:12:13.524292 5378 scope.go:122] "RemoveContainer" containerID="a32847f6d47e11c6ad3d564e8628408756fe6a9c4b0467a1c0d45fb609c4b84d" Oct 30 21:12:13.524614 ip-10-0-138-149 kubenswrapper[5378]: E1030 21:12:13.524592 5378 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a32847f6d47e11c6ad3d564e8628408756fe6a9c4b0467a1c0d45fb609c4b84d\": container with ID starting with a32847f6d47e11c6ad3d564e8628408756fe6a9c4b0467a1c0d45fb609c4b84d not found: ID does not exist" containerID="a32847f6d47e11c6ad3d564e8628408756fe6a9c4b0467a1c0d45fb609c4b84d" Oct 30 21:12:13.524729 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:12:13.524618 5378 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a32847f6d47e11c6ad3d564e8628408756fe6a9c4b0467a1c0d45fb609c4b84d"} err="failed to get container status \"a32847f6d47e11c6ad3d564e8628408756fe6a9c4b0467a1c0d45fb609c4b84d\": rpc error: code = NotFound desc = could not find container \"a32847f6d47e11c6ad3d564e8628408756fe6a9c4b0467a1c0d45fb609c4b84d\": container with ID starting with a32847f6d47e11c6ad3d564e8628408756fe6a9c4b0467a1c0d45fb609c4b84d not found: ID does not exist" Oct 30 21:12:13.524729 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:12:13.524637 5378 scope.go:122] "RemoveContainer" containerID="d802741d82742e989d600f7d722a185de016263597496297fa00dc928f183dff" Oct 30 21:12:13.561864 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:12:13.561832 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:12:13.679994 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:12:13.679916 5378 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-d061d0-openshift-default-86b87f495-j9jnw"] Oct 30 21:12:13.730349 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:12:13.730309 5378 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-d061d0-openshift-default-86b87f495-j9jnw"] Oct 30 21:12:13.746678 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:12:13.746650 5378 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="81398598-ebe6-417f-91f7-90241e726083" path="/var/lib/kubelet/pods/81398598-ebe6-417f-91f7-90241e726083/volumes" Oct 30 21:12:13.846183 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:12:13.846152 5378 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-d061d0-openshift-default-86b87f495-rqk9z"] Oct 30 21:12:13.853724 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:12:13.853684 5378 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-d061d0-openshift-default-86b87f495-rqk9z"] Oct 30 21:12:14.959468 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:12:14.959430 5378 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-390b71-openshift-default-df8b44c8c-5k4nj"] Oct 30 21:12:14.960253 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:12:14.960235 5378 cpu_manager.go:414] "containerMap: removing container" podUID="81398598-ebe6-417f-91f7-90241e726083" containerName="istio-proxy" Oct 30 21:12:14.960299 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:12:14.960254 5378 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="81398598-ebe6-417f-91f7-90241e726083" containerName="istio-proxy" Oct 30 21:12:14.960299 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:12:14.960276 5378 cpu_manager.go:414] "containerMap: removing container" podUID="5aec1868-e81e-40f2-b955-29d704ba75b9" containerName="istio-proxy" Oct 30 21:12:14.960299 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:12:14.960281 5378 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5aec1868-e81e-40f2-b955-29d704ba75b9" containerName="istio-proxy" Oct 30 21:12:14.960410 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:12:14.960384 5378 memory_manager.go:365] "RemoveStaleState removing state" podUID="81398598-ebe6-417f-91f7-90241e726083" containerName="istio-proxy" Oct 30 21:12:14.960410 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:12:14.960394 5378 memory_manager.go:365] "RemoveStaleState removing state" podUID="5aec1868-e81e-40f2-b955-29d704ba75b9" containerName="istio-proxy" Oct 30 21:12:14.964100 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:12:14.964079 5378 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-390b71-openshift-default-df8b44c8c-5k4nj" Oct 30 21:12:15.027851 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:12:15.027816 5378 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-390b71-openshift-default-dockercfg-wqn8x\"" Oct 30 21:12:15.051641 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:12:15.051604 5378 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dd395160-55d3-4dfb-a2b1-6e68c62f32b5-istiod-ca-cert\") pod \"e2e-inf-a-390b71-openshift-default-df8b44c8c-5k4nj\" (UID: \"dd395160-55d3-4dfb-a2b1-6e68c62f32b5\") " pod="openshift-ingress/e2e-inf-a-390b71-openshift-default-df8b44c8c-5k4nj" Oct 30 21:12:15.051866 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:12:15.051817 5378 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9c7fs\" (UniqueName: \"kubernetes.io/projected/dd395160-55d3-4dfb-a2b1-6e68c62f32b5-kube-api-access-9c7fs\") pod \"e2e-inf-a-390b71-openshift-default-df8b44c8c-5k4nj\" (UID: \"dd395160-55d3-4dfb-a2b1-6e68c62f32b5\") " pod="openshift-ingress/e2e-inf-a-390b71-openshift-default-df8b44c8c-5k4nj" Oct 30 21:12:15.051952 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:12:15.051879 5378 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dd395160-55d3-4dfb-a2b1-6e68c62f32b5-istio-data\") pod \"e2e-inf-a-390b71-openshift-default-df8b44c8c-5k4nj\" (UID: \"dd395160-55d3-4dfb-a2b1-6e68c62f32b5\") " pod="openshift-ingress/e2e-inf-a-390b71-openshift-default-df8b44c8c-5k4nj" Oct 30 21:12:15.051952 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:12:15.051918 5378 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dd395160-55d3-4dfb-a2b1-6e68c62f32b5-istio-podinfo\") pod \"e2e-inf-a-390b71-openshift-default-df8b44c8c-5k4nj\" (UID: \"dd395160-55d3-4dfb-a2b1-6e68c62f32b5\") " pod="openshift-ingress/e2e-inf-a-390b71-openshift-default-df8b44c8c-5k4nj" Oct 30 21:12:15.052088 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:12:15.052067 5378 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dd395160-55d3-4dfb-a2b1-6e68c62f32b5-workload-socket\") pod \"e2e-inf-a-390b71-openshift-default-df8b44c8c-5k4nj\" (UID: \"dd395160-55d3-4dfb-a2b1-6e68c62f32b5\") " pod="openshift-ingress/e2e-inf-a-390b71-openshift-default-df8b44c8c-5k4nj" Oct 30 21:12:15.052152 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:12:15.052131 5378 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dd395160-55d3-4dfb-a2b1-6e68c62f32b5-istio-token\") pod \"e2e-inf-a-390b71-openshift-default-df8b44c8c-5k4nj\" (UID: \"dd395160-55d3-4dfb-a2b1-6e68c62f32b5\") " pod="openshift-ingress/e2e-inf-a-390b71-openshift-default-df8b44c8c-5k4nj" Oct 30 21:12:15.052213 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:12:15.052194 5378 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dd395160-55d3-4dfb-a2b1-6e68c62f32b5-workload-certs\") pod \"e2e-inf-a-390b71-openshift-default-df8b44c8c-5k4nj\" (UID: \"dd395160-55d3-4dfb-a2b1-6e68c62f32b5\") " pod="openshift-ingress/e2e-inf-a-390b71-openshift-default-df8b44c8c-5k4nj" Oct 30 21:12:15.052374 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:12:15.052356 5378 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dd395160-55d3-4dfb-a2b1-6e68c62f32b5-istio-envoy\") pod \"e2e-inf-a-390b71-openshift-default-df8b44c8c-5k4nj\" (UID: \"dd395160-55d3-4dfb-a2b1-6e68c62f32b5\") " pod="openshift-ingress/e2e-inf-a-390b71-openshift-default-df8b44c8c-5k4nj" Oct 30 21:12:15.052457 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:12:15.052442 5378 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dd395160-55d3-4dfb-a2b1-6e68c62f32b5-credential-socket\") pod \"e2e-inf-a-390b71-openshift-default-df8b44c8c-5k4nj\" (UID: \"dd395160-55d3-4dfb-a2b1-6e68c62f32b5\") " pod="openshift-ingress/e2e-inf-a-390b71-openshift-default-df8b44c8c-5k4nj" Oct 30 21:12:15.076360 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:12:15.076322 5378 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-390b71-openshift-default-df8b44c8c-5k4nj"] Oct 30 21:12:15.095714 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:12:15.095539 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:12:15.168711 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:12:15.166870 5378 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dd395160-55d3-4dfb-a2b1-6e68c62f32b5-workload-certs\") pod \"e2e-inf-a-390b71-openshift-default-df8b44c8c-5k4nj\" (UID: \"dd395160-55d3-4dfb-a2b1-6e68c62f32b5\") " pod="openshift-ingress/e2e-inf-a-390b71-openshift-default-df8b44c8c-5k4nj" Oct 30 21:12:15.168711 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:12:15.167083 5378 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dd395160-55d3-4dfb-a2b1-6e68c62f32b5-istio-envoy\") pod \"e2e-inf-a-390b71-openshift-default-df8b44c8c-5k4nj\" (UID: \"dd395160-55d3-4dfb-a2b1-6e68c62f32b5\") " pod="openshift-ingress/e2e-inf-a-390b71-openshift-default-df8b44c8c-5k4nj" Oct 30 21:12:15.168711 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:12:15.167176 5378 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dd395160-55d3-4dfb-a2b1-6e68c62f32b5-credential-socket\") pod \"e2e-inf-a-390b71-openshift-default-df8b44c8c-5k4nj\" (UID: \"dd395160-55d3-4dfb-a2b1-6e68c62f32b5\") " pod="openshift-ingress/e2e-inf-a-390b71-openshift-default-df8b44c8c-5k4nj" Oct 30 21:12:15.168711 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:12:15.167217 5378 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dd395160-55d3-4dfb-a2b1-6e68c62f32b5-istiod-ca-cert\") pod \"e2e-inf-a-390b71-openshift-default-df8b44c8c-5k4nj\" (UID: \"dd395160-55d3-4dfb-a2b1-6e68c62f32b5\") " pod="openshift-ingress/e2e-inf-a-390b71-openshift-default-df8b44c8c-5k4nj" Oct 30 21:12:15.168711 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:12:15.167321 5378 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9c7fs\" (UniqueName: \"kubernetes.io/projected/dd395160-55d3-4dfb-a2b1-6e68c62f32b5-kube-api-access-9c7fs\") pod \"e2e-inf-a-390b71-openshift-default-df8b44c8c-5k4nj\" (UID: \"dd395160-55d3-4dfb-a2b1-6e68c62f32b5\") " pod="openshift-ingress/e2e-inf-a-390b71-openshift-default-df8b44c8c-5k4nj" Oct 30 21:12:15.168711 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:12:15.167358 5378 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dd395160-55d3-4dfb-a2b1-6e68c62f32b5-istio-data\") pod \"e2e-inf-a-390b71-openshift-default-df8b44c8c-5k4nj\" (UID: \"dd395160-55d3-4dfb-a2b1-6e68c62f32b5\") " pod="openshift-ingress/e2e-inf-a-390b71-openshift-default-df8b44c8c-5k4nj" Oct 30 21:12:15.168711 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:12:15.167395 5378 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dd395160-55d3-4dfb-a2b1-6e68c62f32b5-istio-podinfo\") pod \"e2e-inf-a-390b71-openshift-default-df8b44c8c-5k4nj\" (UID: \"dd395160-55d3-4dfb-a2b1-6e68c62f32b5\") " pod="openshift-ingress/e2e-inf-a-390b71-openshift-default-df8b44c8c-5k4nj" Oct 30 21:12:15.168711 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:12:15.167462 5378 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dd395160-55d3-4dfb-a2b1-6e68c62f32b5-workload-socket\") pod \"e2e-inf-a-390b71-openshift-default-df8b44c8c-5k4nj\" (UID: \"dd395160-55d3-4dfb-a2b1-6e68c62f32b5\") " pod="openshift-ingress/e2e-inf-a-390b71-openshift-default-df8b44c8c-5k4nj" Oct 30 21:12:15.168711 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:12:15.167548 5378 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dd395160-55d3-4dfb-a2b1-6e68c62f32b5-istio-token\") pod \"e2e-inf-a-390b71-openshift-default-df8b44c8c-5k4nj\" (UID: \"dd395160-55d3-4dfb-a2b1-6e68c62f32b5\") " pod="openshift-ingress/e2e-inf-a-390b71-openshift-default-df8b44c8c-5k4nj" Oct 30 21:12:15.186524 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:12:15.170721 5378 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dd395160-55d3-4dfb-a2b1-6e68c62f32b5-istiod-ca-cert\") pod \"e2e-inf-a-390b71-openshift-default-df8b44c8c-5k4nj\" (UID: \"dd395160-55d3-4dfb-a2b1-6e68c62f32b5\") " pod="openshift-ingress/e2e-inf-a-390b71-openshift-default-df8b44c8c-5k4nj" Oct 30 21:12:15.186930 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:12:15.177822 5378 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dd395160-55d3-4dfb-a2b1-6e68c62f32b5-workload-certs\") pod \"e2e-inf-a-390b71-openshift-default-df8b44c8c-5k4nj\" (UID: \"dd395160-55d3-4dfb-a2b1-6e68c62f32b5\") " pod="openshift-ingress/e2e-inf-a-390b71-openshift-default-df8b44c8c-5k4nj" Oct 30 21:12:15.187081 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:12:15.178769 5378 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dd395160-55d3-4dfb-a2b1-6e68c62f32b5-credential-socket\") pod \"e2e-inf-a-390b71-openshift-default-df8b44c8c-5k4nj\" (UID: \"dd395160-55d3-4dfb-a2b1-6e68c62f32b5\") " pod="openshift-ingress/e2e-inf-a-390b71-openshift-default-df8b44c8c-5k4nj" Oct 30 21:12:15.187195 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:12:15.181323 5378 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dd395160-55d3-4dfb-a2b1-6e68c62f32b5-workload-socket\") pod \"e2e-inf-a-390b71-openshift-default-df8b44c8c-5k4nj\" (UID: \"dd395160-55d3-4dfb-a2b1-6e68c62f32b5\") " pod="openshift-ingress/e2e-inf-a-390b71-openshift-default-df8b44c8c-5k4nj" Oct 30 21:12:15.188866 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:12:15.188815 5378 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dd395160-55d3-4dfb-a2b1-6e68c62f32b5-istio-data\") pod \"e2e-inf-a-390b71-openshift-default-df8b44c8c-5k4nj\" (UID: \"dd395160-55d3-4dfb-a2b1-6e68c62f32b5\") " pod="openshift-ingress/e2e-inf-a-390b71-openshift-default-df8b44c8c-5k4nj" Oct 30 21:12:15.210226 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:12:15.210144 5378 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dd395160-55d3-4dfb-a2b1-6e68c62f32b5-istio-envoy\") pod \"e2e-inf-a-390b71-openshift-default-df8b44c8c-5k4nj\" (UID: \"dd395160-55d3-4dfb-a2b1-6e68c62f32b5\") " pod="openshift-ingress/e2e-inf-a-390b71-openshift-default-df8b44c8c-5k4nj" Oct 30 21:12:15.214168 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:12:15.214135 5378 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dd395160-55d3-4dfb-a2b1-6e68c62f32b5-istio-podinfo\") pod \"e2e-inf-a-390b71-openshift-default-df8b44c8c-5k4nj\" (UID: \"dd395160-55d3-4dfb-a2b1-6e68c62f32b5\") " pod="openshift-ingress/e2e-inf-a-390b71-openshift-default-df8b44c8c-5k4nj" Oct 30 21:12:15.234283 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:12:15.234253 5378 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dd395160-55d3-4dfb-a2b1-6e68c62f32b5-istio-token\") pod \"e2e-inf-a-390b71-openshift-default-df8b44c8c-5k4nj\" (UID: \"dd395160-55d3-4dfb-a2b1-6e68c62f32b5\") " pod="openshift-ingress/e2e-inf-a-390b71-openshift-default-df8b44c8c-5k4nj" Oct 30 21:12:15.234491 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:12:15.234472 5378 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9c7fs\" (UniqueName: \"kubernetes.io/projected/dd395160-55d3-4dfb-a2b1-6e68c62f32b5-kube-api-access-9c7fs\") pod \"e2e-inf-a-390b71-openshift-default-df8b44c8c-5k4nj\" (UID: \"dd395160-55d3-4dfb-a2b1-6e68c62f32b5\") " pod="openshift-ingress/e2e-inf-a-390b71-openshift-default-df8b44c8c-5k4nj" Oct 30 21:12:15.257040 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:12:15.256865 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:12:15.289586 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:12:15.283866 5378 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-390b71-openshift-default-df8b44c8c-5k4nj" Oct 30 21:12:15.747994 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:12:15.747943 5378 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 21:12:15.748135 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:12:15.748086 5378 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 21:12:15.748199 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:12:15.748175 5378 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 21:12:15.753534 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:12:15.753506 5378 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5aec1868-e81e-40f2-b955-29d704ba75b9" path="/var/lib/kubelet/pods/5aec1868-e81e-40f2-b955-29d704ba75b9/volumes" Oct 30 21:12:15.753997 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:12:15.753980 5378 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-390b71-openshift-default-df8b44c8c-5k4nj"] Oct 30 21:12:16.540709 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:12:16.540675 5378 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-390b71-openshift-default-df8b44c8c-5k4nj" event={"ID":"dd395160-55d3-4dfb-a2b1-6e68c62f32b5","Type":"ContainerStarted","Data":"28e80945bafbdf729da98e7b0864087040d0ba4695c109e85aebe64751bebafd"} Oct 30 21:12:16.541090 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:12:16.540725 5378 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-390b71-openshift-default-df8b44c8c-5k4nj" event={"ID":"dd395160-55d3-4dfb-a2b1-6e68c62f32b5","Type":"ContainerStarted","Data":"31b939607c7f5288ea403e221a9a3c6fed3ad68ce72306f58b77af1affacfbe1"} Oct 30 21:12:16.713121 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:12:16.713058 5378 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-390b71-openshift-default-df8b44c8c-5k4nj" podStartSLOduration=2.71304212 podStartE2EDuration="2.71304212s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:12:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:12:16.709000044 +0000 UTC m=+2027.137113966" watchObservedRunningTime="2026-10-30 21:12:16.71304212 +0000 UTC m=+2027.141156042" Oct 30 21:12:16.851236 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:12:16.851200 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:12:17.290584 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:12:17.290494 5378 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-390b71-openshift-default-df8b44c8c-5k4nj" Oct 30 21:12:17.296436 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:12:17.296410 5378 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-390b71-openshift-default-df8b44c8c-5k4nj" Oct 30 21:12:17.547898 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:12:17.547814 5378 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-390b71-openshift-default-df8b44c8c-5k4nj" Oct 30 21:12:17.548680 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:12:17.548661 5378 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-390b71-openshift-default-df8b44c8c-5k4nj" Oct 30 21:12:18.837631 ip-10-0-138-149 kubenswrapper[5378]: E1030 21:12:18.837590 5378 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q6mx7" containerName="extractor" Oct 30 21:12:20.694814 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:12:20.694763 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:12:20.923913 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:12:20.923882 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:12:21.063964 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:12:21.063876 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:12:22.159014 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:12:22.158969 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:12:22.300467 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:12:22.300433 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:12:22.420641 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:12:22.411778 5378 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-1fdaa7-openshift-default-84586db7d5-2svf5"] Oct 30 21:12:22.420641 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:12:22.412047 5378 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-1fdaa7-openshift-default-84586db7d5-2svf5" podUID="44227101-a97a-4bc9-a5b0-fd32cc961337" containerName="istio-proxy" containerID="cri-o://829c7b38cd116d38cc877ae98afebbc0baa8496ea21c8dc8ce8e1e43760ced3b" gracePeriod=30 Oct 30 21:12:22.441802 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:12:22.441766 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:12:25.321897 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:12:25.319832 5378 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w6-dc6125-6644bb494b-2v578"] Oct 30 21:12:25.427157 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:12:25.427125 5378 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w6-dc6125-74d649bf67-vlsfb"] Oct 30 21:12:25.679634 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:12:25.679556 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:12:25.847878 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:12:25.846750 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:12:27.515189 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:12:27.515158 5378 patch_prober.go:29] interesting pod/e2e-worker-w4-1fdaa7-openshift-default-84586db7d5-2svf5 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.98:15021/healthz/ready\": dial tcp 10.133.0.98:15021: connect: connection refused" start-of-body= Oct 30 21:12:27.515551 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:12:27.515212 5378 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-1fdaa7-openshift-default-84586db7d5-2svf5" podUID="44227101-a97a-4bc9-a5b0-fd32cc961337" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.98:15021/healthz/ready\": dial tcp 10.133.0.98:15021: connect: connection refused" Oct 30 21:12:27.654677 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:12:27.654646 5378 generic.go:356] "Generic (PLEG): container finished" podID="44227101-a97a-4bc9-a5b0-fd32cc961337" containerID="829c7b38cd116d38cc877ae98afebbc0baa8496ea21c8dc8ce8e1e43760ced3b" exitCode=0 Oct 30 21:12:27.654866 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:12:27.654721 5378 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-1fdaa7-openshift-default-84586db7d5-2svf5" event={"ID":"44227101-a97a-4bc9-a5b0-fd32cc961337","Type":"ContainerDied","Data":"829c7b38cd116d38cc877ae98afebbc0baa8496ea21c8dc8ce8e1e43760ced3b"} Oct 30 21:12:27.705864 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:12:27.705834 5378 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-1fdaa7-openshift-default-84586db7d5-2svf5" Oct 30 21:12:27.754891 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:12:27.754817 5378 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/44227101-a97a-4bc9-a5b0-fd32cc961337-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/44227101-a97a-4bc9-a5b0-fd32cc961337-credential-socket\") pod \"44227101-a97a-4bc9-a5b0-fd32cc961337\" (UID: \"44227101-a97a-4bc9-a5b0-fd32cc961337\") " Oct 30 21:12:27.755067 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:12:27.754894 5378 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/44227101-a97a-4bc9-a5b0-fd32cc961337-istio-token\" (UniqueName: \"kubernetes.io/projected/44227101-a97a-4bc9-a5b0-fd32cc961337-istio-token\") pod \"44227101-a97a-4bc9-a5b0-fd32cc961337\" (UID: \"44227101-a97a-4bc9-a5b0-fd32cc961337\") " Oct 30 21:12:27.755067 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:12:27.754993 5378 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/44227101-a97a-4bc9-a5b0-fd32cc961337-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/44227101-a97a-4bc9-a5b0-fd32cc961337-istiod-ca-cert\") pod \"44227101-a97a-4bc9-a5b0-fd32cc961337\" (UID: \"44227101-a97a-4bc9-a5b0-fd32cc961337\") " Oct 30 21:12:27.755067 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:12:27.755050 5378 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/44227101-a97a-4bc9-a5b0-fd32cc961337-kube-api-access-c26vh\" (UniqueName: \"kubernetes.io/projected/44227101-a97a-4bc9-a5b0-fd32cc961337-kube-api-access-c26vh\") pod \"44227101-a97a-4bc9-a5b0-fd32cc961337\" (UID: \"44227101-a97a-4bc9-a5b0-fd32cc961337\") " Oct 30 21:12:27.755259 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:12:27.755088 5378 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/44227101-a97a-4bc9-a5b0-fd32cc961337-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/44227101-a97a-4bc9-a5b0-fd32cc961337-istio-data\") pod \"44227101-a97a-4bc9-a5b0-fd32cc961337\" (UID: \"44227101-a97a-4bc9-a5b0-fd32cc961337\") " Oct 30 21:12:27.755259 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:12:27.755141 5378 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/44227101-a97a-4bc9-a5b0-fd32cc961337-credential-socket" pod "44227101-a97a-4bc9-a5b0-fd32cc961337" (UID: "44227101-a97a-4bc9-a5b0-fd32cc961337"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:12:27.755259 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:12:27.755237 5378 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/44227101-a97a-4bc9-a5b0-fd32cc961337-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/44227101-a97a-4bc9-a5b0-fd32cc961337-workload-certs\") pod \"44227101-a97a-4bc9-a5b0-fd32cc961337\" (UID: \"44227101-a97a-4bc9-a5b0-fd32cc961337\") " Oct 30 21:12:27.755927 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:12:27.755517 5378 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/44227101-a97a-4bc9-a5b0-fd32cc961337-istiod-ca-cert" pod "44227101-a97a-4bc9-a5b0-fd32cc961337" (UID: "44227101-a97a-4bc9-a5b0-fd32cc961337"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:12:27.756053 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:12:27.755786 5378 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/44227101-a97a-4bc9-a5b0-fd32cc961337-istio-data" pod "44227101-a97a-4bc9-a5b0-fd32cc961337" (UID: "44227101-a97a-4bc9-a5b0-fd32cc961337"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:12:27.756053 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:12:27.755802 5378 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/44227101-a97a-4bc9-a5b0-fd32cc961337-workload-certs" pod "44227101-a97a-4bc9-a5b0-fd32cc961337" (UID: "44227101-a97a-4bc9-a5b0-fd32cc961337"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:12:27.756053 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:12:27.755903 5378 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/44227101-a97a-4bc9-a5b0-fd32cc961337-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/44227101-a97a-4bc9-a5b0-fd32cc961337-workload-socket\") pod \"44227101-a97a-4bc9-a5b0-fd32cc961337\" (UID: \"44227101-a97a-4bc9-a5b0-fd32cc961337\") " Oct 30 21:12:27.756242 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:12:27.756060 5378 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/44227101-a97a-4bc9-a5b0-fd32cc961337-workload-socket" pod "44227101-a97a-4bc9-a5b0-fd32cc961337" (UID: "44227101-a97a-4bc9-a5b0-fd32cc961337"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:12:27.756242 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:12:27.756121 5378 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/44227101-a97a-4bc9-a5b0-fd32cc961337-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/44227101-a97a-4bc9-a5b0-fd32cc961337-istio-envoy\") pod \"44227101-a97a-4bc9-a5b0-fd32cc961337\" (UID: \"44227101-a97a-4bc9-a5b0-fd32cc961337\") " Oct 30 21:12:27.756242 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:12:27.756192 5378 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/44227101-a97a-4bc9-a5b0-fd32cc961337-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/44227101-a97a-4bc9-a5b0-fd32cc961337-istio-podinfo\") pod \"44227101-a97a-4bc9-a5b0-fd32cc961337\" (UID: \"44227101-a97a-4bc9-a5b0-fd32cc961337\") " Oct 30 21:12:27.757045 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:12:27.757017 5378 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/44227101-a97a-4bc9-a5b0-fd32cc961337-workload-certs\") on node \"ip-10-0-138-149.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:12:27.757172 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:12:27.757050 5378 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/44227101-a97a-4bc9-a5b0-fd32cc961337-workload-socket\") on node \"ip-10-0-138-149.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:12:27.757172 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:12:27.757065 5378 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/44227101-a97a-4bc9-a5b0-fd32cc961337-credential-socket\") on node \"ip-10-0-138-149.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:12:27.757172 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:12:27.757079 5378 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/44227101-a97a-4bc9-a5b0-fd32cc961337-istiod-ca-cert\") on node \"ip-10-0-138-149.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:12:27.757172 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:12:27.757092 5378 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/44227101-a97a-4bc9-a5b0-fd32cc961337-istio-data\") on node \"ip-10-0-138-149.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:12:27.758311 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:12:27.758287 5378 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/44227101-a97a-4bc9-a5b0-fd32cc961337-istio-token" pod "44227101-a97a-4bc9-a5b0-fd32cc961337" (UID: "44227101-a97a-4bc9-a5b0-fd32cc961337"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:12:27.759783 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:12:27.759757 5378 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/44227101-a97a-4bc9-a5b0-fd32cc961337-istio-envoy" pod "44227101-a97a-4bc9-a5b0-fd32cc961337" (UID: "44227101-a97a-4bc9-a5b0-fd32cc961337"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:12:27.759956 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:12:27.759931 5378 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/44227101-a97a-4bc9-a5b0-fd32cc961337-kube-api-access-c26vh" pod "44227101-a97a-4bc9-a5b0-fd32cc961337" (UID: "44227101-a97a-4bc9-a5b0-fd32cc961337"). InnerVolumeSpecName "kube-api-access-c26vh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:12:27.760492 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:12:27.760396 5378 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/44227101-a97a-4bc9-a5b0-fd32cc961337-istio-podinfo" pod "44227101-a97a-4bc9-a5b0-fd32cc961337" (UID: "44227101-a97a-4bc9-a5b0-fd32cc961337"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 21:12:27.858943 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:12:27.858910 5378 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/44227101-a97a-4bc9-a5b0-fd32cc961337-istio-envoy\") on node \"ip-10-0-138-149.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:12:27.858943 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:12:27.858943 5378 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/44227101-a97a-4bc9-a5b0-fd32cc961337-istio-podinfo\") on node \"ip-10-0-138-149.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:12:27.859130 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:12:27.858958 5378 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/44227101-a97a-4bc9-a5b0-fd32cc961337-istio-token\") on node \"ip-10-0-138-149.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:12:27.859130 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:12:27.858972 5378 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c26vh\" (UniqueName: \"kubernetes.io/projected/44227101-a97a-4bc9-a5b0-fd32cc961337-kube-api-access-c26vh\") on node \"ip-10-0-138-149.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:12:28.173007 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:12:28.172975 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:12:28.663614 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:12:28.663551 5378 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-1fdaa7-openshift-default-84586db7d5-2svf5" Oct 30 21:12:28.663614 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:12:28.663591 5378 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-1fdaa7-openshift-default-84586db7d5-2svf5" event={"ID":"44227101-a97a-4bc9-a5b0-fd32cc961337","Type":"ContainerDied","Data":"e563014471b67319a7023c03ec2a61ae98b2602ea3e6985cd000141e356ca4f8"} Oct 30 21:12:28.664148 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:12:28.663665 5378 scope.go:122] "RemoveContainer" containerID="829c7b38cd116d38cc877ae98afebbc0baa8496ea21c8dc8ce8e1e43760ced3b" Oct 30 21:12:28.833024 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:12:28.832994 5378 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-1fdaa7-openshift-default-84586db7d5-2svf5"] Oct 30 21:12:28.883038 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:12:28.883004 5378 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-1fdaa7-openshift-default-84586db7d5-2svf5"] Oct 30 21:12:29.575393 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:12:29.575362 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:12:29.753402 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:12:29.753372 5378 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="44227101-a97a-4bc9-a5b0-fd32cc961337" path="/var/lib/kubelet/pods/44227101-a97a-4bc9-a5b0-fd32cc961337/volumes" Oct 30 21:12:29.771040 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:12:29.771004 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:12:30.320511 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:12:30.320470 5378 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-a-w6-dc6125-6644bb494b-2v578" podUID="168599a3-7daf-4eba-a259-978486b56bd4" containerName="payload-processing" containerID="cri-o://a087ee550d15f63a1420aec405fb77c4bbdd4785f2dc958ce88cf68789e9763c" gracePeriod=25 Oct 30 21:12:30.427929 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:12:30.427897 5378 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-dc6125-74d649bf67-vlsfb" podUID="5ff92f96-e5bc-4b04-893b-16efd49f9d3f" containerName="payload-pre-processing" containerID="cri-o://119bfe8aa3cd04e4d95f140e1a87b0542f4940384e5eb1292d5ecffb20e995bf" gracePeriod=25 Oct 30 21:12:30.630668 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:12:30.630633 5378 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w6-dc6125-6644bb494b-2v578" Oct 30 21:12:30.739766 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:12:30.736749 5378 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/168599a3-7daf-4eba-a259-978486b56bd4-kube-api-access-8w7rb\" (UniqueName: \"kubernetes.io/projected/168599a3-7daf-4eba-a259-978486b56bd4-kube-api-access-8w7rb\") pod \"168599a3-7daf-4eba-a259-978486b56bd4\" (UID: \"168599a3-7daf-4eba-a259-978486b56bd4\") " Oct 30 21:12:30.739766 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:12:30.736930 5378 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/168599a3-7daf-4eba-a259-978486b56bd4-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/168599a3-7daf-4eba-a259-978486b56bd4-plugins-config-volume\") pod \"168599a3-7daf-4eba-a259-978486b56bd4\" (UID: \"168599a3-7daf-4eba-a259-978486b56bd4\") " Oct 30 21:12:30.777665 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:12:30.771719 5378 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/168599a3-7daf-4eba-a259-978486b56bd4-plugins-config-volume" pod "168599a3-7daf-4eba-a259-978486b56bd4" (UID: "168599a3-7daf-4eba-a259-978486b56bd4"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:12:30.872649 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:12:30.865321 5378 generic.go:356] "Generic (PLEG): container finished" podID="5ff92f96-e5bc-4b04-893b-16efd49f9d3f" containerID="119bfe8aa3cd04e4d95f140e1a87b0542f4940384e5eb1292d5ecffb20e995bf" exitCode=0 Oct 30 21:12:30.872649 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:12:30.865685 5378 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-dc6125-74d649bf67-vlsfb" event={"ID":"5ff92f96-e5bc-4b04-893b-16efd49f9d3f","Type":"ContainerDied","Data":"119bfe8aa3cd04e4d95f140e1a87b0542f4940384e5eb1292d5ecffb20e995bf"} Oct 30 21:12:30.894591 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:12:30.875594 5378 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/168599a3-7daf-4eba-a259-978486b56bd4-plugins-config-volume\") on node \"ip-10-0-138-149.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:12:31.161702 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:12:31.153241 5378 generic.go:356] "Generic (PLEG): container finished" podID="168599a3-7daf-4eba-a259-978486b56bd4" containerID="a087ee550d15f63a1420aec405fb77c4bbdd4785f2dc958ce88cf68789e9763c" exitCode=0 Oct 30 21:12:31.180801 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:12:31.180769 5378 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w6-dc6125-6644bb494b-2v578" Oct 30 21:12:31.181705 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:12:31.181672 5378 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w6-dc6125-6644bb494b-2v578" event={"ID":"168599a3-7daf-4eba-a259-978486b56bd4","Type":"ContainerDied","Data":"a087ee550d15f63a1420aec405fb77c4bbdd4785f2dc958ce88cf68789e9763c"} Oct 30 21:12:31.181995 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:12:31.181939 5378 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w6-dc6125-6644bb494b-2v578" event={"ID":"168599a3-7daf-4eba-a259-978486b56bd4","Type":"ContainerDied","Data":"bbaca4227c2b51fab16e5f55f3e6e891bc90f96c1c661b63070c28d78b7f2aed"} Oct 30 21:12:31.182228 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:12:31.182205 5378 scope.go:122] "RemoveContainer" containerID="a087ee550d15f63a1420aec405fb77c4bbdd4785f2dc958ce88cf68789e9763c" Oct 30 21:12:31.294930 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:12:31.294608 5378 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/168599a3-7daf-4eba-a259-978486b56bd4-kube-api-access-8w7rb" pod "168599a3-7daf-4eba-a259-978486b56bd4" (UID: "168599a3-7daf-4eba-a259-978486b56bd4"). InnerVolumeSpecName "kube-api-access-8w7rb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:12:31.383535 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:12:31.383510 5378 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-dc6125-74d649bf67-vlsfb" Oct 30 21:12:31.392135 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:12:31.392108 5378 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8w7rb\" (UniqueName: \"kubernetes.io/projected/168599a3-7daf-4eba-a259-978486b56bd4-kube-api-access-8w7rb\") on node \"ip-10-0-138-149.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:12:31.399370 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:12:31.399344 5378 scope.go:122] "RemoveContainer" containerID="a087ee550d15f63a1420aec405fb77c4bbdd4785f2dc958ce88cf68789e9763c" Oct 30 21:12:31.399780 ip-10-0-138-149 kubenswrapper[5378]: E1030 21:12:31.399753 5378 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a087ee550d15f63a1420aec405fb77c4bbdd4785f2dc958ce88cf68789e9763c\": container with ID starting with a087ee550d15f63a1420aec405fb77c4bbdd4785f2dc958ce88cf68789e9763c not found: ID does not exist" containerID="a087ee550d15f63a1420aec405fb77c4bbdd4785f2dc958ce88cf68789e9763c" Oct 30 21:12:31.399872 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:12:31.399794 5378 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a087ee550d15f63a1420aec405fb77c4bbdd4785f2dc958ce88cf68789e9763c"} err="failed to get container status \"a087ee550d15f63a1420aec405fb77c4bbdd4785f2dc958ce88cf68789e9763c\": rpc error: code = NotFound desc = could not find container \"a087ee550d15f63a1420aec405fb77c4bbdd4785f2dc958ce88cf68789e9763c\": container with ID starting with a087ee550d15f63a1420aec405fb77c4bbdd4785f2dc958ce88cf68789e9763c not found: ID does not exist" Oct 30 21:12:31.493048 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:12:31.493021 5378 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5ff92f96-e5bc-4b04-893b-16efd49f9d3f-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5ff92f96-e5bc-4b04-893b-16efd49f9d3f-plugins-config-volume\") pod \"5ff92f96-e5bc-4b04-893b-16efd49f9d3f\" (UID: \"5ff92f96-e5bc-4b04-893b-16efd49f9d3f\") " Oct 30 21:12:31.493217 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:12:31.493201 5378 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5ff92f96-e5bc-4b04-893b-16efd49f9d3f-kube-api-access-hxrq7\" (UniqueName: \"kubernetes.io/projected/5ff92f96-e5bc-4b04-893b-16efd49f9d3f-kube-api-access-hxrq7\") pod \"5ff92f96-e5bc-4b04-893b-16efd49f9d3f\" (UID: \"5ff92f96-e5bc-4b04-893b-16efd49f9d3f\") " Oct 30 21:12:31.493435 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:12:31.493408 5378 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5ff92f96-e5bc-4b04-893b-16efd49f9d3f-plugins-config-volume" pod "5ff92f96-e5bc-4b04-893b-16efd49f9d3f" (UID: "5ff92f96-e5bc-4b04-893b-16efd49f9d3f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:12:31.493876 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:12:31.493856 5378 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5ff92f96-e5bc-4b04-893b-16efd49f9d3f-plugins-config-volume\") on node \"ip-10-0-138-149.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:12:31.495293 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:12:31.495269 5378 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5ff92f96-e5bc-4b04-893b-16efd49f9d3f-kube-api-access-hxrq7" pod "5ff92f96-e5bc-4b04-893b-16efd49f9d3f" (UID: "5ff92f96-e5bc-4b04-893b-16efd49f9d3f"). InnerVolumeSpecName "kube-api-access-hxrq7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:12:31.594884 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:12:31.594854 5378 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hxrq7\" (UniqueName: \"kubernetes.io/projected/5ff92f96-e5bc-4b04-893b-16efd49f9d3f-kube-api-access-hxrq7\") on node \"ip-10-0-138-149.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:12:31.615083 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:12:31.615051 5378 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w6-dc6125-6644bb494b-2v578"] Oct 30 21:12:31.623858 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:12:31.623827 5378 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w6-dc6125-6644bb494b-2v578"] Oct 30 21:12:31.744610 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:12:31.744506 5378 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="168599a3-7daf-4eba-a259-978486b56bd4" path="/var/lib/kubelet/pods/168599a3-7daf-4eba-a259-978486b56bd4/volumes" Oct 30 21:12:32.174559 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:12:32.174532 5378 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-dc6125-74d649bf67-vlsfb" Oct 30 21:12:32.175017 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:12:32.174581 5378 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-dc6125-74d649bf67-vlsfb" event={"ID":"5ff92f96-e5bc-4b04-893b-16efd49f9d3f","Type":"ContainerDied","Data":"0cc8cede770432d7556933712d081b303ad2fa199c0028d5af935a277f65d762"} Oct 30 21:12:32.175017 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:12:32.174651 5378 scope.go:122] "RemoveContainer" containerID="119bfe8aa3cd04e4d95f140e1a87b0542f4940384e5eb1292d5ecffb20e995bf" Oct 30 21:12:32.341810 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:12:32.341778 5378 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w6-dc6125-74d649bf67-vlsfb"] Oct 30 21:12:32.345515 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:12:32.345492 5378 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w6-dc6125-74d649bf67-vlsfb"] Oct 30 21:12:33.500541 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:12:33.500507 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:12:33.680830 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:12:33.673998 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:12:33.750515 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:12:33.750476 5378 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5ff92f96-e5bc-4b04-893b-16efd49f9d3f" path="/var/lib/kubelet/pods/5ff92f96-e5bc-4b04-893b-16efd49f9d3f/volumes" Oct 30 21:12:39.107683 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:12:39.107645 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:12:39.361723 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:12:39.361630 5378 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w1-4d1110-694854c68-k9bh2"] Oct 30 21:12:39.483416 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:12:39.483377 5378 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w1-4d1110-5cfd7bf99f-mxznk"] Oct 30 21:12:39.731469 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:12:39.731387 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:12:39.888828 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:12:39.888793 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:12:44.362162 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:12:44.362105 5378 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-4d1110-694854c68-k9bh2" podUID="5a5302ee-2e3c-427b-a222-940f0a913372" containerName="payload-processing" containerID="cri-o://cc2bd4ae2f1dd4e16538a348649f0aa24a6da5f156470f318aeced97ed8f43a5" gracePeriod=25 Oct 30 21:12:44.458166 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:12:44.458129 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:12:44.484286 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:12:44.484224 5378 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-4d1110-5cfd7bf99f-mxznk" podUID="b8b5625b-b0ec-4e9c-8b80-338ea02008f9" containerName="payload-pre-processing" containerID="cri-o://a6bf94a3efd1b7f0fb91c26f016b4d32f36dadb7c5e68d4c5eec54334139837d" gracePeriod=25 Oct 30 21:12:44.719139 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:12:44.719107 5378 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-4d1110-694854c68-k9bh2" Oct 30 21:12:44.812996 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:12:44.812972 5378 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5a5302ee-2e3c-427b-a222-940f0a913372-kube-api-access-v45ps\" (UniqueName: \"kubernetes.io/projected/5a5302ee-2e3c-427b-a222-940f0a913372-kube-api-access-v45ps\") pod \"5a5302ee-2e3c-427b-a222-940f0a913372\" (UID: \"5a5302ee-2e3c-427b-a222-940f0a913372\") " Oct 30 21:12:44.813155 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:12:44.813143 5378 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5a5302ee-2e3c-427b-a222-940f0a913372-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5a5302ee-2e3c-427b-a222-940f0a913372-plugins-config-volume\") pod \"5a5302ee-2e3c-427b-a222-940f0a913372\" (UID: \"5a5302ee-2e3c-427b-a222-940f0a913372\") " Oct 30 21:12:44.813496 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:12:44.813470 5378 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5a5302ee-2e3c-427b-a222-940f0a913372-plugins-config-volume" pod "5a5302ee-2e3c-427b-a222-940f0a913372" (UID: "5a5302ee-2e3c-427b-a222-940f0a913372"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:12:44.813780 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:12:44.813761 5378 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5a5302ee-2e3c-427b-a222-940f0a913372-plugins-config-volume\") on node \"ip-10-0-138-149.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:12:44.815027 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:12:44.815003 5378 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5a5302ee-2e3c-427b-a222-940f0a913372-kube-api-access-v45ps" pod "5a5302ee-2e3c-427b-a222-940f0a913372" (UID: "5a5302ee-2e3c-427b-a222-940f0a913372"). InnerVolumeSpecName "kube-api-access-v45ps". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:12:44.819391 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:12:44.819372 5378 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-4d1110-5cfd7bf99f-mxznk" Oct 30 21:12:44.914415 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:12:44.914339 5378 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b8b5625b-b0ec-4e9c-8b80-338ea02008f9-kube-api-access-sg5fl\" (UniqueName: \"kubernetes.io/projected/b8b5625b-b0ec-4e9c-8b80-338ea02008f9-kube-api-access-sg5fl\") pod \"b8b5625b-b0ec-4e9c-8b80-338ea02008f9\" (UID: \"b8b5625b-b0ec-4e9c-8b80-338ea02008f9\") " Oct 30 21:12:44.914544 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:12:44.914459 5378 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b8b5625b-b0ec-4e9c-8b80-338ea02008f9-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b8b5625b-b0ec-4e9c-8b80-338ea02008f9-plugins-config-volume\") pod \"b8b5625b-b0ec-4e9c-8b80-338ea02008f9\" (UID: \"b8b5625b-b0ec-4e9c-8b80-338ea02008f9\") " Oct 30 21:12:44.914885 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:12:44.914846 5378 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b8b5625b-b0ec-4e9c-8b80-338ea02008f9-plugins-config-volume" pod "b8b5625b-b0ec-4e9c-8b80-338ea02008f9" (UID: "b8b5625b-b0ec-4e9c-8b80-338ea02008f9"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:12:44.915028 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:12:44.914974 5378 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b8b5625b-b0ec-4e9c-8b80-338ea02008f9-plugins-config-volume\") on node \"ip-10-0-138-149.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:12:44.915028 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:12:44.914995 5378 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v45ps\" (UniqueName: \"kubernetes.io/projected/5a5302ee-2e3c-427b-a222-940f0a913372-kube-api-access-v45ps\") on node \"ip-10-0-138-149.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:12:44.916397 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:12:44.916378 5378 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b8b5625b-b0ec-4e9c-8b80-338ea02008f9-kube-api-access-sg5fl" pod "b8b5625b-b0ec-4e9c-8b80-338ea02008f9" (UID: "b8b5625b-b0ec-4e9c-8b80-338ea02008f9"). InnerVolumeSpecName "kube-api-access-sg5fl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:12:45.016488 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:12:45.016453 5378 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sg5fl\" (UniqueName: \"kubernetes.io/projected/b8b5625b-b0ec-4e9c-8b80-338ea02008f9-kube-api-access-sg5fl\") on node \"ip-10-0-138-149.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:12:45.382512 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:12:45.382436 5378 generic.go:356] "Generic (PLEG): container finished" podID="b8b5625b-b0ec-4e9c-8b80-338ea02008f9" containerID="a6bf94a3efd1b7f0fb91c26f016b4d32f36dadb7c5e68d4c5eec54334139837d" exitCode=0 Oct 30 21:12:45.382512 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:12:45.382487 5378 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-4d1110-5cfd7bf99f-mxznk" Oct 30 21:12:45.382945 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:12:45.382510 5378 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-4d1110-5cfd7bf99f-mxznk" event={"ID":"b8b5625b-b0ec-4e9c-8b80-338ea02008f9","Type":"ContainerDied","Data":"a6bf94a3efd1b7f0fb91c26f016b4d32f36dadb7c5e68d4c5eec54334139837d"} Oct 30 21:12:45.382945 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:12:45.382600 5378 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-4d1110-5cfd7bf99f-mxznk" event={"ID":"b8b5625b-b0ec-4e9c-8b80-338ea02008f9","Type":"ContainerDied","Data":"140b404b03f8cb4536b436f62bc5f571b930c0eb9793ffead0af2a645c73cd93"} Oct 30 21:12:45.382945 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:12:45.382636 5378 scope.go:122] "RemoveContainer" containerID="a6bf94a3efd1b7f0fb91c26f016b4d32f36dadb7c5e68d4c5eec54334139837d" Oct 30 21:12:45.384325 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:12:45.384201 5378 generic.go:356] "Generic (PLEG): container finished" podID="5a5302ee-2e3c-427b-a222-940f0a913372" containerID="cc2bd4ae2f1dd4e16538a348649f0aa24a6da5f156470f318aeced97ed8f43a5" exitCode=0 Oct 30 21:12:45.384325 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:12:45.384260 5378 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-4d1110-694854c68-k9bh2" Oct 30 21:12:45.384325 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:12:45.384294 5378 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-4d1110-694854c68-k9bh2" event={"ID":"5a5302ee-2e3c-427b-a222-940f0a913372","Type":"ContainerDied","Data":"cc2bd4ae2f1dd4e16538a348649f0aa24a6da5f156470f318aeced97ed8f43a5"} Oct 30 21:12:45.384547 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:12:45.384343 5378 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-4d1110-694854c68-k9bh2" event={"ID":"5a5302ee-2e3c-427b-a222-940f0a913372","Type":"ContainerDied","Data":"7e7805d35b79f061e22ef571b2c514ae77e5e26d807d266e4ee008bf4529ead9"} Oct 30 21:12:45.395300 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:12:45.395283 5378 scope.go:122] "RemoveContainer" containerID="a6bf94a3efd1b7f0fb91c26f016b4d32f36dadb7c5e68d4c5eec54334139837d" Oct 30 21:12:45.395602 ip-10-0-138-149 kubenswrapper[5378]: E1030 21:12:45.395550 5378 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a6bf94a3efd1b7f0fb91c26f016b4d32f36dadb7c5e68d4c5eec54334139837d\": container with ID starting with a6bf94a3efd1b7f0fb91c26f016b4d32f36dadb7c5e68d4c5eec54334139837d not found: ID does not exist" containerID="a6bf94a3efd1b7f0fb91c26f016b4d32f36dadb7c5e68d4c5eec54334139837d" Oct 30 21:12:45.395704 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:12:45.395606 5378 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a6bf94a3efd1b7f0fb91c26f016b4d32f36dadb7c5e68d4c5eec54334139837d"} err="failed to get container status \"a6bf94a3efd1b7f0fb91c26f016b4d32f36dadb7c5e68d4c5eec54334139837d\": rpc error: code = NotFound desc = could not find container \"a6bf94a3efd1b7f0fb91c26f016b4d32f36dadb7c5e68d4c5eec54334139837d\": container with ID starting with a6bf94a3efd1b7f0fb91c26f016b4d32f36dadb7c5e68d4c5eec54334139837d not found: ID does not exist" Oct 30 21:12:45.395704 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:12:45.395622 5378 scope.go:122] "RemoveContainer" containerID="cc2bd4ae2f1dd4e16538a348649f0aa24a6da5f156470f318aeced97ed8f43a5" Oct 30 21:12:45.404517 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:12:45.404494 5378 scope.go:122] "RemoveContainer" containerID="cc2bd4ae2f1dd4e16538a348649f0aa24a6da5f156470f318aeced97ed8f43a5" Oct 30 21:12:45.404814 ip-10-0-138-149 kubenswrapper[5378]: E1030 21:12:45.404796 5378 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cc2bd4ae2f1dd4e16538a348649f0aa24a6da5f156470f318aeced97ed8f43a5\": container with ID starting with cc2bd4ae2f1dd4e16538a348649f0aa24a6da5f156470f318aeced97ed8f43a5 not found: ID does not exist" containerID="cc2bd4ae2f1dd4e16538a348649f0aa24a6da5f156470f318aeced97ed8f43a5" Oct 30 21:12:45.404882 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:12:45.404824 5378 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cc2bd4ae2f1dd4e16538a348649f0aa24a6da5f156470f318aeced97ed8f43a5"} err="failed to get container status \"cc2bd4ae2f1dd4e16538a348649f0aa24a6da5f156470f318aeced97ed8f43a5\": rpc error: code = NotFound desc = could not find container \"cc2bd4ae2f1dd4e16538a348649f0aa24a6da5f156470f318aeced97ed8f43a5\": container with ID starting with cc2bd4ae2f1dd4e16538a348649f0aa24a6da5f156470f318aeced97ed8f43a5 not found: ID does not exist" Oct 30 21:12:45.551413 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:12:45.551384 5378 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w1-4d1110-5cfd7bf99f-mxznk"] Oct 30 21:12:45.601723 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:12:45.601697 5378 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w1-4d1110-5cfd7bf99f-mxznk"] Oct 30 21:12:45.712658 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:12:45.712550 5378 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w1-4d1110-694854c68-k9bh2"] Oct 30 21:12:45.716773 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:12:45.716744 5378 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w1-4d1110-694854c68-k9bh2"] Oct 30 21:12:45.744019 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:12:45.743991 5378 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5a5302ee-2e3c-427b-a222-940f0a913372" path="/var/lib/kubelet/pods/5a5302ee-2e3c-427b-a222-940f0a913372/volumes" Oct 30 21:12:45.744506 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:12:45.744490 5378 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b8b5625b-b0ec-4e9c-8b80-338ea02008f9" path="/var/lib/kubelet/pods/b8b5625b-b0ec-4e9c-8b80-338ea02008f9/volumes" Oct 30 21:12:47.781530 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:12:47.781493 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:12:48.839033 ip-10-0-138-149 kubenswrapper[5378]: E1030 21:12:48.838995 5378 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q6mx7" containerName="extractor" Oct 30 21:12:49.305690 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:12:49.305654 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:12:49.463133 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:12:49.463100 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:12:51.549770 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:12:51.549739 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:12:51.692793 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:12:51.692740 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:12:54.658024 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:12:54.657985 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:12:59.199334 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:12:59.199292 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:13:00.484858 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:13:00.484802 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:13:00.885260 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:13:00.885210 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:13:01.303658 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:13:01.303522 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:13:01.942049 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:13:01.942009 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:13:02.114255 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:13:02.114224 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:13:02.323974 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:13:02.323892 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:13:02.454592 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:13:02.449407 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:13:03.140012 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:13:03.139973 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:13:07.515357 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:13:07.515290 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:13:08.004815 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:13:08.004763 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:13:08.208727 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:13:08.208691 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:13:08.347013 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:13:08.346973 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:13:09.713216 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:13:09.713184 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:13:09.867902 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:13:09.867868 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:13:10.057816 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:13:10.057726 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:13:10.085073 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:13:10.085038 5378 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-f3b172-openshift-default-d64ffcd45-k78ls"] Oct 30 21:13:10.085302 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:13:10.085279 5378 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-f3b172-openshift-default-d64ffcd45-k78ls" podUID="34b71d39-551a-4d2e-bf08-8df0cf45fc9e" containerName="istio-proxy" containerID="cri-o://926cb2ae14c55ff798f1a2b06bfbafc9ed278b76db65445d73811db30e54fdec" gracePeriod=30 Oct 30 21:13:10.571459 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:13:10.571425 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:13:11.281527 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:13:11.281479 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:13:11.863844 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:13:11.863777 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:13:12.596960 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:13:12.596928 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:13:13.800334 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:13:13.800301 5378 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-r9g59"] Oct 30 21:13:14.868169 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:13:14.868136 5378 patch_prober.go:29] interesting pod/e2e-inf-a-f3b172-openshift-default-d64ffcd45-k78ls container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 21:13:14.868557 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:13:14.868202 5378 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-a-f3b172-openshift-default-d64ffcd45-k78ls" podUID="34b71d39-551a-4d2e-bf08-8df0cf45fc9e" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 21:13:15.374974 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:13:15.374947 5378 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-f3b172-openshift-default-d64ffcd45-k78ls" Oct 30 21:13:15.456903 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:13:15.456808 5378 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/34b71d39-551a-4d2e-bf08-8df0cf45fc9e-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/34b71d39-551a-4d2e-bf08-8df0cf45fc9e-workload-certs\") pod \"34b71d39-551a-4d2e-bf08-8df0cf45fc9e\" (UID: \"34b71d39-551a-4d2e-bf08-8df0cf45fc9e\") " Oct 30 21:13:15.457062 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:13:15.456903 5378 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/34b71d39-551a-4d2e-bf08-8df0cf45fc9e-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/34b71d39-551a-4d2e-bf08-8df0cf45fc9e-credential-socket\") pod \"34b71d39-551a-4d2e-bf08-8df0cf45fc9e\" (UID: \"34b71d39-551a-4d2e-bf08-8df0cf45fc9e\") " Oct 30 21:13:15.457062 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:13:15.456974 5378 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/34b71d39-551a-4d2e-bf08-8df0cf45fc9e-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/34b71d39-551a-4d2e-bf08-8df0cf45fc9e-istio-podinfo\") pod \"34b71d39-551a-4d2e-bf08-8df0cf45fc9e\" (UID: \"34b71d39-551a-4d2e-bf08-8df0cf45fc9e\") " Oct 30 21:13:15.457062 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:13:15.457004 5378 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/34b71d39-551a-4d2e-bf08-8df0cf45fc9e-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/34b71d39-551a-4d2e-bf08-8df0cf45fc9e-workload-socket\") pod \"34b71d39-551a-4d2e-bf08-8df0cf45fc9e\" (UID: \"34b71d39-551a-4d2e-bf08-8df0cf45fc9e\") " Oct 30 21:13:15.457062 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:13:15.457054 5378 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/34b71d39-551a-4d2e-bf08-8df0cf45fc9e-istio-token\" (UniqueName: \"kubernetes.io/projected/34b71d39-551a-4d2e-bf08-8df0cf45fc9e-istio-token\") pod \"34b71d39-551a-4d2e-bf08-8df0cf45fc9e\" (UID: \"34b71d39-551a-4d2e-bf08-8df0cf45fc9e\") " Oct 30 21:13:15.457300 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:13:15.457083 5378 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/34b71d39-551a-4d2e-bf08-8df0cf45fc9e-workload-certs" pod "34b71d39-551a-4d2e-bf08-8df0cf45fc9e" (UID: "34b71d39-551a-4d2e-bf08-8df0cf45fc9e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:13:15.457300 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:13:15.457114 5378 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/34b71d39-551a-4d2e-bf08-8df0cf45fc9e-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/34b71d39-551a-4d2e-bf08-8df0cf45fc9e-istiod-ca-cert\") pod \"34b71d39-551a-4d2e-bf08-8df0cf45fc9e\" (UID: \"34b71d39-551a-4d2e-bf08-8df0cf45fc9e\") " Oct 30 21:13:15.457300 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:13:15.457192 5378 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/34b71d39-551a-4d2e-bf08-8df0cf45fc9e-credential-socket" pod "34b71d39-551a-4d2e-bf08-8df0cf45fc9e" (UID: "34b71d39-551a-4d2e-bf08-8df0cf45fc9e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:13:15.457300 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:13:15.457235 5378 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/34b71d39-551a-4d2e-bf08-8df0cf45fc9e-kube-api-access-tkd7f\" (UniqueName: \"kubernetes.io/projected/34b71d39-551a-4d2e-bf08-8df0cf45fc9e-kube-api-access-tkd7f\") pod \"34b71d39-551a-4d2e-bf08-8df0cf45fc9e\" (UID: \"34b71d39-551a-4d2e-bf08-8df0cf45fc9e\") " Oct 30 21:13:15.457300 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:13:15.457286 5378 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/34b71d39-551a-4d2e-bf08-8df0cf45fc9e-workload-socket" pod "34b71d39-551a-4d2e-bf08-8df0cf45fc9e" (UID: "34b71d39-551a-4d2e-bf08-8df0cf45fc9e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:13:15.457691 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:13:15.457353 5378 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/34b71d39-551a-4d2e-bf08-8df0cf45fc9e-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/34b71d39-551a-4d2e-bf08-8df0cf45fc9e-istio-envoy\") pod \"34b71d39-551a-4d2e-bf08-8df0cf45fc9e\" (UID: \"34b71d39-551a-4d2e-bf08-8df0cf45fc9e\") " Oct 30 21:13:15.457691 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:13:15.457399 5378 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/34b71d39-551a-4d2e-bf08-8df0cf45fc9e-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/34b71d39-551a-4d2e-bf08-8df0cf45fc9e-istio-data\") pod \"34b71d39-551a-4d2e-bf08-8df0cf45fc9e\" (UID: \"34b71d39-551a-4d2e-bf08-8df0cf45fc9e\") " Oct 30 21:13:15.457691 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:13:15.457548 5378 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/34b71d39-551a-4d2e-bf08-8df0cf45fc9e-istiod-ca-cert" pod "34b71d39-551a-4d2e-bf08-8df0cf45fc9e" (UID: "34b71d39-551a-4d2e-bf08-8df0cf45fc9e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:13:15.457867 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:13:15.457692 5378 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/34b71d39-551a-4d2e-bf08-8df0cf45fc9e-istio-data" pod "34b71d39-551a-4d2e-bf08-8df0cf45fc9e" (UID: "34b71d39-551a-4d2e-bf08-8df0cf45fc9e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:13:15.458230 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:13:15.458202 5378 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/34b71d39-551a-4d2e-bf08-8df0cf45fc9e-workload-certs\") on node \"ip-10-0-138-149.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:13:15.458371 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:13:15.458232 5378 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/34b71d39-551a-4d2e-bf08-8df0cf45fc9e-credential-socket\") on node \"ip-10-0-138-149.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:13:15.458371 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:13:15.458249 5378 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/34b71d39-551a-4d2e-bf08-8df0cf45fc9e-workload-socket\") on node \"ip-10-0-138-149.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:13:15.458371 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:13:15.458263 5378 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/34b71d39-551a-4d2e-bf08-8df0cf45fc9e-istiod-ca-cert\") on node \"ip-10-0-138-149.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:13:15.458371 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:13:15.458278 5378 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/34b71d39-551a-4d2e-bf08-8df0cf45fc9e-istio-data\") on node \"ip-10-0-138-149.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:13:15.459359 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:13:15.459326 5378 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/34b71d39-551a-4d2e-bf08-8df0cf45fc9e-istio-podinfo" pod "34b71d39-551a-4d2e-bf08-8df0cf45fc9e" (UID: "34b71d39-551a-4d2e-bf08-8df0cf45fc9e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 21:13:15.459475 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:13:15.459405 5378 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/34b71d39-551a-4d2e-bf08-8df0cf45fc9e-istio-envoy" pod "34b71d39-551a-4d2e-bf08-8df0cf45fc9e" (UID: "34b71d39-551a-4d2e-bf08-8df0cf45fc9e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:13:15.459475 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:13:15.459412 5378 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/34b71d39-551a-4d2e-bf08-8df0cf45fc9e-istio-token" pod "34b71d39-551a-4d2e-bf08-8df0cf45fc9e" (UID: "34b71d39-551a-4d2e-bf08-8df0cf45fc9e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:13:15.459605 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:13:15.459514 5378 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/34b71d39-551a-4d2e-bf08-8df0cf45fc9e-kube-api-access-tkd7f" pod "34b71d39-551a-4d2e-bf08-8df0cf45fc9e" (UID: "34b71d39-551a-4d2e-bf08-8df0cf45fc9e"). InnerVolumeSpecName "kube-api-access-tkd7f". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:13:15.560156 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:13:15.560127 5378 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/34b71d39-551a-4d2e-bf08-8df0cf45fc9e-istio-envoy\") on node \"ip-10-0-138-149.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:13:15.560156 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:13:15.560153 5378 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/34b71d39-551a-4d2e-bf08-8df0cf45fc9e-istio-podinfo\") on node \"ip-10-0-138-149.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:13:15.560326 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:13:15.560164 5378 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/34b71d39-551a-4d2e-bf08-8df0cf45fc9e-istio-token\") on node \"ip-10-0-138-149.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:13:15.560326 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:13:15.560173 5378 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tkd7f\" (UniqueName: \"kubernetes.io/projected/34b71d39-551a-4d2e-bf08-8df0cf45fc9e-kube-api-access-tkd7f\") on node \"ip-10-0-138-149.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:13:15.704903 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:13:15.704869 5378 generic.go:356] "Generic (PLEG): container finished" podID="34b71d39-551a-4d2e-bf08-8df0cf45fc9e" containerID="926cb2ae14c55ff798f1a2b06bfbafc9ed278b76db65445d73811db30e54fdec" exitCode=0 Oct 30 21:13:15.705092 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:13:15.704930 5378 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-f3b172-openshift-default-d64ffcd45-k78ls" Oct 30 21:13:15.705092 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:13:15.704953 5378 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-f3b172-openshift-default-d64ffcd45-k78ls" event={"ID":"34b71d39-551a-4d2e-bf08-8df0cf45fc9e","Type":"ContainerDied","Data":"926cb2ae14c55ff798f1a2b06bfbafc9ed278b76db65445d73811db30e54fdec"} Oct 30 21:13:15.705092 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:13:15.705020 5378 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-f3b172-openshift-default-d64ffcd45-k78ls" event={"ID":"34b71d39-551a-4d2e-bf08-8df0cf45fc9e","Type":"ContainerDied","Data":"454ae9fd6c93d5b4243d38292fe626e2ff0e57b0d8767abbe78fcd4d73a04e4f"} Oct 30 21:13:15.705092 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:13:15.705049 5378 scope.go:122] "RemoveContainer" containerID="926cb2ae14c55ff798f1a2b06bfbafc9ed278b76db65445d73811db30e54fdec" Oct 30 21:13:15.716533 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:13:15.716513 5378 scope.go:122] "RemoveContainer" containerID="62c77678f268f015b05bf2bc59bc3683ebf9d7164731bcbf27258a0c9b82483d" Oct 30 21:13:15.725804 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:13:15.725786 5378 scope.go:122] "RemoveContainer" containerID="926cb2ae14c55ff798f1a2b06bfbafc9ed278b76db65445d73811db30e54fdec" Oct 30 21:13:15.726076 ip-10-0-138-149 kubenswrapper[5378]: E1030 21:13:15.726057 5378 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"926cb2ae14c55ff798f1a2b06bfbafc9ed278b76db65445d73811db30e54fdec\": container with ID starting with 926cb2ae14c55ff798f1a2b06bfbafc9ed278b76db65445d73811db30e54fdec not found: ID does not exist" containerID="926cb2ae14c55ff798f1a2b06bfbafc9ed278b76db65445d73811db30e54fdec" Oct 30 21:13:15.726122 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:13:15.726086 5378 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"926cb2ae14c55ff798f1a2b06bfbafc9ed278b76db65445d73811db30e54fdec"} err="failed to get container status \"926cb2ae14c55ff798f1a2b06bfbafc9ed278b76db65445d73811db30e54fdec\": rpc error: code = NotFound desc = could not find container \"926cb2ae14c55ff798f1a2b06bfbafc9ed278b76db65445d73811db30e54fdec\": container with ID starting with 926cb2ae14c55ff798f1a2b06bfbafc9ed278b76db65445d73811db30e54fdec not found: ID does not exist" Oct 30 21:13:15.726122 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:13:15.726109 5378 scope.go:122] "RemoveContainer" containerID="62c77678f268f015b05bf2bc59bc3683ebf9d7164731bcbf27258a0c9b82483d" Oct 30 21:13:15.726394 ip-10-0-138-149 kubenswrapper[5378]: E1030 21:13:15.726376 5378 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"62c77678f268f015b05bf2bc59bc3683ebf9d7164731bcbf27258a0c9b82483d\": container with ID starting with 62c77678f268f015b05bf2bc59bc3683ebf9d7164731bcbf27258a0c9b82483d not found: ID does not exist" containerID="62c77678f268f015b05bf2bc59bc3683ebf9d7164731bcbf27258a0c9b82483d" Oct 30 21:13:15.726451 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:13:15.726399 5378 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"62c77678f268f015b05bf2bc59bc3683ebf9d7164731bcbf27258a0c9b82483d"} err="failed to get container status \"62c77678f268f015b05bf2bc59bc3683ebf9d7164731bcbf27258a0c9b82483d\": rpc error: code = NotFound desc = could not find container \"62c77678f268f015b05bf2bc59bc3683ebf9d7164731bcbf27258a0c9b82483d\": container with ID starting with 62c77678f268f015b05bf2bc59bc3683ebf9d7164731bcbf27258a0c9b82483d not found: ID does not exist" Oct 30 21:13:15.872770 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:13:15.872733 5378 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-f3b172-openshift-default-d64ffcd45-k78ls"] Oct 30 21:13:15.878519 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:13:15.878492 5378 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-f3b172-openshift-default-d64ffcd45-k78ls"] Oct 30 21:13:15.969297 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:13:15.969213 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:13:16.144800 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:13:16.144771 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:13:16.337544 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:13:16.337462 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:13:17.744962 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:13:17.744917 5378 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="34b71d39-551a-4d2e-bf08-8df0cf45fc9e" path="/var/lib/kubelet/pods/34b71d39-551a-4d2e-bf08-8df0cf45fc9e/volumes" Oct 30 21:13:18.246349 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:13:18.246309 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:13:18.842584 ip-10-0-138-149 kubenswrapper[5378]: E1030 21:13:18.841769 5378 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q6mx7" containerName="extractor" Oct 30 21:13:19.369765 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:13:19.369693 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:13:19.582588 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:13:19.563669 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:13:23.125364 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:13:23.125326 5378 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-de53c2-6dc7f4bf96-b5cbp"] Oct 30 21:13:23.233306 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:13:23.233266 5378 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-de53c2-fdf4f8fc5-8fjrg"] Oct 30 21:13:23.501383 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:13:23.501282 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:13:23.641687 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:13:23.640092 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:13:24.009891 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:13:24.009860 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:13:28.125903 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:13:28.125841 5378 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-de53c2-6dc7f4bf96-b5cbp" podUID="d9131396-6236-4a21-8c77-0f78fcc851f8" containerName="payload-processing" containerID="cri-o://d910afbb1d2150b7cbdb312e1517a33d01df6c475363dfee8ad46c9fbf4f0c17" gracePeriod=25 Oct 30 21:13:28.234112 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:13:28.234061 5378 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-de53c2-fdf4f8fc5-8fjrg" podUID="4f0b0d35-65b6-4721-8939-8d2d6dbb8fc5" containerName="payload-pre-processing" containerID="cri-o://5f4c9210b74cb706168c1d5e32e749971cb46563c40a0c9342f8bed2956cc472" gracePeriod=25 Oct 30 21:13:28.413452 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:13:28.413427 5378 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-de53c2-6dc7f4bf96-b5cbp" Oct 30 21:13:28.523169 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:13:28.523143 5378 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-de53c2-fdf4f8fc5-8fjrg" Oct 30 21:13:28.536144 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:13:28.536113 5378 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d9131396-6236-4a21-8c77-0f78fcc851f8-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d9131396-6236-4a21-8c77-0f78fcc851f8-plugins-config-volume\") pod \"d9131396-6236-4a21-8c77-0f78fcc851f8\" (UID: \"d9131396-6236-4a21-8c77-0f78fcc851f8\") " Oct 30 21:13:28.536288 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:13:28.536212 5378 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d9131396-6236-4a21-8c77-0f78fcc851f8-kube-api-access-fgh58\" (UniqueName: \"kubernetes.io/projected/d9131396-6236-4a21-8c77-0f78fcc851f8-kube-api-access-fgh58\") pod \"d9131396-6236-4a21-8c77-0f78fcc851f8\" (UID: \"d9131396-6236-4a21-8c77-0f78fcc851f8\") " Oct 30 21:13:28.538863 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:13:28.538831 5378 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d9131396-6236-4a21-8c77-0f78fcc851f8-kube-api-access-fgh58" pod "d9131396-6236-4a21-8c77-0f78fcc851f8" (UID: "d9131396-6236-4a21-8c77-0f78fcc851f8"). InnerVolumeSpecName "kube-api-access-fgh58". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:13:28.540804 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:13:28.540774 5378 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d9131396-6236-4a21-8c77-0f78fcc851f8-plugins-config-volume" pod "d9131396-6236-4a21-8c77-0f78fcc851f8" (UID: "d9131396-6236-4a21-8c77-0f78fcc851f8"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:13:28.638050 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:13:28.637966 5378 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4f0b0d35-65b6-4721-8939-8d2d6dbb8fc5-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4f0b0d35-65b6-4721-8939-8d2d6dbb8fc5-plugins-config-volume\") pod \"4f0b0d35-65b6-4721-8939-8d2d6dbb8fc5\" (UID: \"4f0b0d35-65b6-4721-8939-8d2d6dbb8fc5\") " Oct 30 21:13:28.638237 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:13:28.638079 5378 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4f0b0d35-65b6-4721-8939-8d2d6dbb8fc5-kube-api-access-4tqb7\" (UniqueName: \"kubernetes.io/projected/4f0b0d35-65b6-4721-8939-8d2d6dbb8fc5-kube-api-access-4tqb7\") pod \"4f0b0d35-65b6-4721-8939-8d2d6dbb8fc5\" (UID: \"4f0b0d35-65b6-4721-8939-8d2d6dbb8fc5\") " Oct 30 21:13:28.638418 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:13:28.638393 5378 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4f0b0d35-65b6-4721-8939-8d2d6dbb8fc5-plugins-config-volume" pod "4f0b0d35-65b6-4721-8939-8d2d6dbb8fc5" (UID: "4f0b0d35-65b6-4721-8939-8d2d6dbb8fc5"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:13:28.638738 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:13:28.638720 5378 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d9131396-6236-4a21-8c77-0f78fcc851f8-plugins-config-volume\") on node \"ip-10-0-138-149.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:13:28.638815 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:13:28.638742 5378 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4f0b0d35-65b6-4721-8939-8d2d6dbb8fc5-plugins-config-volume\") on node \"ip-10-0-138-149.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:13:28.638815 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:13:28.638756 5378 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fgh58\" (UniqueName: \"kubernetes.io/projected/d9131396-6236-4a21-8c77-0f78fcc851f8-kube-api-access-fgh58\") on node \"ip-10-0-138-149.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:13:28.640168 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:13:28.640148 5378 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4f0b0d35-65b6-4721-8939-8d2d6dbb8fc5-kube-api-access-4tqb7" pod "4f0b0d35-65b6-4721-8939-8d2d6dbb8fc5" (UID: "4f0b0d35-65b6-4721-8939-8d2d6dbb8fc5"). InnerVolumeSpecName "kube-api-access-4tqb7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:13:28.740049 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:13:28.740013 5378 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4tqb7\" (UniqueName: \"kubernetes.io/projected/4f0b0d35-65b6-4721-8939-8d2d6dbb8fc5-kube-api-access-4tqb7\") on node \"ip-10-0-138-149.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:13:28.997070 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:13:28.996988 5378 generic.go:356] "Generic (PLEG): container finished" podID="4f0b0d35-65b6-4721-8939-8d2d6dbb8fc5" containerID="5f4c9210b74cb706168c1d5e32e749971cb46563c40a0c9342f8bed2956cc472" exitCode=0 Oct 30 21:13:28.997070 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:13:28.997041 5378 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-de53c2-fdf4f8fc5-8fjrg" Oct 30 21:13:28.997277 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:13:28.997077 5378 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-de53c2-fdf4f8fc5-8fjrg" event={"ID":"4f0b0d35-65b6-4721-8939-8d2d6dbb8fc5","Type":"ContainerDied","Data":"5f4c9210b74cb706168c1d5e32e749971cb46563c40a0c9342f8bed2956cc472"} Oct 30 21:13:28.997277 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:13:28.997134 5378 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-de53c2-fdf4f8fc5-8fjrg" event={"ID":"4f0b0d35-65b6-4721-8939-8d2d6dbb8fc5","Type":"ContainerDied","Data":"84c1ab1dd2b5659080fda374f62ae1c19e53db0e72977c1c7601565997120985"} Oct 30 21:13:28.997277 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:13:28.997159 5378 scope.go:122] "RemoveContainer" containerID="5f4c9210b74cb706168c1d5e32e749971cb46563c40a0c9342f8bed2956cc472" Oct 30 21:13:28.998738 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:13:28.998703 5378 generic.go:356] "Generic (PLEG): container finished" podID="d9131396-6236-4a21-8c77-0f78fcc851f8" containerID="d910afbb1d2150b7cbdb312e1517a33d01df6c475363dfee8ad46c9fbf4f0c17" exitCode=0 Oct 30 21:13:28.998864 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:13:28.998736 5378 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-de53c2-6dc7f4bf96-b5cbp" event={"ID":"d9131396-6236-4a21-8c77-0f78fcc851f8","Type":"ContainerDied","Data":"d910afbb1d2150b7cbdb312e1517a33d01df6c475363dfee8ad46c9fbf4f0c17"} Oct 30 21:13:28.998864 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:13:28.998765 5378 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-de53c2-6dc7f4bf96-b5cbp" Oct 30 21:13:28.998864 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:13:28.998788 5378 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-de53c2-6dc7f4bf96-b5cbp" event={"ID":"d9131396-6236-4a21-8c77-0f78fcc851f8","Type":"ContainerDied","Data":"d0b614b6f2d876c78da6098ef42d28727c39136acda5fd7b03e2190307be4b97"} Oct 30 21:13:29.009390 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:13:29.009375 5378 scope.go:122] "RemoveContainer" containerID="5f4c9210b74cb706168c1d5e32e749971cb46563c40a0c9342f8bed2956cc472" Oct 30 21:13:29.009931 ip-10-0-138-149 kubenswrapper[5378]: E1030 21:13:29.009904 5378 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5f4c9210b74cb706168c1d5e32e749971cb46563c40a0c9342f8bed2956cc472\": container with ID starting with 5f4c9210b74cb706168c1d5e32e749971cb46563c40a0c9342f8bed2956cc472 not found: ID does not exist" containerID="5f4c9210b74cb706168c1d5e32e749971cb46563c40a0c9342f8bed2956cc472" Oct 30 21:13:29.010108 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:13:29.009939 5378 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5f4c9210b74cb706168c1d5e32e749971cb46563c40a0c9342f8bed2956cc472"} err="failed to get container status \"5f4c9210b74cb706168c1d5e32e749971cb46563c40a0c9342f8bed2956cc472\": rpc error: code = NotFound desc = could not find container \"5f4c9210b74cb706168c1d5e32e749971cb46563c40a0c9342f8bed2956cc472\": container with ID starting with 5f4c9210b74cb706168c1d5e32e749971cb46563c40a0c9342f8bed2956cc472 not found: ID does not exist" Oct 30 21:13:29.010108 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:13:29.009961 5378 scope.go:122] "RemoveContainer" containerID="d910afbb1d2150b7cbdb312e1517a33d01df6c475363dfee8ad46c9fbf4f0c17" Oct 30 21:13:29.021776 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:13:29.021746 5378 scope.go:122] "RemoveContainer" containerID="d910afbb1d2150b7cbdb312e1517a33d01df6c475363dfee8ad46c9fbf4f0c17" Oct 30 21:13:29.022046 ip-10-0-138-149 kubenswrapper[5378]: E1030 21:13:29.022019 5378 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d910afbb1d2150b7cbdb312e1517a33d01df6c475363dfee8ad46c9fbf4f0c17\": container with ID starting with d910afbb1d2150b7cbdb312e1517a33d01df6c475363dfee8ad46c9fbf4f0c17 not found: ID does not exist" containerID="d910afbb1d2150b7cbdb312e1517a33d01df6c475363dfee8ad46c9fbf4f0c17" Oct 30 21:13:29.022106 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:13:29.022047 5378 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d910afbb1d2150b7cbdb312e1517a33d01df6c475363dfee8ad46c9fbf4f0c17"} err="failed to get container status \"d910afbb1d2150b7cbdb312e1517a33d01df6c475363dfee8ad46c9fbf4f0c17\": rpc error: code = NotFound desc = could not find container \"d910afbb1d2150b7cbdb312e1517a33d01df6c475363dfee8ad46c9fbf4f0c17\": container with ID starting with d910afbb1d2150b7cbdb312e1517a33d01df6c475363dfee8ad46c9fbf4f0c17 not found: ID does not exist" Oct 30 21:13:29.169274 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:13:29.169249 5378 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-de53c2-6dc7f4bf96-b5cbp"] Oct 30 21:13:29.175075 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:13:29.175048 5378 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-de53c2-6dc7f4bf96-b5cbp"] Oct 30 21:13:29.287131 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:13:29.287049 5378 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-de53c2-fdf4f8fc5-8fjrg"] Oct 30 21:13:29.294094 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:13:29.294065 5378 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-de53c2-fdf4f8fc5-8fjrg"] Oct 30 21:13:29.317272 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:13:29.317233 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:13:29.744878 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:13:29.744849 5378 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4f0b0d35-65b6-4721-8939-8d2d6dbb8fc5" path="/var/lib/kubelet/pods/4f0b0d35-65b6-4721-8939-8d2d6dbb8fc5/volumes" Oct 30 21:13:29.745318 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:13:29.745303 5378 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d9131396-6236-4a21-8c77-0f78fcc851f8" path="/var/lib/kubelet/pods/d9131396-6236-4a21-8c77-0f78fcc851f8/volumes" Oct 30 21:13:30.021312 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:13:30.021236 5378 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-7x9jg_c2ff359c-cf92-4a5b-8610-f6eb41acbe73/ovn-acl-logging/0.log" Oct 30 21:13:30.032884 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:13:30.032860 5378 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-7x9jg_c2ff359c-cf92-4a5b-8610-f6eb41acbe73/ovn-acl-logging/0.log" Oct 30 21:13:34.654589 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:13:34.652167 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:13:39.893028 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:13:39.892986 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:13:44.389595 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:13:44.389544 5378 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-dc6125-5854c458b7-gdqwh"] Oct 30 21:13:44.390012 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:13:44.389784 5378 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-dc6125-5854c458b7-gdqwh" podUID="7e2d9d3a-d5d8-4187-bf82-2b451b8224f4" containerName="maas-api" containerID="cri-o://edea6ba8e081374d3f91c3f3b33c5ce72ddaf2ab7e6744e582def8210a053210" gracePeriod=30 Oct 30 21:13:44.539518 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:13:44.539481 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:13:44.757675 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:13:44.757639 5378 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-dc6125-5854c458b7-gdqwh" Oct 30 21:13:44.908586 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:13:44.905300 5378 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7e2d9d3a-d5d8-4187-bf82-2b451b8224f4-kube-api-access-qxhjr\" (UniqueName: \"kubernetes.io/projected/7e2d9d3a-d5d8-4187-bf82-2b451b8224f4-kube-api-access-qxhjr\") pod \"7e2d9d3a-d5d8-4187-bf82-2b451b8224f4\" (UID: \"7e2d9d3a-d5d8-4187-bf82-2b451b8224f4\") " Oct 30 21:13:44.908586 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:13:44.905447 5378 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7e2d9d3a-d5d8-4187-bf82-2b451b8224f4-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7e2d9d3a-d5d8-4187-bf82-2b451b8224f4-maas-api-tls\") pod \"7e2d9d3a-d5d8-4187-bf82-2b451b8224f4\" (UID: \"7e2d9d3a-d5d8-4187-bf82-2b451b8224f4\") " Oct 30 21:13:44.908586 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:13:44.905548 5378 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7e2d9d3a-d5d8-4187-bf82-2b451b8224f4-metrics-certs\" (UniqueName: \"kubernetes.io/secret/7e2d9d3a-d5d8-4187-bf82-2b451b8224f4-metrics-certs\") pod \"7e2d9d3a-d5d8-4187-bf82-2b451b8224f4\" (UID: \"7e2d9d3a-d5d8-4187-bf82-2b451b8224f4\") " Oct 30 21:13:44.908586 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:13:44.905665 5378 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7e2d9d3a-d5d8-4187-bf82-2b451b8224f4-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7e2d9d3a-d5d8-4187-bf82-2b451b8224f4-openshift-service-ca\") pod \"7e2d9d3a-d5d8-4187-bf82-2b451b8224f4\" (UID: \"7e2d9d3a-d5d8-4187-bf82-2b451b8224f4\") " Oct 30 21:13:44.916824 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:13:44.916787 5378 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7e2d9d3a-d5d8-4187-bf82-2b451b8224f4-openshift-service-ca" pod "7e2d9d3a-d5d8-4187-bf82-2b451b8224f4" (UID: "7e2d9d3a-d5d8-4187-bf82-2b451b8224f4"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:13:44.986740 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:13:44.986686 5378 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7e2d9d3a-d5d8-4187-bf82-2b451b8224f4-metrics-certs" pod "7e2d9d3a-d5d8-4187-bf82-2b451b8224f4" (UID: "7e2d9d3a-d5d8-4187-bf82-2b451b8224f4"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:13:44.987035 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:13:44.987015 5378 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7e2d9d3a-d5d8-4187-bf82-2b451b8224f4-maas-api-tls" pod "7e2d9d3a-d5d8-4187-bf82-2b451b8224f4" (UID: "7e2d9d3a-d5d8-4187-bf82-2b451b8224f4"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:13:44.987826 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:13:44.987764 5378 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7e2d9d3a-d5d8-4187-bf82-2b451b8224f4-kube-api-access-qxhjr" pod "7e2d9d3a-d5d8-4187-bf82-2b451b8224f4" (UID: "7e2d9d3a-d5d8-4187-bf82-2b451b8224f4"). InnerVolumeSpecName "kube-api-access-qxhjr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:13:45.008260 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:13:45.008217 5378 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qxhjr\" (UniqueName: \"kubernetes.io/projected/7e2d9d3a-d5d8-4187-bf82-2b451b8224f4-kube-api-access-qxhjr\") on node \"ip-10-0-138-149.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:13:45.008469 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:13:45.008451 5378 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7e2d9d3a-d5d8-4187-bf82-2b451b8224f4-maas-api-tls\") on node \"ip-10-0-138-149.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:13:45.008587 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:13:45.008575 5378 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7e2d9d3a-d5d8-4187-bf82-2b451b8224f4-metrics-certs\") on node \"ip-10-0-138-149.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:13:45.008726 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:13:45.008713 5378 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7e2d9d3a-d5d8-4187-bf82-2b451b8224f4-openshift-service-ca\") on node \"ip-10-0-138-149.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:13:45.119342 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:13:45.119310 5378 generic.go:356] "Generic (PLEG): container finished" podID="7e2d9d3a-d5d8-4187-bf82-2b451b8224f4" containerID="edea6ba8e081374d3f91c3f3b33c5ce72ddaf2ab7e6744e582def8210a053210" exitCode=0 Oct 30 21:13:45.119518 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:13:45.119352 5378 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-dc6125-5854c458b7-gdqwh" Oct 30 21:13:45.119518 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:13:45.119386 5378 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-dc6125-5854c458b7-gdqwh" event={"ID":"7e2d9d3a-d5d8-4187-bf82-2b451b8224f4","Type":"ContainerDied","Data":"edea6ba8e081374d3f91c3f3b33c5ce72ddaf2ab7e6744e582def8210a053210"} Oct 30 21:13:45.119518 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:13:45.119445 5378 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-dc6125-5854c458b7-gdqwh" event={"ID":"7e2d9d3a-d5d8-4187-bf82-2b451b8224f4","Type":"ContainerDied","Data":"343ff3e67bfd250e64bf81a07a17e4505bdd5d1f883c7dce6de84f634f4e21c7"} Oct 30 21:13:45.119518 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:13:45.119468 5378 scope.go:122] "RemoveContainer" containerID="edea6ba8e081374d3f91c3f3b33c5ce72ddaf2ab7e6744e582def8210a053210" Oct 30 21:13:45.131286 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:13:45.131266 5378 scope.go:122] "RemoveContainer" containerID="edea6ba8e081374d3f91c3f3b33c5ce72ddaf2ab7e6744e582def8210a053210" Oct 30 21:13:45.131536 ip-10-0-138-149 kubenswrapper[5378]: E1030 21:13:45.131516 5378 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"edea6ba8e081374d3f91c3f3b33c5ce72ddaf2ab7e6744e582def8210a053210\": container with ID starting with edea6ba8e081374d3f91c3f3b33c5ce72ddaf2ab7e6744e582def8210a053210 not found: ID does not exist" containerID="edea6ba8e081374d3f91c3f3b33c5ce72ddaf2ab7e6744e582def8210a053210" Oct 30 21:13:45.131643 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:13:45.131544 5378 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"edea6ba8e081374d3f91c3f3b33c5ce72ddaf2ab7e6744e582def8210a053210"} err="failed to get container status \"edea6ba8e081374d3f91c3f3b33c5ce72ddaf2ab7e6744e582def8210a053210\": rpc error: code = NotFound desc = could not find container \"edea6ba8e081374d3f91c3f3b33c5ce72ddaf2ab7e6744e582def8210a053210\": container with ID starting with edea6ba8e081374d3f91c3f3b33c5ce72ddaf2ab7e6744e582def8210a053210 not found: ID does not exist" Oct 30 21:13:45.291094 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:13:45.291021 5378 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-dc6125-5854c458b7-gdqwh"] Oct 30 21:13:45.340498 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:13:45.340457 5378 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-dc6125-5854c458b7-gdqwh"] Oct 30 21:13:45.744266 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:13:45.744223 5378 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7e2d9d3a-d5d8-4187-bf82-2b451b8224f4" path="/var/lib/kubelet/pods/7e2d9d3a-d5d8-4187-bf82-2b451b8224f4/volumes" Oct 30 21:13:46.224911 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:13:46.224876 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:13:46.521861 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:13:46.521759 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:13:46.673092 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:13:46.673039 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:13:47.235273 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:13:47.235236 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:13:48.116410 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:13:48.116364 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:13:48.848437 ip-10-0-138-149 kubenswrapper[5378]: E1030 21:13:48.848385 5378 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q6mx7" containerName="extractor" Oct 30 21:13:51.059859 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:13:51.059827 5378 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-de53c2-697d9756c6-t4xvj"] Oct 30 21:13:51.060356 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:13:51.060066 5378 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-de53c2-697d9756c6-t4xvj" podUID="b3981dd6-e944-47d1-83a4-1b4689d74bcb" containerName="maas-api" containerID="cri-o://2eb97482976b47870badffb3d92c28732a708331b9ed83781b7f261f78cb7a25" gracePeriod=30 Oct 30 21:13:51.178438 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:13:51.178415 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:13:51.417926 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:13:51.417897 5378 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-de53c2-697d9756c6-t4xvj" Oct 30 21:13:51.592441 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:13:51.590032 5378 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b3981dd6-e944-47d1-83a4-1b4689d74bcb-kube-api-access-tqqtq\" (UniqueName: \"kubernetes.io/projected/b3981dd6-e944-47d1-83a4-1b4689d74bcb-kube-api-access-tqqtq\") pod \"b3981dd6-e944-47d1-83a4-1b4689d74bcb\" (UID: \"b3981dd6-e944-47d1-83a4-1b4689d74bcb\") " Oct 30 21:13:51.592441 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:13:51.590135 5378 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b3981dd6-e944-47d1-83a4-1b4689d74bcb-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b3981dd6-e944-47d1-83a4-1b4689d74bcb-maas-api-tls\") pod \"b3981dd6-e944-47d1-83a4-1b4689d74bcb\" (UID: \"b3981dd6-e944-47d1-83a4-1b4689d74bcb\") " Oct 30 21:13:51.592441 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:13:51.590411 5378 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b3981dd6-e944-47d1-83a4-1b4689d74bcb-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b3981dd6-e944-47d1-83a4-1b4689d74bcb-openshift-service-ca\") pod \"b3981dd6-e944-47d1-83a4-1b4689d74bcb\" (UID: \"b3981dd6-e944-47d1-83a4-1b4689d74bcb\") " Oct 30 21:13:51.592441 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:13:51.590471 5378 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b3981dd6-e944-47d1-83a4-1b4689d74bcb-metrics-certs\" (UniqueName: \"kubernetes.io/secret/b3981dd6-e944-47d1-83a4-1b4689d74bcb-metrics-certs\") pod \"b3981dd6-e944-47d1-83a4-1b4689d74bcb\" (UID: \"b3981dd6-e944-47d1-83a4-1b4689d74bcb\") " Oct 30 21:13:51.592441 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:13:51.591591 5378 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b3981dd6-e944-47d1-83a4-1b4689d74bcb-openshift-service-ca" pod "b3981dd6-e944-47d1-83a4-1b4689d74bcb" (UID: "b3981dd6-e944-47d1-83a4-1b4689d74bcb"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:13:51.600433 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:13:51.600363 5378 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b3981dd6-e944-47d1-83a4-1b4689d74bcb-openshift-service-ca\") on node \"ip-10-0-138-149.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:13:51.620180 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:13:51.620143 5378 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b3981dd6-e944-47d1-83a4-1b4689d74bcb-metrics-certs" pod "b3981dd6-e944-47d1-83a4-1b4689d74bcb" (UID: "b3981dd6-e944-47d1-83a4-1b4689d74bcb"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:13:51.620362 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:13:51.620257 5378 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b3981dd6-e944-47d1-83a4-1b4689d74bcb-maas-api-tls" pod "b3981dd6-e944-47d1-83a4-1b4689d74bcb" (UID: "b3981dd6-e944-47d1-83a4-1b4689d74bcb"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:13:51.620613 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:13:51.620582 5378 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b3981dd6-e944-47d1-83a4-1b4689d74bcb-kube-api-access-tqqtq" pod "b3981dd6-e944-47d1-83a4-1b4689d74bcb" (UID: "b3981dd6-e944-47d1-83a4-1b4689d74bcb"). InnerVolumeSpecName "kube-api-access-tqqtq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:13:51.701888 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:13:51.701852 5378 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b3981dd6-e944-47d1-83a4-1b4689d74bcb-metrics-certs\") on node \"ip-10-0-138-149.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:13:51.701888 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:13:51.701887 5378 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tqqtq\" (UniqueName: \"kubernetes.io/projected/b3981dd6-e944-47d1-83a4-1b4689d74bcb-kube-api-access-tqqtq\") on node \"ip-10-0-138-149.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:13:51.702102 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:13:51.701897 5378 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b3981dd6-e944-47d1-83a4-1b4689d74bcb-maas-api-tls\") on node \"ip-10-0-138-149.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:13:51.862397 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:13:51.862360 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:13:52.019633 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:13:52.019498 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:13:52.289653 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:13:52.288900 5378 generic.go:356] "Generic (PLEG): container finished" podID="b3981dd6-e944-47d1-83a4-1b4689d74bcb" containerID="2eb97482976b47870badffb3d92c28732a708331b9ed83781b7f261f78cb7a25" exitCode=0 Oct 30 21:13:52.289653 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:13:52.289058 5378 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-de53c2-697d9756c6-t4xvj" event={"ID":"b3981dd6-e944-47d1-83a4-1b4689d74bcb","Type":"ContainerDied","Data":"2eb97482976b47870badffb3d92c28732a708331b9ed83781b7f261f78cb7a25"} Oct 30 21:13:52.289653 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:13:52.289115 5378 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-de53c2-697d9756c6-t4xvj" event={"ID":"b3981dd6-e944-47d1-83a4-1b4689d74bcb","Type":"ContainerDied","Data":"fd4073727d235cca1689eb406ded5358e4bb3853ebfad97963b4604f8590bf1f"} Oct 30 21:13:52.289653 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:13:52.289148 5378 scope.go:122] "RemoveContainer" containerID="2eb97482976b47870badffb3d92c28732a708331b9ed83781b7f261f78cb7a25" Oct 30 21:13:52.289653 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:13:52.289369 5378 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-de53c2-697d9756c6-t4xvj" Oct 30 21:13:52.342086 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:13:52.341604 5378 scope.go:122] "RemoveContainer" containerID="2eb97482976b47870badffb3d92c28732a708331b9ed83781b7f261f78cb7a25" Oct 30 21:13:52.342086 ip-10-0-138-149 kubenswrapper[5378]: E1030 21:13:52.341994 5378 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2eb97482976b47870badffb3d92c28732a708331b9ed83781b7f261f78cb7a25\": container with ID starting with 2eb97482976b47870badffb3d92c28732a708331b9ed83781b7f261f78cb7a25 not found: ID does not exist" containerID="2eb97482976b47870badffb3d92c28732a708331b9ed83781b7f261f78cb7a25" Oct 30 21:13:52.342086 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:13:52.342027 5378 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2eb97482976b47870badffb3d92c28732a708331b9ed83781b7f261f78cb7a25"} err="failed to get container status \"2eb97482976b47870badffb3d92c28732a708331b9ed83781b7f261f78cb7a25\": rpc error: code = NotFound desc = could not find container \"2eb97482976b47870badffb3d92c28732a708331b9ed83781b7f261f78cb7a25\": container with ID starting with 2eb97482976b47870badffb3d92c28732a708331b9ed83781b7f261f78cb7a25 not found: ID does not exist" Oct 30 21:13:52.465680 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:13:52.465648 5378 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-de53c2-697d9756c6-t4xvj"] Oct 30 21:13:52.516379 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:13:52.516346 5378 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-de53c2-697d9756c6-t4xvj"] Oct 30 21:13:52.618413 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:13:52.618379 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:13:53.030505 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:13:53.030427 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:13:53.490589 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:13:53.489676 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:13:53.818652 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:13:53.818421 5378 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b3981dd6-e944-47d1-83a4-1b4689d74bcb" path="/var/lib/kubelet/pods/b3981dd6-e944-47d1-83a4-1b4689d74bcb/volumes" Oct 30 21:13:54.202969 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:13:54.202931 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:13:54.569786 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:13:54.569700 5378 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-390b71-64988c8666-xfnmq"] Oct 30 21:13:54.570595 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:13:54.570540 5378 cpu_manager.go:414] "containerMap: removing container" podUID="b3981dd6-e944-47d1-83a4-1b4689d74bcb" containerName="maas-api" Oct 30 21:13:54.570595 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:13:54.570575 5378 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b3981dd6-e944-47d1-83a4-1b4689d74bcb" containerName="maas-api" Oct 30 21:13:54.570595 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:13:54.570590 5378 cpu_manager.go:414] "containerMap: removing container" podUID="44227101-a97a-4bc9-a5b0-fd32cc961337" containerName="istio-proxy" Oct 30 21:13:54.570595 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:13:54.570597 5378 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="44227101-a97a-4bc9-a5b0-fd32cc961337" containerName="istio-proxy" Oct 30 21:13:54.570769 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:13:54.570608 5378 cpu_manager.go:414] "containerMap: removing container" podUID="7e2d9d3a-d5d8-4187-bf82-2b451b8224f4" containerName="maas-api" Oct 30 21:13:54.570769 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:13:54.570613 5378 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7e2d9d3a-d5d8-4187-bf82-2b451b8224f4" containerName="maas-api" Oct 30 21:13:54.570769 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:13:54.570621 5378 cpu_manager.go:414] "containerMap: removing container" podUID="5a5302ee-2e3c-427b-a222-940f0a913372" containerName="payload-processing" Oct 30 21:13:54.570769 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:13:54.570626 5378 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5a5302ee-2e3c-427b-a222-940f0a913372" containerName="payload-processing" Oct 30 21:13:54.570769 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:13:54.570636 5378 cpu_manager.go:414] "containerMap: removing container" podUID="168599a3-7daf-4eba-a259-978486b56bd4" containerName="payload-processing" Oct 30 21:13:54.570769 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:13:54.570641 5378 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="168599a3-7daf-4eba-a259-978486b56bd4" containerName="payload-processing" Oct 30 21:13:54.570769 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:13:54.570654 5378 cpu_manager.go:414] "containerMap: removing container" podUID="34b71d39-551a-4d2e-bf08-8df0cf45fc9e" containerName="istio-proxy" Oct 30 21:13:54.570769 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:13:54.570659 5378 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="34b71d39-551a-4d2e-bf08-8df0cf45fc9e" containerName="istio-proxy" Oct 30 21:13:54.570769 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:13:54.570678 5378 cpu_manager.go:414] "containerMap: removing container" podUID="d9131396-6236-4a21-8c77-0f78fcc851f8" containerName="payload-processing" Oct 30 21:13:54.570769 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:13:54.570683 5378 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d9131396-6236-4a21-8c77-0f78fcc851f8" containerName="payload-processing" Oct 30 21:13:54.570769 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:13:54.570688 5378 cpu_manager.go:414] "containerMap: removing container" podUID="b8b5625b-b0ec-4e9c-8b80-338ea02008f9" containerName="payload-pre-processing" Oct 30 21:13:54.570769 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:13:54.570692 5378 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b8b5625b-b0ec-4e9c-8b80-338ea02008f9" containerName="payload-pre-processing" Oct 30 21:13:54.570769 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:13:54.570700 5378 cpu_manager.go:414] "containerMap: removing container" podUID="34b71d39-551a-4d2e-bf08-8df0cf45fc9e" containerName="istio-proxy" Oct 30 21:13:54.570769 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:13:54.570705 5378 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="34b71d39-551a-4d2e-bf08-8df0cf45fc9e" containerName="istio-proxy" Oct 30 21:13:54.570769 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:13:54.570722 5378 cpu_manager.go:414] "containerMap: removing container" podUID="5ff92f96-e5bc-4b04-893b-16efd49f9d3f" containerName="payload-pre-processing" Oct 30 21:13:54.570769 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:13:54.570726 5378 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5ff92f96-e5bc-4b04-893b-16efd49f9d3f" containerName="payload-pre-processing" Oct 30 21:13:54.570769 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:13:54.570740 5378 cpu_manager.go:414] "containerMap: removing container" podUID="4f0b0d35-65b6-4721-8939-8d2d6dbb8fc5" containerName="payload-pre-processing" Oct 30 21:13:54.570769 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:13:54.570744 5378 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4f0b0d35-65b6-4721-8939-8d2d6dbb8fc5" containerName="payload-pre-processing" Oct 30 21:13:54.571314 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:13:54.570842 5378 memory_manager.go:365] "RemoveStaleState removing state" podUID="b3981dd6-e944-47d1-83a4-1b4689d74bcb" containerName="maas-api" Oct 30 21:13:54.571314 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:13:54.570852 5378 memory_manager.go:365] "RemoveStaleState removing state" podUID="44227101-a97a-4bc9-a5b0-fd32cc961337" containerName="istio-proxy" Oct 30 21:13:54.571314 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:13:54.570860 5378 memory_manager.go:365] "RemoveStaleState removing state" podUID="7e2d9d3a-d5d8-4187-bf82-2b451b8224f4" containerName="maas-api" Oct 30 21:13:54.571314 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:13:54.570866 5378 memory_manager.go:365] "RemoveStaleState removing state" podUID="5a5302ee-2e3c-427b-a222-940f0a913372" containerName="payload-processing" Oct 30 21:13:54.571314 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:13:54.570874 5378 memory_manager.go:365] "RemoveStaleState removing state" podUID="168599a3-7daf-4eba-a259-978486b56bd4" containerName="payload-processing" Oct 30 21:13:54.571314 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:13:54.570880 5378 memory_manager.go:365] "RemoveStaleState removing state" podUID="34b71d39-551a-4d2e-bf08-8df0cf45fc9e" containerName="istio-proxy" Oct 30 21:13:54.571314 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:13:54.570889 5378 memory_manager.go:365] "RemoveStaleState removing state" podUID="d9131396-6236-4a21-8c77-0f78fcc851f8" containerName="payload-processing" Oct 30 21:13:54.571314 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:13:54.570895 5378 memory_manager.go:365] "RemoveStaleState removing state" podUID="b8b5625b-b0ec-4e9c-8b80-338ea02008f9" containerName="payload-pre-processing" Oct 30 21:13:54.571314 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:13:54.570901 5378 memory_manager.go:365] "RemoveStaleState removing state" podUID="34b71d39-551a-4d2e-bf08-8df0cf45fc9e" containerName="istio-proxy" Oct 30 21:13:54.571314 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:13:54.570909 5378 memory_manager.go:365] "RemoveStaleState removing state" podUID="5ff92f96-e5bc-4b04-893b-16efd49f9d3f" containerName="payload-pre-processing" Oct 30 21:13:54.571314 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:13:54.570917 5378 memory_manager.go:365] "RemoveStaleState removing state" podUID="4f0b0d35-65b6-4721-8939-8d2d6dbb8fc5" containerName="payload-pre-processing" Oct 30 21:13:54.575867 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:13:54.575849 5378 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-390b71-64988c8666-xfnmq" Oct 30 21:13:54.601264 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:13:54.601233 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:13:54.651104 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:13:54.651069 5378 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-390b71-64988c8666-xfnmq"] Oct 30 21:13:54.651391 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:13:54.651358 5378 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1a34eaf4-4b44-4e51-b497-eada6dd5a4fd-openshift-service-ca\") pod \"maas-api-e2e-inf-a-390b71-64988c8666-xfnmq\" (UID: \"1a34eaf4-4b44-4e51-b497-eada6dd5a4fd\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-390b71-64988c8666-xfnmq" Oct 30 21:13:54.651524 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:13:54.651505 5378 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1a34eaf4-4b44-4e51-b497-eada6dd5a4fd-maas-api-tls\") pod \"maas-api-e2e-inf-a-390b71-64988c8666-xfnmq\" (UID: \"1a34eaf4-4b44-4e51-b497-eada6dd5a4fd\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-390b71-64988c8666-xfnmq" Oct 30 21:13:54.651612 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:13:54.651555 5378 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1a34eaf4-4b44-4e51-b497-eada6dd5a4fd-metrics-certs\") pod \"maas-api-e2e-inf-a-390b71-64988c8666-xfnmq\" (UID: \"1a34eaf4-4b44-4e51-b497-eada6dd5a4fd\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-390b71-64988c8666-xfnmq" Oct 30 21:13:54.651723 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:13:54.651709 5378 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-skhkx\" (UniqueName: \"kubernetes.io/projected/1a34eaf4-4b44-4e51-b497-eada6dd5a4fd-kube-api-access-skhkx\") pod \"maas-api-e2e-inf-a-390b71-64988c8666-xfnmq\" (UID: \"1a34eaf4-4b44-4e51-b497-eada6dd5a4fd\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-390b71-64988c8666-xfnmq" Oct 30 21:13:54.677751 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:13:54.677718 5378 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-a-390b71-serving-cert\"" Oct 30 21:13:54.753200 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:13:54.753163 5378 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-skhkx\" (UniqueName: \"kubernetes.io/projected/1a34eaf4-4b44-4e51-b497-eada6dd5a4fd-kube-api-access-skhkx\") pod \"maas-api-e2e-inf-a-390b71-64988c8666-xfnmq\" (UID: \"1a34eaf4-4b44-4e51-b497-eada6dd5a4fd\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-390b71-64988c8666-xfnmq" Oct 30 21:13:54.753395 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:13:54.753339 5378 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1a34eaf4-4b44-4e51-b497-eada6dd5a4fd-openshift-service-ca\") pod \"maas-api-e2e-inf-a-390b71-64988c8666-xfnmq\" (UID: \"1a34eaf4-4b44-4e51-b497-eada6dd5a4fd\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-390b71-64988c8666-xfnmq" Oct 30 21:13:54.754098 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:13:54.754069 5378 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1a34eaf4-4b44-4e51-b497-eada6dd5a4fd-openshift-service-ca\") pod \"maas-api-e2e-inf-a-390b71-64988c8666-xfnmq\" (UID: \"1a34eaf4-4b44-4e51-b497-eada6dd5a4fd\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-390b71-64988c8666-xfnmq" Oct 30 21:13:54.754234 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:13:54.754189 5378 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1a34eaf4-4b44-4e51-b497-eada6dd5a4fd-maas-api-tls\") pod \"maas-api-e2e-inf-a-390b71-64988c8666-xfnmq\" (UID: \"1a34eaf4-4b44-4e51-b497-eada6dd5a4fd\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-390b71-64988c8666-xfnmq" Oct 30 21:13:54.754287 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:13:54.754237 5378 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1a34eaf4-4b44-4e51-b497-eada6dd5a4fd-metrics-certs\") pod \"maas-api-e2e-inf-a-390b71-64988c8666-xfnmq\" (UID: \"1a34eaf4-4b44-4e51-b497-eada6dd5a4fd\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-390b71-64988c8666-xfnmq" Oct 30 21:13:54.754811 ip-10-0-138-149 kubenswrapper[5378]: E1030 21:13:54.754792 5378 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-inf-a-390b71-serving-cert: secret "maas-api-e2e-inf-a-390b71-serving-cert" not found Oct 30 21:13:54.754866 ip-10-0-138-149 kubenswrapper[5378]: E1030 21:13:54.754855 5378 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1a34eaf4-4b44-4e51-b497-eada6dd5a4fd-maas-api-tls podName:1a34eaf4-4b44-4e51-b497-eada6dd5a4fd nodeName:}" failed. No retries permitted until 2026-10-30 21:13:55.254837559 +0000 UTC m=+2125.682951462 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/1a34eaf4-4b44-4e51-b497-eada6dd5a4fd-maas-api-tls") pod "maas-api-e2e-inf-a-390b71-64988c8666-xfnmq" (UID: "1a34eaf4-4b44-4e51-b497-eada6dd5a4fd") : secret "maas-api-e2e-inf-a-390b71-serving-cert" not found Oct 30 21:13:54.757948 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:13:54.757926 5378 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1a34eaf4-4b44-4e51-b497-eada6dd5a4fd-metrics-certs\") pod \"maas-api-e2e-inf-a-390b71-64988c8666-xfnmq\" (UID: \"1a34eaf4-4b44-4e51-b497-eada6dd5a4fd\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-390b71-64988c8666-xfnmq" Oct 30 21:13:54.772606 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:13:54.772552 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:13:54.879588 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:13:54.875595 5378 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-skhkx\" (UniqueName: \"kubernetes.io/projected/1a34eaf4-4b44-4e51-b497-eada6dd5a4fd-kube-api-access-skhkx\") pod \"maas-api-e2e-inf-a-390b71-64988c8666-xfnmq\" (UID: \"1a34eaf4-4b44-4e51-b497-eada6dd5a4fd\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-390b71-64988c8666-xfnmq" Oct 30 21:13:55.069053 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:13:55.069004 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:13:55.237151 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:13:55.237061 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:13:55.279105 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:13:55.279069 5378 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1a34eaf4-4b44-4e51-b497-eada6dd5a4fd-maas-api-tls\") pod \"maas-api-e2e-inf-a-390b71-64988c8666-xfnmq\" (UID: \"1a34eaf4-4b44-4e51-b497-eada6dd5a4fd\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-390b71-64988c8666-xfnmq" Oct 30 21:13:55.281498 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:13:55.281476 5378 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1a34eaf4-4b44-4e51-b497-eada6dd5a4fd-maas-api-tls\") pod \"maas-api-e2e-inf-a-390b71-64988c8666-xfnmq\" (UID: \"1a34eaf4-4b44-4e51-b497-eada6dd5a4fd\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-390b71-64988c8666-xfnmq" Oct 30 21:13:55.514345 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:13:55.512908 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:13:55.514345 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:13:55.513691 5378 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-390b71-64988c8666-xfnmq" Oct 30 21:13:55.660159 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:13:55.660120 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:13:55.770996 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:13:55.770897 5378 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w6-dc6125-openshift-default-659bcc49bf-ms7zg"] Oct 30 21:13:55.771168 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:13:55.771130 5378 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w6-dc6125-openshift-default-659bcc49bf-ms7zg" podUID="d3278471-08a8-4d1e-8030-c6256d75265d" containerName="istio-proxy" containerID="cri-o://131c50322efa7a5ed2cdfe258fb864f7e2a5f175926e843d96445986a020710f" gracePeriod=30 Oct 30 21:13:55.931735 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:13:55.931702 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:13:55.999683 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:13:55.996721 5378 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-390b71-64988c8666-xfnmq"] Oct 30 21:13:56.072194 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:13:56.071403 5378 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 30 21:13:56.389155 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:13:56.389118 5378 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-390b71-64988c8666-xfnmq" event={"ID":"1a34eaf4-4b44-4e51-b497-eada6dd5a4fd","Type":"ContainerStarted","Data":"c87b3a1240cd2d3a130b8cb71b568bf11633db536dd3a49f9c10a2ef82e9901f"} Oct 30 21:13:57.397515 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:13:57.397423 5378 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-390b71-64988c8666-xfnmq" event={"ID":"1a34eaf4-4b44-4e51-b497-eada6dd5a4fd","Type":"ContainerStarted","Data":"02e66a089d3895d4ae17d332127b2a1c579141bb1f755ea4ef8c3728a36e2cf6"} Oct 30 21:13:57.397515 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:13:57.397498 5378 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-390b71-64988c8666-xfnmq" Oct 30 21:13:57.523509 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:13:57.523457 5378 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-390b71-64988c8666-xfnmq" podStartSLOduration=2.478128077 podStartE2EDuration="3.523443257s" totalImagesPullingTime="1.04531518s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:13:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:13:57.519908753 +0000 UTC m=+2127.948022671" watchObservedRunningTime="2026-10-30 21:13:57.523443257 +0000 UTC m=+2127.951557177" Oct 30 21:14:00.469218 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:14:00.469186 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:14:00.693761 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:14:00.693702 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:14:00.832867 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:14:00.832840 5378 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-de53c2-openshift-default-58f78dcbb7-lxxs8"] Oct 30 21:14:00.833055 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:14:00.833035 5378 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w2-de53c2-openshift-default-58f78dcbb7-lxxs8" podUID="a5fd3bfb-32af-4284-9bed-e0e55a9dacd3" containerName="istio-proxy" containerID="cri-o://8bb6aa694c2f2f98db299ea198a6a5ac67e48d239e865c892c29e989901a4070" gracePeriod=30 Oct 30 21:14:00.843219 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:14:00.843178 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:14:01.285509 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:14:01.285482 5378 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w6-dc6125-openshift-default-659bcc49bf-ms7zg" Oct 30 21:14:01.427664 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:14:01.427574 5378 generic.go:356] "Generic (PLEG): container finished" podID="d3278471-08a8-4d1e-8030-c6256d75265d" containerID="131c50322efa7a5ed2cdfe258fb864f7e2a5f175926e843d96445986a020710f" exitCode=0 Oct 30 21:14:01.427664 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:14:01.427648 5378 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w6-dc6125-openshift-default-659bcc49bf-ms7zg" Oct 30 21:14:01.427905 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:14:01.427687 5378 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w6-dc6125-openshift-default-659bcc49bf-ms7zg" event={"ID":"d3278471-08a8-4d1e-8030-c6256d75265d","Type":"ContainerDied","Data":"131c50322efa7a5ed2cdfe258fb864f7e2a5f175926e843d96445986a020710f"} Oct 30 21:14:01.427905 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:14:01.427737 5378 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w6-dc6125-openshift-default-659bcc49bf-ms7zg" event={"ID":"d3278471-08a8-4d1e-8030-c6256d75265d","Type":"ContainerDied","Data":"c1cc2cdbb15e6bbf444201e3f80e48d92e9e2577b02b79f902acfd29b1a7c9fc"} Oct 30 21:14:01.427905 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:14:01.427764 5378 scope.go:122] "RemoveContainer" containerID="131c50322efa7a5ed2cdfe258fb864f7e2a5f175926e843d96445986a020710f" Oct 30 21:14:01.436477 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:14:01.436444 5378 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d3278471-08a8-4d1e-8030-c6256d75265d-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d3278471-08a8-4d1e-8030-c6256d75265d-workload-certs\") pod \"d3278471-08a8-4d1e-8030-c6256d75265d\" (UID: \"d3278471-08a8-4d1e-8030-c6256d75265d\") " Oct 30 21:14:01.436671 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:14:01.436510 5378 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d3278471-08a8-4d1e-8030-c6256d75265d-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d3278471-08a8-4d1e-8030-c6256d75265d-istiod-ca-cert\") pod \"d3278471-08a8-4d1e-8030-c6256d75265d\" (UID: \"d3278471-08a8-4d1e-8030-c6256d75265d\") " Oct 30 21:14:01.436671 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:14:01.436598 5378 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d3278471-08a8-4d1e-8030-c6256d75265d-istio-token\" (UniqueName: \"kubernetes.io/projected/d3278471-08a8-4d1e-8030-c6256d75265d-istio-token\") pod \"d3278471-08a8-4d1e-8030-c6256d75265d\" (UID: \"d3278471-08a8-4d1e-8030-c6256d75265d\") " Oct 30 21:14:01.436671 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:14:01.436638 5378 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d3278471-08a8-4d1e-8030-c6256d75265d-kube-api-access-nm92g\" (UniqueName: \"kubernetes.io/projected/d3278471-08a8-4d1e-8030-c6256d75265d-kube-api-access-nm92g\") pod \"d3278471-08a8-4d1e-8030-c6256d75265d\" (UID: \"d3278471-08a8-4d1e-8030-c6256d75265d\") " Oct 30 21:14:01.436796 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:14:01.436677 5378 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d3278471-08a8-4d1e-8030-c6256d75265d-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d3278471-08a8-4d1e-8030-c6256d75265d-istio-envoy\") pod \"d3278471-08a8-4d1e-8030-c6256d75265d\" (UID: \"d3278471-08a8-4d1e-8030-c6256d75265d\") " Oct 30 21:14:01.436796 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:14:01.436722 5378 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d3278471-08a8-4d1e-8030-c6256d75265d-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d3278471-08a8-4d1e-8030-c6256d75265d-workload-socket\") pod \"d3278471-08a8-4d1e-8030-c6256d75265d\" (UID: \"d3278471-08a8-4d1e-8030-c6256d75265d\") " Oct 30 21:14:01.436867 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:14:01.436850 5378 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d3278471-08a8-4d1e-8030-c6256d75265d-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d3278471-08a8-4d1e-8030-c6256d75265d-credential-socket\") pod \"d3278471-08a8-4d1e-8030-c6256d75265d\" (UID: \"d3278471-08a8-4d1e-8030-c6256d75265d\") " Oct 30 21:14:01.436934 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:14:01.436920 5378 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/d3278471-08a8-4d1e-8030-c6256d75265d-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d3278471-08a8-4d1e-8030-c6256d75265d-istio-podinfo\") pod \"d3278471-08a8-4d1e-8030-c6256d75265d\" (UID: \"d3278471-08a8-4d1e-8030-c6256d75265d\") " Oct 30 21:14:01.436980 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:14:01.436966 5378 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d3278471-08a8-4d1e-8030-c6256d75265d-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d3278471-08a8-4d1e-8030-c6256d75265d-istio-data\") pod \"d3278471-08a8-4d1e-8030-c6256d75265d\" (UID: \"d3278471-08a8-4d1e-8030-c6256d75265d\") " Oct 30 21:14:01.438469 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:14:01.438442 5378 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d3278471-08a8-4d1e-8030-c6256d75265d-istio-data" pod "d3278471-08a8-4d1e-8030-c6256d75265d" (UID: "d3278471-08a8-4d1e-8030-c6256d75265d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:14:01.439194 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:14:01.439085 5378 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d3278471-08a8-4d1e-8030-c6256d75265d-istiod-ca-cert" pod "d3278471-08a8-4d1e-8030-c6256d75265d" (UID: "d3278471-08a8-4d1e-8030-c6256d75265d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:14:01.439467 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:14:01.439332 5378 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d3278471-08a8-4d1e-8030-c6256d75265d-workload-certs" pod "d3278471-08a8-4d1e-8030-c6256d75265d" (UID: "d3278471-08a8-4d1e-8030-c6256d75265d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:14:01.440232 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:14:01.440011 5378 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d3278471-08a8-4d1e-8030-c6256d75265d-workload-socket" pod "d3278471-08a8-4d1e-8030-c6256d75265d" (UID: "d3278471-08a8-4d1e-8030-c6256d75265d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:14:01.440458 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:14:01.440392 5378 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d3278471-08a8-4d1e-8030-c6256d75265d-credential-socket" pod "d3278471-08a8-4d1e-8030-c6256d75265d" (UID: "d3278471-08a8-4d1e-8030-c6256d75265d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:14:01.442067 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:14:01.441380 5378 scope.go:122] "RemoveContainer" containerID="b66cc3f1cfa154e91f53a673aaac48f9fcd49cde853157e43d7fc96cae4a8cf5" Oct 30 21:14:01.442067 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:14:01.441448 5378 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d3278471-08a8-4d1e-8030-c6256d75265d-kube-api-access-nm92g" pod "d3278471-08a8-4d1e-8030-c6256d75265d" (UID: "d3278471-08a8-4d1e-8030-c6256d75265d"). InnerVolumeSpecName "kube-api-access-nm92g". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:14:01.442869 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:14:01.442816 5378 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d3278471-08a8-4d1e-8030-c6256d75265d-istio-envoy" pod "d3278471-08a8-4d1e-8030-c6256d75265d" (UID: "d3278471-08a8-4d1e-8030-c6256d75265d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:14:01.443290 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:14:01.443261 5378 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d3278471-08a8-4d1e-8030-c6256d75265d-istio-token" pod "d3278471-08a8-4d1e-8030-c6256d75265d" (UID: "d3278471-08a8-4d1e-8030-c6256d75265d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:14:01.444984 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:14:01.444959 5378 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d3278471-08a8-4d1e-8030-c6256d75265d-istio-podinfo" pod "d3278471-08a8-4d1e-8030-c6256d75265d" (UID: "d3278471-08a8-4d1e-8030-c6256d75265d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 21:14:01.473174 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:14:01.473150 5378 scope.go:122] "RemoveContainer" containerID="131c50322efa7a5ed2cdfe258fb864f7e2a5f175926e843d96445986a020710f" Oct 30 21:14:01.473494 ip-10-0-138-149 kubenswrapper[5378]: E1030 21:14:01.473473 5378 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"131c50322efa7a5ed2cdfe258fb864f7e2a5f175926e843d96445986a020710f\": container with ID starting with 131c50322efa7a5ed2cdfe258fb864f7e2a5f175926e843d96445986a020710f not found: ID does not exist" containerID="131c50322efa7a5ed2cdfe258fb864f7e2a5f175926e843d96445986a020710f" Oct 30 21:14:01.473543 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:14:01.473500 5378 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"131c50322efa7a5ed2cdfe258fb864f7e2a5f175926e843d96445986a020710f"} err="failed to get container status \"131c50322efa7a5ed2cdfe258fb864f7e2a5f175926e843d96445986a020710f\": rpc error: code = NotFound desc = could not find container \"131c50322efa7a5ed2cdfe258fb864f7e2a5f175926e843d96445986a020710f\": container with ID starting with 131c50322efa7a5ed2cdfe258fb864f7e2a5f175926e843d96445986a020710f not found: ID does not exist" Oct 30 21:14:01.473543 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:14:01.473520 5378 scope.go:122] "RemoveContainer" containerID="b66cc3f1cfa154e91f53a673aaac48f9fcd49cde853157e43d7fc96cae4a8cf5" Oct 30 21:14:01.473821 ip-10-0-138-149 kubenswrapper[5378]: E1030 21:14:01.473799 5378 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b66cc3f1cfa154e91f53a673aaac48f9fcd49cde853157e43d7fc96cae4a8cf5\": container with ID starting with b66cc3f1cfa154e91f53a673aaac48f9fcd49cde853157e43d7fc96cae4a8cf5 not found: ID does not exist" containerID="b66cc3f1cfa154e91f53a673aaac48f9fcd49cde853157e43d7fc96cae4a8cf5" Oct 30 21:14:01.473863 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:14:01.473829 5378 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b66cc3f1cfa154e91f53a673aaac48f9fcd49cde853157e43d7fc96cae4a8cf5"} err="failed to get container status \"b66cc3f1cfa154e91f53a673aaac48f9fcd49cde853157e43d7fc96cae4a8cf5\": rpc error: code = NotFound desc = could not find container \"b66cc3f1cfa154e91f53a673aaac48f9fcd49cde853157e43d7fc96cae4a8cf5\": container with ID starting with b66cc3f1cfa154e91f53a673aaac48f9fcd49cde853157e43d7fc96cae4a8cf5 not found: ID does not exist" Oct 30 21:14:01.538973 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:14:01.538944 5378 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d3278471-08a8-4d1e-8030-c6256d75265d-workload-certs\") on node \"ip-10-0-138-149.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:14:01.538973 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:14:01.538972 5378 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d3278471-08a8-4d1e-8030-c6256d75265d-istiod-ca-cert\") on node \"ip-10-0-138-149.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:14:01.539165 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:14:01.538988 5378 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d3278471-08a8-4d1e-8030-c6256d75265d-istio-token\") on node \"ip-10-0-138-149.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:14:01.539165 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:14:01.539003 5378 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nm92g\" (UniqueName: \"kubernetes.io/projected/d3278471-08a8-4d1e-8030-c6256d75265d-kube-api-access-nm92g\") on node \"ip-10-0-138-149.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:14:01.539165 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:14:01.539016 5378 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d3278471-08a8-4d1e-8030-c6256d75265d-istio-envoy\") on node \"ip-10-0-138-149.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:14:01.539165 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:14:01.539031 5378 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d3278471-08a8-4d1e-8030-c6256d75265d-workload-socket\") on node \"ip-10-0-138-149.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:14:01.539165 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:14:01.539045 5378 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d3278471-08a8-4d1e-8030-c6256d75265d-credential-socket\") on node \"ip-10-0-138-149.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:14:01.539165 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:14:01.539058 5378 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d3278471-08a8-4d1e-8030-c6256d75265d-istio-podinfo\") on node \"ip-10-0-138-149.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:14:01.539165 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:14:01.539073 5378 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d3278471-08a8-4d1e-8030-c6256d75265d-istio-data\") on node \"ip-10-0-138-149.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:14:01.855317 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:14:01.855283 5378 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w6-dc6125-openshift-default-659bcc49bf-ms7zg"] Oct 30 21:14:01.905171 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:14:01.905138 5378 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w6-dc6125-openshift-default-659bcc49bf-ms7zg"] Oct 30 21:14:02.552236 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:14:02.552183 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:14:02.703530 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:14:02.703497 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:14:02.912161 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:14:02.912123 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:14:02.922460 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:14:02.922428 5378 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-ef7e2d-openshift-default-6b6dbc765f-679n5"] Oct 30 21:14:02.922929 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:14:02.922898 5378 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-a-ef7e2d-openshift-default-6b6dbc765f-679n5" podUID="a5f9447f-85f1-40c0-b794-3a39315da3cc" containerName="istio-proxy" containerID="cri-o://0df859899b8cb42a90723ced69e4a6a0adbc2676b84d2ab26af9ceb4b80b81a6" gracePeriod=30 Oct 30 21:14:03.409500 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:14:03.409469 5378 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-390b71-64988c8666-xfnmq" Oct 30 21:14:03.743907 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:14:03.743835 5378 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d3278471-08a8-4d1e-8030-c6256d75265d" path="/var/lib/kubelet/pods/d3278471-08a8-4d1e-8030-c6256d75265d/volumes" Oct 30 21:14:05.095545 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:14:05.095069 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:14:05.989762 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:14:05.989725 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:14:06.284553 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:14:06.284491 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:14:06.294783 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:14:06.294762 5378 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-de53c2-openshift-default-58f78dcbb7-lxxs8" Oct 30 21:14:06.471007 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:14:06.470721 5378 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a5fd3bfb-32af-4284-9bed-e0e55a9dacd3-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a5fd3bfb-32af-4284-9bed-e0e55a9dacd3-workload-certs\") pod \"a5fd3bfb-32af-4284-9bed-e0e55a9dacd3\" (UID: \"a5fd3bfb-32af-4284-9bed-e0e55a9dacd3\") " Oct 30 21:14:06.471007 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:14:06.470834 5378 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a5fd3bfb-32af-4284-9bed-e0e55a9dacd3-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a5fd3bfb-32af-4284-9bed-e0e55a9dacd3-istio-data\") pod \"a5fd3bfb-32af-4284-9bed-e0e55a9dacd3\" (UID: \"a5fd3bfb-32af-4284-9bed-e0e55a9dacd3\") " Oct 30 21:14:06.471007 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:14:06.470881 5378 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a5fd3bfb-32af-4284-9bed-e0e55a9dacd3-istio-token\" (UniqueName: \"kubernetes.io/projected/a5fd3bfb-32af-4284-9bed-e0e55a9dacd3-istio-token\") pod \"a5fd3bfb-32af-4284-9bed-e0e55a9dacd3\" (UID: \"a5fd3bfb-32af-4284-9bed-e0e55a9dacd3\") " Oct 30 21:14:06.471007 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:14:06.470962 5378 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a5fd3bfb-32af-4284-9bed-e0e55a9dacd3-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a5fd3bfb-32af-4284-9bed-e0e55a9dacd3-workload-socket\") pod \"a5fd3bfb-32af-4284-9bed-e0e55a9dacd3\" (UID: \"a5fd3bfb-32af-4284-9bed-e0e55a9dacd3\") " Oct 30 21:14:06.471007 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:14:06.470985 5378 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a5fd3bfb-32af-4284-9bed-e0e55a9dacd3-workload-certs" pod "a5fd3bfb-32af-4284-9bed-e0e55a9dacd3" (UID: "a5fd3bfb-32af-4284-9bed-e0e55a9dacd3"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:14:06.471605 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:14:06.471096 5378 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a5fd3bfb-32af-4284-9bed-e0e55a9dacd3-istio-data" pod "a5fd3bfb-32af-4284-9bed-e0e55a9dacd3" (UID: "a5fd3bfb-32af-4284-9bed-e0e55a9dacd3"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:14:06.471605 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:14:06.471132 5378 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a5fd3bfb-32af-4284-9bed-e0e55a9dacd3-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a5fd3bfb-32af-4284-9bed-e0e55a9dacd3-istiod-ca-cert\") pod \"a5fd3bfb-32af-4284-9bed-e0e55a9dacd3\" (UID: \"a5fd3bfb-32af-4284-9bed-e0e55a9dacd3\") " Oct 30 21:14:06.471605 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:14:06.471222 5378 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/a5fd3bfb-32af-4284-9bed-e0e55a9dacd3-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a5fd3bfb-32af-4284-9bed-e0e55a9dacd3-istio-podinfo\") pod \"a5fd3bfb-32af-4284-9bed-e0e55a9dacd3\" (UID: \"a5fd3bfb-32af-4284-9bed-e0e55a9dacd3\") " Oct 30 21:14:06.471605 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:14:06.471227 5378 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a5fd3bfb-32af-4284-9bed-e0e55a9dacd3-workload-socket" pod "a5fd3bfb-32af-4284-9bed-e0e55a9dacd3" (UID: "a5fd3bfb-32af-4284-9bed-e0e55a9dacd3"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:14:06.471605 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:14:06.471287 5378 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a5fd3bfb-32af-4284-9bed-e0e55a9dacd3-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a5fd3bfb-32af-4284-9bed-e0e55a9dacd3-istio-envoy\") pod \"a5fd3bfb-32af-4284-9bed-e0e55a9dacd3\" (UID: \"a5fd3bfb-32af-4284-9bed-e0e55a9dacd3\") " Oct 30 21:14:06.471605 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:14:06.471331 5378 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a5fd3bfb-32af-4284-9bed-e0e55a9dacd3-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a5fd3bfb-32af-4284-9bed-e0e55a9dacd3-credential-socket\") pod \"a5fd3bfb-32af-4284-9bed-e0e55a9dacd3\" (UID: \"a5fd3bfb-32af-4284-9bed-e0e55a9dacd3\") " Oct 30 21:14:06.471605 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:14:06.471366 5378 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a5fd3bfb-32af-4284-9bed-e0e55a9dacd3-kube-api-access-r5mfq\" (UniqueName: \"kubernetes.io/projected/a5fd3bfb-32af-4284-9bed-e0e55a9dacd3-kube-api-access-r5mfq\") pod \"a5fd3bfb-32af-4284-9bed-e0e55a9dacd3\" (UID: \"a5fd3bfb-32af-4284-9bed-e0e55a9dacd3\") " Oct 30 21:14:06.471605 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:14:06.471506 5378 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a5fd3bfb-32af-4284-9bed-e0e55a9dacd3-istiod-ca-cert" pod "a5fd3bfb-32af-4284-9bed-e0e55a9dacd3" (UID: "a5fd3bfb-32af-4284-9bed-e0e55a9dacd3"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:14:06.472046 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:14:06.472020 5378 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a5fd3bfb-32af-4284-9bed-e0e55a9dacd3-workload-certs\") on node \"ip-10-0-138-149.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:14:06.472046 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:14:06.472042 5378 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a5fd3bfb-32af-4284-9bed-e0e55a9dacd3-istio-data\") on node \"ip-10-0-138-149.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:14:06.472162 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:14:06.472056 5378 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a5fd3bfb-32af-4284-9bed-e0e55a9dacd3-workload-socket\") on node \"ip-10-0-138-149.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:14:06.472162 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:14:06.472068 5378 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a5fd3bfb-32af-4284-9bed-e0e55a9dacd3-istiod-ca-cert\") on node \"ip-10-0-138-149.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:14:06.472643 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:14:06.472613 5378 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a5fd3bfb-32af-4284-9bed-e0e55a9dacd3-credential-socket" pod "a5fd3bfb-32af-4284-9bed-e0e55a9dacd3" (UID: "a5fd3bfb-32af-4284-9bed-e0e55a9dacd3"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:14:06.473281 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:14:06.473229 5378 generic.go:356] "Generic (PLEG): container finished" podID="a5fd3bfb-32af-4284-9bed-e0e55a9dacd3" containerID="8bb6aa694c2f2f98db299ea198a6a5ac67e48d239e865c892c29e989901a4070" exitCode=0 Oct 30 21:14:06.473397 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:14:06.473297 5378 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-de53c2-openshift-default-58f78dcbb7-lxxs8" event={"ID":"a5fd3bfb-32af-4284-9bed-e0e55a9dacd3","Type":"ContainerDied","Data":"8bb6aa694c2f2f98db299ea198a6a5ac67e48d239e865c892c29e989901a4070"} Oct 30 21:14:06.473397 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:14:06.473349 5378 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-de53c2-openshift-default-58f78dcbb7-lxxs8" event={"ID":"a5fd3bfb-32af-4284-9bed-e0e55a9dacd3","Type":"ContainerDied","Data":"edc1a3f617cdfdf7d015bbf35f946a161f2e29a7ba9a97f9b321a0fee14d142b"} Oct 30 21:14:06.473397 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:14:06.473382 5378 scope.go:122] "RemoveContainer" containerID="8bb6aa694c2f2f98db299ea198a6a5ac67e48d239e865c892c29e989901a4070" Oct 30 21:14:06.473854 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:14:06.473821 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:14:06.474558 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:14:06.474354 5378 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-de53c2-openshift-default-58f78dcbb7-lxxs8" Oct 30 21:14:06.475745 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:14:06.475720 5378 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a5fd3bfb-32af-4284-9bed-e0e55a9dacd3-istio-token" pod "a5fd3bfb-32af-4284-9bed-e0e55a9dacd3" (UID: "a5fd3bfb-32af-4284-9bed-e0e55a9dacd3"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:14:06.475745 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:14:06.475738 5378 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a5fd3bfb-32af-4284-9bed-e0e55a9dacd3-istio-envoy" pod "a5fd3bfb-32af-4284-9bed-e0e55a9dacd3" (UID: "a5fd3bfb-32af-4284-9bed-e0e55a9dacd3"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:14:06.476081 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:14:06.476049 5378 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a5fd3bfb-32af-4284-9bed-e0e55a9dacd3-istio-podinfo" pod "a5fd3bfb-32af-4284-9bed-e0e55a9dacd3" (UID: "a5fd3bfb-32af-4284-9bed-e0e55a9dacd3"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 21:14:06.476984 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:14:06.476949 5378 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a5fd3bfb-32af-4284-9bed-e0e55a9dacd3-kube-api-access-r5mfq" pod "a5fd3bfb-32af-4284-9bed-e0e55a9dacd3" (UID: "a5fd3bfb-32af-4284-9bed-e0e55a9dacd3"). InnerVolumeSpecName "kube-api-access-r5mfq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:14:06.508020 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:14:06.507999 5378 scope.go:122] "RemoveContainer" containerID="36d9a82525d731c9a76b2b3ef95df6120d9c572ec2ea1468eb7a27de25648b71" Oct 30 21:14:06.518389 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:14:06.518365 5378 scope.go:122] "RemoveContainer" containerID="8bb6aa694c2f2f98db299ea198a6a5ac67e48d239e865c892c29e989901a4070" Oct 30 21:14:06.518678 ip-10-0-138-149 kubenswrapper[5378]: E1030 21:14:06.518657 5378 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8bb6aa694c2f2f98db299ea198a6a5ac67e48d239e865c892c29e989901a4070\": container with ID starting with 8bb6aa694c2f2f98db299ea198a6a5ac67e48d239e865c892c29e989901a4070 not found: ID does not exist" containerID="8bb6aa694c2f2f98db299ea198a6a5ac67e48d239e865c892c29e989901a4070" Oct 30 21:14:06.518832 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:14:06.518727 5378 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8bb6aa694c2f2f98db299ea198a6a5ac67e48d239e865c892c29e989901a4070"} err="failed to get container status \"8bb6aa694c2f2f98db299ea198a6a5ac67e48d239e865c892c29e989901a4070\": rpc error: code = NotFound desc = could not find container \"8bb6aa694c2f2f98db299ea198a6a5ac67e48d239e865c892c29e989901a4070\": container with ID starting with 8bb6aa694c2f2f98db299ea198a6a5ac67e48d239e865c892c29e989901a4070 not found: ID does not exist" Oct 30 21:14:06.518832 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:14:06.518756 5378 scope.go:122] "RemoveContainer" containerID="36d9a82525d731c9a76b2b3ef95df6120d9c572ec2ea1468eb7a27de25648b71" Oct 30 21:14:06.519099 ip-10-0-138-149 kubenswrapper[5378]: E1030 21:14:06.519079 5378 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"36d9a82525d731c9a76b2b3ef95df6120d9c572ec2ea1468eb7a27de25648b71\": container with ID starting with 36d9a82525d731c9a76b2b3ef95df6120d9c572ec2ea1468eb7a27de25648b71 not found: ID does not exist" containerID="36d9a82525d731c9a76b2b3ef95df6120d9c572ec2ea1468eb7a27de25648b71" Oct 30 21:14:06.519161 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:14:06.519108 5378 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"36d9a82525d731c9a76b2b3ef95df6120d9c572ec2ea1468eb7a27de25648b71"} err="failed to get container status \"36d9a82525d731c9a76b2b3ef95df6120d9c572ec2ea1468eb7a27de25648b71\": rpc error: code = NotFound desc = could not find container \"36d9a82525d731c9a76b2b3ef95df6120d9c572ec2ea1468eb7a27de25648b71\": container with ID starting with 36d9a82525d731c9a76b2b3ef95df6120d9c572ec2ea1468eb7a27de25648b71 not found: ID does not exist" Oct 30 21:14:06.573220 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:14:06.573191 5378 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a5fd3bfb-32af-4284-9bed-e0e55a9dacd3-istio-token\") on node \"ip-10-0-138-149.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:14:06.573220 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:14:06.573215 5378 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a5fd3bfb-32af-4284-9bed-e0e55a9dacd3-istio-podinfo\") on node \"ip-10-0-138-149.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:14:06.573220 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:14:06.573226 5378 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a5fd3bfb-32af-4284-9bed-e0e55a9dacd3-istio-envoy\") on node \"ip-10-0-138-149.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:14:06.573400 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:14:06.573237 5378 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a5fd3bfb-32af-4284-9bed-e0e55a9dacd3-credential-socket\") on node \"ip-10-0-138-149.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:14:06.573400 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:14:06.573251 5378 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r5mfq\" (UniqueName: \"kubernetes.io/projected/a5fd3bfb-32af-4284-9bed-e0e55a9dacd3-kube-api-access-r5mfq\") on node \"ip-10-0-138-149.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:14:06.617404 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:14:06.617375 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:14:06.900263 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:14:06.900234 5378 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-de53c2-openshift-default-58f78dcbb7-lxxs8"] Oct 30 21:14:06.950550 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:14:06.950519 5378 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-de53c2-openshift-default-58f78dcbb7-lxxs8"] Oct 30 21:14:07.743934 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:14:07.743901 5378 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a5fd3bfb-32af-4284-9bed-e0e55a9dacd3" path="/var/lib/kubelet/pods/a5fd3bfb-32af-4284-9bed-e0e55a9dacd3/volumes" Oct 30 21:14:08.190976 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:14:08.190950 5378 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-ef7e2d-openshift-default-6b6dbc765f-679n5" Oct 30 21:14:08.231662 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:14:08.231636 5378 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a5f9447f-85f1-40c0-b794-3a39315da3cc-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a5f9447f-85f1-40c0-b794-3a39315da3cc-workload-socket\") pod \"a5f9447f-85f1-40c0-b794-3a39315da3cc\" (UID: \"a5f9447f-85f1-40c0-b794-3a39315da3cc\") " Oct 30 21:14:08.231775 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:14:08.231691 5378 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a5f9447f-85f1-40c0-b794-3a39315da3cc-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a5f9447f-85f1-40c0-b794-3a39315da3cc-istio-data\") pod \"a5f9447f-85f1-40c0-b794-3a39315da3cc\" (UID: \"a5f9447f-85f1-40c0-b794-3a39315da3cc\") " Oct 30 21:14:08.231775 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:14:08.231718 5378 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a5f9447f-85f1-40c0-b794-3a39315da3cc-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a5f9447f-85f1-40c0-b794-3a39315da3cc-credential-socket\") pod \"a5f9447f-85f1-40c0-b794-3a39315da3cc\" (UID: \"a5f9447f-85f1-40c0-b794-3a39315da3cc\") " Oct 30 21:14:08.231848 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:14:08.231806 5378 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/a5f9447f-85f1-40c0-b794-3a39315da3cc-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a5f9447f-85f1-40c0-b794-3a39315da3cc-istio-podinfo\") pod \"a5f9447f-85f1-40c0-b794-3a39315da3cc\" (UID: \"a5f9447f-85f1-40c0-b794-3a39315da3cc\") " Oct 30 21:14:08.231895 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:14:08.231845 5378 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a5f9447f-85f1-40c0-b794-3a39315da3cc-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a5f9447f-85f1-40c0-b794-3a39315da3cc-istiod-ca-cert\") pod \"a5f9447f-85f1-40c0-b794-3a39315da3cc\" (UID: \"a5f9447f-85f1-40c0-b794-3a39315da3cc\") " Oct 30 21:14:08.231945 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:14:08.231882 5378 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a5f9447f-85f1-40c0-b794-3a39315da3cc-workload-socket" pod "a5f9447f-85f1-40c0-b794-3a39315da3cc" (UID: "a5f9447f-85f1-40c0-b794-3a39315da3cc"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:14:08.232000 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:14:08.231972 5378 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a5f9447f-85f1-40c0-b794-3a39315da3cc-istio-token\" (UniqueName: \"kubernetes.io/projected/a5f9447f-85f1-40c0-b794-3a39315da3cc-istio-token\") pod \"a5f9447f-85f1-40c0-b794-3a39315da3cc\" (UID: \"a5f9447f-85f1-40c0-b794-3a39315da3cc\") " Oct 30 21:14:08.232058 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:14:08.231992 5378 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a5f9447f-85f1-40c0-b794-3a39315da3cc-istio-data" pod "a5f9447f-85f1-40c0-b794-3a39315da3cc" (UID: "a5f9447f-85f1-40c0-b794-3a39315da3cc"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:14:08.232058 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:14:08.232021 5378 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a5f9447f-85f1-40c0-b794-3a39315da3cc-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a5f9447f-85f1-40c0-b794-3a39315da3cc-workload-certs\") pod \"a5f9447f-85f1-40c0-b794-3a39315da3cc\" (UID: \"a5f9447f-85f1-40c0-b794-3a39315da3cc\") " Oct 30 21:14:08.232058 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:14:08.232027 5378 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a5f9447f-85f1-40c0-b794-3a39315da3cc-credential-socket" pod "a5f9447f-85f1-40c0-b794-3a39315da3cc" (UID: "a5f9447f-85f1-40c0-b794-3a39315da3cc"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:14:08.232058 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:14:08.232053 5378 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a5f9447f-85f1-40c0-b794-3a39315da3cc-kube-api-access-jh4g2\" (UniqueName: \"kubernetes.io/projected/a5f9447f-85f1-40c0-b794-3a39315da3cc-kube-api-access-jh4g2\") pod \"a5f9447f-85f1-40c0-b794-3a39315da3cc\" (UID: \"a5f9447f-85f1-40c0-b794-3a39315da3cc\") " Oct 30 21:14:08.232282 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:14:08.232088 5378 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a5f9447f-85f1-40c0-b794-3a39315da3cc-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a5f9447f-85f1-40c0-b794-3a39315da3cc-istio-envoy\") pod \"a5f9447f-85f1-40c0-b794-3a39315da3cc\" (UID: \"a5f9447f-85f1-40c0-b794-3a39315da3cc\") " Oct 30 21:14:08.232282 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:14:08.232264 5378 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a5f9447f-85f1-40c0-b794-3a39315da3cc-workload-certs" pod "a5f9447f-85f1-40c0-b794-3a39315da3cc" (UID: "a5f9447f-85f1-40c0-b794-3a39315da3cc"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:14:08.232378 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:14:08.232286 5378 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a5f9447f-85f1-40c0-b794-3a39315da3cc-istiod-ca-cert" pod "a5f9447f-85f1-40c0-b794-3a39315da3cc" (UID: "a5f9447f-85f1-40c0-b794-3a39315da3cc"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:14:08.232750 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:14:08.232716 5378 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a5f9447f-85f1-40c0-b794-3a39315da3cc-workload-certs\") on node \"ip-10-0-138-149.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:14:08.232750 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:14:08.232745 5378 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a5f9447f-85f1-40c0-b794-3a39315da3cc-workload-socket\") on node \"ip-10-0-138-149.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:14:08.232905 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:14:08.232760 5378 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a5f9447f-85f1-40c0-b794-3a39315da3cc-istio-data\") on node \"ip-10-0-138-149.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:14:08.232905 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:14:08.232774 5378 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a5f9447f-85f1-40c0-b794-3a39315da3cc-credential-socket\") on node \"ip-10-0-138-149.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:14:08.232905 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:14:08.232790 5378 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a5f9447f-85f1-40c0-b794-3a39315da3cc-istiod-ca-cert\") on node \"ip-10-0-138-149.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:14:08.234103 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:14:08.234081 5378 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a5f9447f-85f1-40c0-b794-3a39315da3cc-istio-envoy" pod "a5f9447f-85f1-40c0-b794-3a39315da3cc" (UID: "a5f9447f-85f1-40c0-b794-3a39315da3cc"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:14:08.234201 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:14:08.234122 5378 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a5f9447f-85f1-40c0-b794-3a39315da3cc-istio-token" pod "a5f9447f-85f1-40c0-b794-3a39315da3cc" (UID: "a5f9447f-85f1-40c0-b794-3a39315da3cc"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:14:08.234201 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:14:08.234163 5378 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a5f9447f-85f1-40c0-b794-3a39315da3cc-istio-podinfo" pod "a5f9447f-85f1-40c0-b794-3a39315da3cc" (UID: "a5f9447f-85f1-40c0-b794-3a39315da3cc"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 21:14:08.234274 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:14:08.234215 5378 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a5f9447f-85f1-40c0-b794-3a39315da3cc-kube-api-access-jh4g2" pod "a5f9447f-85f1-40c0-b794-3a39315da3cc" (UID: "a5f9447f-85f1-40c0-b794-3a39315da3cc"). InnerVolumeSpecName "kube-api-access-jh4g2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:14:08.333849 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:14:08.333783 5378 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a5f9447f-85f1-40c0-b794-3a39315da3cc-istio-podinfo\") on node \"ip-10-0-138-149.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:14:08.333849 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:14:08.333810 5378 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a5f9447f-85f1-40c0-b794-3a39315da3cc-istio-token\") on node \"ip-10-0-138-149.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:14:08.333849 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:14:08.333819 5378 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jh4g2\" (UniqueName: \"kubernetes.io/projected/a5f9447f-85f1-40c0-b794-3a39315da3cc-kube-api-access-jh4g2\") on node \"ip-10-0-138-149.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:14:08.333849 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:14:08.333829 5378 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a5f9447f-85f1-40c0-b794-3a39315da3cc-istio-envoy\") on node \"ip-10-0-138-149.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:14:08.490008 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:14:08.489974 5378 generic.go:356] "Generic (PLEG): container finished" podID="a5f9447f-85f1-40c0-b794-3a39315da3cc" containerID="0df859899b8cb42a90723ced69e4a6a0adbc2676b84d2ab26af9ceb4b80b81a6" exitCode=0 Oct 30 21:14:08.490162 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:14:08.490024 5378 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-ef7e2d-openshift-default-6b6dbc765f-679n5" Oct 30 21:14:08.490162 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:14:08.490042 5378 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-ef7e2d-openshift-default-6b6dbc765f-679n5" event={"ID":"a5f9447f-85f1-40c0-b794-3a39315da3cc","Type":"ContainerDied","Data":"0df859899b8cb42a90723ced69e4a6a0adbc2676b84d2ab26af9ceb4b80b81a6"} Oct 30 21:14:08.490162 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:14:08.490099 5378 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-ef7e2d-openshift-default-6b6dbc765f-679n5" event={"ID":"a5f9447f-85f1-40c0-b794-3a39315da3cc","Type":"ContainerDied","Data":"2702d21e7604e7d1095494b46ab9c4e9bd63a1f6957d6a4c279dcebcebbc1237"} Oct 30 21:14:08.490162 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:14:08.490130 5378 scope.go:122] "RemoveContainer" containerID="0df859899b8cb42a90723ced69e4a6a0adbc2676b84d2ab26af9ceb4b80b81a6" Oct 30 21:14:08.501391 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:14:08.501369 5378 scope.go:122] "RemoveContainer" containerID="e6437a86d114df1af144051ae1f9213ff0b6b1a6c558b343f17d3a8cecda7675" Oct 30 21:14:08.510487 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:14:08.510470 5378 scope.go:122] "RemoveContainer" containerID="0df859899b8cb42a90723ced69e4a6a0adbc2676b84d2ab26af9ceb4b80b81a6" Oct 30 21:14:08.510772 ip-10-0-138-149 kubenswrapper[5378]: E1030 21:14:08.510754 5378 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0df859899b8cb42a90723ced69e4a6a0adbc2676b84d2ab26af9ceb4b80b81a6\": container with ID starting with 0df859899b8cb42a90723ced69e4a6a0adbc2676b84d2ab26af9ceb4b80b81a6 not found: ID does not exist" containerID="0df859899b8cb42a90723ced69e4a6a0adbc2676b84d2ab26af9ceb4b80b81a6" Oct 30 21:14:08.510828 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:14:08.510782 5378 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0df859899b8cb42a90723ced69e4a6a0adbc2676b84d2ab26af9ceb4b80b81a6"} err="failed to get container status \"0df859899b8cb42a90723ced69e4a6a0adbc2676b84d2ab26af9ceb4b80b81a6\": rpc error: code = NotFound desc = could not find container \"0df859899b8cb42a90723ced69e4a6a0adbc2676b84d2ab26af9ceb4b80b81a6\": container with ID starting with 0df859899b8cb42a90723ced69e4a6a0adbc2676b84d2ab26af9ceb4b80b81a6 not found: ID does not exist" Oct 30 21:14:08.510828 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:14:08.510799 5378 scope.go:122] "RemoveContainer" containerID="e6437a86d114df1af144051ae1f9213ff0b6b1a6c558b343f17d3a8cecda7675" Oct 30 21:14:08.511003 ip-10-0-138-149 kubenswrapper[5378]: E1030 21:14:08.510983 5378 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e6437a86d114df1af144051ae1f9213ff0b6b1a6c558b343f17d3a8cecda7675\": container with ID starting with e6437a86d114df1af144051ae1f9213ff0b6b1a6c558b343f17d3a8cecda7675 not found: ID does not exist" containerID="e6437a86d114df1af144051ae1f9213ff0b6b1a6c558b343f17d3a8cecda7675" Oct 30 21:14:08.511062 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:14:08.511012 5378 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e6437a86d114df1af144051ae1f9213ff0b6b1a6c558b343f17d3a8cecda7675"} err="failed to get container status \"e6437a86d114df1af144051ae1f9213ff0b6b1a6c558b343f17d3a8cecda7675\": rpc error: code = NotFound desc = could not find container \"e6437a86d114df1af144051ae1f9213ff0b6b1a6c558b343f17d3a8cecda7675\": container with ID starting with e6437a86d114df1af144051ae1f9213ff0b6b1a6c558b343f17d3a8cecda7675 not found: ID does not exist" Oct 30 21:14:08.549385 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:14:08.549354 5378 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w6-7eaf74-d47bd994b-tww2z"] Oct 30 21:14:08.671794 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:14:08.671759 5378 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w6-7eaf74-6b776cb9f-rqvlh"] Oct 30 21:14:08.675223 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:14:08.675198 5378 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-ef7e2d-openshift-default-6b6dbc765f-679n5"] Oct 30 21:14:08.678137 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:14:08.678115 5378 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-a-ef7e2d-openshift-default-6b6dbc765f-679n5"] Oct 30 21:14:08.949542 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:14:08.949461 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:14:09.120589 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:14:09.113832 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:14:09.746149 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:14:09.746118 5378 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a5f9447f-85f1-40c0-b794-3a39315da3cc" path="/var/lib/kubelet/pods/a5f9447f-85f1-40c0-b794-3a39315da3cc/volumes" Oct 30 21:14:10.593956 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:14:10.593926 5378 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-r9g59"] Oct 30 21:14:10.645752 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:14:10.645717 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:14:10.832363 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:14:10.832332 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:14:11.024098 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:14:11.024067 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:14:12.972680 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:14:12.972643 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:14:13.550137 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:14:13.550073 5378 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-b-w6-7eaf74-d47bd994b-tww2z" podUID="72d82ec7-ffcb-4267-9639-81357c29430d" containerName="payload-processing" containerID="cri-o://4c3b1462b6a5d0d56598d8e9368d10be76bd32e259eaaa402b5adefe2af05e56" gracePeriod=25 Oct 30 21:14:13.672812 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:14:13.672779 5378 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-7eaf74-6b776cb9f-rqvlh" podUID="f8a30c85-5cd7-4961-ae05-e7c587f48573" containerName="payload-pre-processing" containerID="cri-o://442677e71a348c1221d49dedce080d924ce1b3de0ea5408090a1f681d7428439" gracePeriod=25 Oct 30 21:14:13.846829 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:14:13.846803 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:14:13.865515 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:14:13.865487 5378 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w6-7eaf74-d47bd994b-tww2z" Oct 30 21:14:13.997648 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:14:13.997607 5378 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/72d82ec7-ffcb-4267-9639-81357c29430d-kube-api-access-svzl2\" (UniqueName: \"kubernetes.io/projected/72d82ec7-ffcb-4267-9639-81357c29430d-kube-api-access-svzl2\") pod \"72d82ec7-ffcb-4267-9639-81357c29430d\" (UID: \"72d82ec7-ffcb-4267-9639-81357c29430d\") " Oct 30 21:14:13.998057 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:14:13.997911 5378 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/72d82ec7-ffcb-4267-9639-81357c29430d-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/72d82ec7-ffcb-4267-9639-81357c29430d-plugins-config-volume\") pod \"72d82ec7-ffcb-4267-9639-81357c29430d\" (UID: \"72d82ec7-ffcb-4267-9639-81357c29430d\") " Oct 30 21:14:13.998876 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:14:13.998852 5378 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/72d82ec7-ffcb-4267-9639-81357c29430d-plugins-config-volume" pod "72d82ec7-ffcb-4267-9639-81357c29430d" (UID: "72d82ec7-ffcb-4267-9639-81357c29430d"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:14:14.009506 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:14:14.009471 5378 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/72d82ec7-ffcb-4267-9639-81357c29430d-kube-api-access-svzl2" pod "72d82ec7-ffcb-4267-9639-81357c29430d" (UID: "72d82ec7-ffcb-4267-9639-81357c29430d"). InnerVolumeSpecName "kube-api-access-svzl2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:14:14.012185 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:14:14.012165 5378 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-7eaf74-6b776cb9f-rqvlh" Oct 30 21:14:14.099314 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:14:14.099288 5378 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f8a30c85-5cd7-4961-ae05-e7c587f48573-kube-api-access-8w6p7\" (UniqueName: \"kubernetes.io/projected/f8a30c85-5cd7-4961-ae05-e7c587f48573-kube-api-access-8w6p7\") pod \"f8a30c85-5cd7-4961-ae05-e7c587f48573\" (UID: \"f8a30c85-5cd7-4961-ae05-e7c587f48573\") " Oct 30 21:14:14.099512 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:14:14.099449 5378 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f8a30c85-5cd7-4961-ae05-e7c587f48573-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f8a30c85-5cd7-4961-ae05-e7c587f48573-plugins-config-volume\") pod \"f8a30c85-5cd7-4961-ae05-e7c587f48573\" (UID: \"f8a30c85-5cd7-4961-ae05-e7c587f48573\") " Oct 30 21:14:14.100181 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:14:14.100148 5378 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f8a30c85-5cd7-4961-ae05-e7c587f48573-plugins-config-volume" pod "f8a30c85-5cd7-4961-ae05-e7c587f48573" (UID: "f8a30c85-5cd7-4961-ae05-e7c587f48573"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:14:14.100380 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:14:14.100359 5378 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/72d82ec7-ffcb-4267-9639-81357c29430d-plugins-config-volume\") on node \"ip-10-0-138-149.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:14:14.100459 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:14:14.100382 5378 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f8a30c85-5cd7-4961-ae05-e7c587f48573-plugins-config-volume\") on node \"ip-10-0-138-149.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:14:14.100459 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:14:14.100397 5378 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-svzl2\" (UniqueName: \"kubernetes.io/projected/72d82ec7-ffcb-4267-9639-81357c29430d-kube-api-access-svzl2\") on node \"ip-10-0-138-149.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:14:14.103311 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:14:14.103280 5378 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f8a30c85-5cd7-4961-ae05-e7c587f48573-kube-api-access-8w6p7" pod "f8a30c85-5cd7-4961-ae05-e7c587f48573" (UID: "f8a30c85-5cd7-4961-ae05-e7c587f48573"). InnerVolumeSpecName "kube-api-access-8w6p7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:14:14.135440 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:14:14.135410 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:14:14.202519 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:14:14.202483 5378 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8w6p7\" (UniqueName: \"kubernetes.io/projected/f8a30c85-5cd7-4961-ae05-e7c587f48573-kube-api-access-8w6p7\") on node \"ip-10-0-138-149.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:14:14.323591 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:14:14.323094 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:14:14.563738 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:14:14.563657 5378 generic.go:356] "Generic (PLEG): container finished" podID="f8a30c85-5cd7-4961-ae05-e7c587f48573" containerID="442677e71a348c1221d49dedce080d924ce1b3de0ea5408090a1f681d7428439" exitCode=0 Oct 30 21:14:14.563738 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:14:14.563723 5378 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-7eaf74-6b776cb9f-rqvlh" Oct 30 21:14:14.563966 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:14:14.563745 5378 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-7eaf74-6b776cb9f-rqvlh" event={"ID":"f8a30c85-5cd7-4961-ae05-e7c587f48573","Type":"ContainerDied","Data":"442677e71a348c1221d49dedce080d924ce1b3de0ea5408090a1f681d7428439"} Oct 30 21:14:14.563966 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:14:14.563799 5378 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-7eaf74-6b776cb9f-rqvlh" event={"ID":"f8a30c85-5cd7-4961-ae05-e7c587f48573","Type":"ContainerDied","Data":"27ded346fb41fea2a6bdcec473c11c73429283df72e12f9978c7eb27c027be4d"} Oct 30 21:14:14.563966 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:14:14.563828 5378 scope.go:122] "RemoveContainer" containerID="442677e71a348c1221d49dedce080d924ce1b3de0ea5408090a1f681d7428439" Oct 30 21:14:14.565286 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:14:14.565261 5378 generic.go:356] "Generic (PLEG): container finished" podID="72d82ec7-ffcb-4267-9639-81357c29430d" containerID="4c3b1462b6a5d0d56598d8e9368d10be76bd32e259eaaa402b5adefe2af05e56" exitCode=0 Oct 30 21:14:14.565595 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:14:14.565299 5378 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w6-7eaf74-d47bd994b-tww2z" event={"ID":"72d82ec7-ffcb-4267-9639-81357c29430d","Type":"ContainerDied","Data":"4c3b1462b6a5d0d56598d8e9368d10be76bd32e259eaaa402b5adefe2af05e56"} Oct 30 21:14:14.565595 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:14:14.565321 5378 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w6-7eaf74-d47bd994b-tww2z" Oct 30 21:14:14.565595 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:14:14.565359 5378 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w6-7eaf74-d47bd994b-tww2z" event={"ID":"72d82ec7-ffcb-4267-9639-81357c29430d","Type":"ContainerDied","Data":"b1bb32cc48b8e972032ef99d8cb5a22de4160084e3bc1730f3aae7cdabea776e"} Oct 30 21:14:14.581769 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:14:14.581752 5378 scope.go:122] "RemoveContainer" containerID="442677e71a348c1221d49dedce080d924ce1b3de0ea5408090a1f681d7428439" Oct 30 21:14:14.582074 ip-10-0-138-149 kubenswrapper[5378]: E1030 21:14:14.582052 5378 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"442677e71a348c1221d49dedce080d924ce1b3de0ea5408090a1f681d7428439\": container with ID starting with 442677e71a348c1221d49dedce080d924ce1b3de0ea5408090a1f681d7428439 not found: ID does not exist" containerID="442677e71a348c1221d49dedce080d924ce1b3de0ea5408090a1f681d7428439" Oct 30 21:14:14.582201 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:14:14.582079 5378 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"442677e71a348c1221d49dedce080d924ce1b3de0ea5408090a1f681d7428439"} err="failed to get container status \"442677e71a348c1221d49dedce080d924ce1b3de0ea5408090a1f681d7428439\": rpc error: code = NotFound desc = could not find container \"442677e71a348c1221d49dedce080d924ce1b3de0ea5408090a1f681d7428439\": container with ID starting with 442677e71a348c1221d49dedce080d924ce1b3de0ea5408090a1f681d7428439 not found: ID does not exist" Oct 30 21:14:14.582201 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:14:14.582098 5378 scope.go:122] "RemoveContainer" containerID="4c3b1462b6a5d0d56598d8e9368d10be76bd32e259eaaa402b5adefe2af05e56" Oct 30 21:14:14.591404 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:14:14.591386 5378 scope.go:122] "RemoveContainer" containerID="4c3b1462b6a5d0d56598d8e9368d10be76bd32e259eaaa402b5adefe2af05e56" Oct 30 21:14:14.591757 ip-10-0-138-149 kubenswrapper[5378]: E1030 21:14:14.591740 5378 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4c3b1462b6a5d0d56598d8e9368d10be76bd32e259eaaa402b5adefe2af05e56\": container with ID starting with 4c3b1462b6a5d0d56598d8e9368d10be76bd32e259eaaa402b5adefe2af05e56 not found: ID does not exist" containerID="4c3b1462b6a5d0d56598d8e9368d10be76bd32e259eaaa402b5adefe2af05e56" Oct 30 21:14:14.591809 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:14:14.591762 5378 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4c3b1462b6a5d0d56598d8e9368d10be76bd32e259eaaa402b5adefe2af05e56"} err="failed to get container status \"4c3b1462b6a5d0d56598d8e9368d10be76bd32e259eaaa402b5adefe2af05e56\": rpc error: code = NotFound desc = could not find container \"4c3b1462b6a5d0d56598d8e9368d10be76bd32e259eaaa402b5adefe2af05e56\": container with ID starting with 4c3b1462b6a5d0d56598d8e9368d10be76bd32e259eaaa402b5adefe2af05e56 not found: ID does not exist" Oct 30 21:14:14.742523 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:14:14.742490 5378 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w6-7eaf74-6b776cb9f-rqvlh"] Oct 30 21:14:14.791865 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:14:14.791833 5378 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w6-7eaf74-6b776cb9f-rqvlh"] Oct 30 21:14:14.813887 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:14:14.813820 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:14:14.859269 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:14:14.859243 5378 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w6-7eaf74-d47bd994b-tww2z"] Oct 30 21:14:14.865115 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:14:14.865092 5378 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w6-7eaf74-d47bd994b-tww2z"] Oct 30 21:14:15.744778 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:14:15.744747 5378 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="72d82ec7-ffcb-4267-9639-81357c29430d" path="/var/lib/kubelet/pods/72d82ec7-ffcb-4267-9639-81357c29430d/volumes" Oct 30 21:14:15.745167 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:14:15.745156 5378 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f8a30c85-5cd7-4961-ae05-e7c587f48573" path="/var/lib/kubelet/pods/f8a30c85-5cd7-4961-ae05-e7c587f48573/volumes" Oct 30 21:14:17.846655 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:14:17.846595 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:14:18.099548 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:14:18.099454 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:14:18.314831 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:14:18.314782 5378 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-f0fe13-7447d4f599-smmk4"] Oct 30 21:14:18.316830 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:14:18.316123 5378 cpu_manager.go:414] "containerMap: removing container" podUID="d3278471-08a8-4d1e-8030-c6256d75265d" containerName="istio-proxy" Oct 30 21:14:18.317070 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:14:18.317049 5378 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d3278471-08a8-4d1e-8030-c6256d75265d" containerName="istio-proxy" Oct 30 21:14:18.317193 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:14:18.317181 5378 cpu_manager.go:414] "containerMap: removing container" podUID="a5f9447f-85f1-40c0-b794-3a39315da3cc" containerName="istio-proxy" Oct 30 21:14:18.317270 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:14:18.317259 5378 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a5f9447f-85f1-40c0-b794-3a39315da3cc" containerName="istio-proxy" Oct 30 21:14:18.317400 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:14:18.317385 5378 cpu_manager.go:414] "containerMap: removing container" podUID="a5f9447f-85f1-40c0-b794-3a39315da3cc" containerName="istio-proxy" Oct 30 21:14:18.317546 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:14:18.317535 5378 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a5f9447f-85f1-40c0-b794-3a39315da3cc" containerName="istio-proxy" Oct 30 21:14:18.317685 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:14:18.317673 5378 cpu_manager.go:414] "containerMap: removing container" podUID="a5fd3bfb-32af-4284-9bed-e0e55a9dacd3" containerName="istio-proxy" Oct 30 21:14:18.317772 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:14:18.317761 5378 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a5fd3bfb-32af-4284-9bed-e0e55a9dacd3" containerName="istio-proxy" Oct 30 21:14:18.317882 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:14:18.317871 5378 cpu_manager.go:414] "containerMap: removing container" podUID="72d82ec7-ffcb-4267-9639-81357c29430d" containerName="payload-processing" Oct 30 21:14:18.317956 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:14:18.317946 5378 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="72d82ec7-ffcb-4267-9639-81357c29430d" containerName="payload-processing" Oct 30 21:14:18.318043 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:14:18.318031 5378 cpu_manager.go:414] "containerMap: removing container" podUID="f8a30c85-5cd7-4961-ae05-e7c587f48573" containerName="payload-pre-processing" Oct 30 21:14:18.318134 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:14:18.318122 5378 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f8a30c85-5cd7-4961-ae05-e7c587f48573" containerName="payload-pre-processing" Oct 30 21:14:18.318402 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:14:18.318387 5378 memory_manager.go:365] "RemoveStaleState removing state" podUID="a5f9447f-85f1-40c0-b794-3a39315da3cc" containerName="istio-proxy" Oct 30 21:14:18.318507 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:14:18.318498 5378 memory_manager.go:365] "RemoveStaleState removing state" podUID="a5fd3bfb-32af-4284-9bed-e0e55a9dacd3" containerName="istio-proxy" Oct 30 21:14:18.318609 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:14:18.318598 5378 memory_manager.go:365] "RemoveStaleState removing state" podUID="72d82ec7-ffcb-4267-9639-81357c29430d" containerName="payload-processing" Oct 30 21:14:18.318688 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:14:18.318679 5378 memory_manager.go:365] "RemoveStaleState removing state" podUID="d3278471-08a8-4d1e-8030-c6256d75265d" containerName="istio-proxy" Oct 30 21:14:18.318777 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:14:18.318767 5378 memory_manager.go:365] "RemoveStaleState removing state" podUID="f8a30c85-5cd7-4961-ae05-e7c587f48573" containerName="payload-pre-processing" Oct 30 21:14:18.318867 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:14:18.318855 5378 memory_manager.go:365] "RemoveStaleState removing state" podUID="a5fd3bfb-32af-4284-9bed-e0e55a9dacd3" containerName="istio-proxy" Oct 30 21:14:18.318951 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:14:18.318940 5378 memory_manager.go:365] "RemoveStaleState removing state" podUID="d3278471-08a8-4d1e-8030-c6256d75265d" containerName="istio-proxy" Oct 30 21:14:18.319037 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:14:18.319020 5378 memory_manager.go:365] "RemoveStaleState removing state" podUID="a5f9447f-85f1-40c0-b794-3a39315da3cc" containerName="istio-proxy" Oct 30 21:14:18.326212 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:14:18.326179 5378 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-f0fe13-7447d4f599-smmk4" Oct 30 21:14:18.346142 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:14:18.346114 5378 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-f0fe13-7447d4f599-smmk4"] Oct 30 21:14:18.392655 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:14:18.390891 5378 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-a-f0fe13-dockercfg-hcgsl\"" Oct 30 21:14:18.430799 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:14:18.430769 5378 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-a-f0fe13\"" Oct 30 21:14:18.550854 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:14:18.550816 5378 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dpwbr\" (UniqueName: \"kubernetes.io/projected/ef431fc4-98f7-4a6f-85c1-5989dc6665f6-kube-api-access-dpwbr\") pod \"payload-pre-processing-e2e-inf-a-f0fe13-7447d4f599-smmk4\" (UID: \"ef431fc4-98f7-4a6f-85c1-5989dc6665f6\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-f0fe13-7447d4f599-smmk4" Oct 30 21:14:18.551169 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:14:18.551147 5378 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ef431fc4-98f7-4a6f-85c1-5989dc6665f6-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-f0fe13-7447d4f599-smmk4\" (UID: \"ef431fc4-98f7-4a6f-85c1-5989dc6665f6\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-f0fe13-7447d4f599-smmk4" Oct 30 21:14:18.567157 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:14:18.565778 5378 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-f0fe13-7cb54c466b-s9889"] Oct 30 21:14:18.570847 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:14:18.570806 5378 cpu_manager.go:414] "containerMap: removing container" podUID="d3278471-08a8-4d1e-8030-c6256d75265d" containerName="istio-proxy" Oct 30 21:14:18.570847 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:14:18.570844 5378 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d3278471-08a8-4d1e-8030-c6256d75265d" containerName="istio-proxy" Oct 30 21:14:18.571080 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:14:18.570904 5378 cpu_manager.go:414] "containerMap: removing container" podUID="a5fd3bfb-32af-4284-9bed-e0e55a9dacd3" containerName="istio-proxy" Oct 30 21:14:18.571080 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:14:18.570914 5378 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a5fd3bfb-32af-4284-9bed-e0e55a9dacd3" containerName="istio-proxy" Oct 30 21:14:18.641742 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:14:18.575729 5378 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-f0fe13-7cb54c466b-s9889" Oct 30 21:14:18.641742 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:14:18.592507 5378 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-f0fe13-7cb54c466b-s9889"] Oct 30 21:14:18.655180 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:14:18.655027 5378 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z5rgn\" (UniqueName: \"kubernetes.io/projected/97e8a2e3-2092-4e54-8621-c33b9404e163-kube-api-access-z5rgn\") pod \"payload-processing-e2e-inf-a-f0fe13-7cb54c466b-s9889\" (UID: \"97e8a2e3-2092-4e54-8621-c33b9404e163\") " pod="openshift-ingress/payload-processing-e2e-inf-a-f0fe13-7cb54c466b-s9889" Oct 30 21:14:18.655344 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:14:18.655195 5378 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/97e8a2e3-2092-4e54-8621-c33b9404e163-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-f0fe13-7cb54c466b-s9889\" (UID: \"97e8a2e3-2092-4e54-8621-c33b9404e163\") " pod="openshift-ingress/payload-processing-e2e-inf-a-f0fe13-7cb54c466b-s9889" Oct 30 21:14:18.655344 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:14:18.655274 5378 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dpwbr\" (UniqueName: \"kubernetes.io/projected/ef431fc4-98f7-4a6f-85c1-5989dc6665f6-kube-api-access-dpwbr\") pod \"payload-pre-processing-e2e-inf-a-f0fe13-7447d4f599-smmk4\" (UID: \"ef431fc4-98f7-4a6f-85c1-5989dc6665f6\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-f0fe13-7447d4f599-smmk4" Oct 30 21:14:18.655344 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:14:18.655323 5378 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ef431fc4-98f7-4a6f-85c1-5989dc6665f6-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-f0fe13-7447d4f599-smmk4\" (UID: \"ef431fc4-98f7-4a6f-85c1-5989dc6665f6\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-f0fe13-7447d4f599-smmk4" Oct 30 21:14:18.656532 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:14:18.656482 5378 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ef431fc4-98f7-4a6f-85c1-5989dc6665f6-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-f0fe13-7447d4f599-smmk4\" (UID: \"ef431fc4-98f7-4a6f-85c1-5989dc6665f6\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-f0fe13-7447d4f599-smmk4" Oct 30 21:14:18.719494 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:14:18.719466 5378 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dpwbr\" (UniqueName: \"kubernetes.io/projected/ef431fc4-98f7-4a6f-85c1-5989dc6665f6-kube-api-access-dpwbr\") pod \"payload-pre-processing-e2e-inf-a-f0fe13-7447d4f599-smmk4\" (UID: \"ef431fc4-98f7-4a6f-85c1-5989dc6665f6\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-f0fe13-7447d4f599-smmk4" Oct 30 21:14:18.756327 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:14:18.756296 5378 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/97e8a2e3-2092-4e54-8621-c33b9404e163-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-f0fe13-7cb54c466b-s9889\" (UID: \"97e8a2e3-2092-4e54-8621-c33b9404e163\") " pod="openshift-ingress/payload-processing-e2e-inf-a-f0fe13-7cb54c466b-s9889" Oct 30 21:14:18.756518 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:14:18.756500 5378 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z5rgn\" (UniqueName: \"kubernetes.io/projected/97e8a2e3-2092-4e54-8621-c33b9404e163-kube-api-access-z5rgn\") pod \"payload-processing-e2e-inf-a-f0fe13-7cb54c466b-s9889\" (UID: \"97e8a2e3-2092-4e54-8621-c33b9404e163\") " pod="openshift-ingress/payload-processing-e2e-inf-a-f0fe13-7cb54c466b-s9889" Oct 30 21:14:18.757628 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:14:18.757604 5378 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/97e8a2e3-2092-4e54-8621-c33b9404e163-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-f0fe13-7cb54c466b-s9889\" (UID: \"97e8a2e3-2092-4e54-8621-c33b9404e163\") " pod="openshift-ingress/payload-processing-e2e-inf-a-f0fe13-7cb54c466b-s9889" Oct 30 21:14:18.815327 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:14:18.815299 5378 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z5rgn\" (UniqueName: \"kubernetes.io/projected/97e8a2e3-2092-4e54-8621-c33b9404e163-kube-api-access-z5rgn\") pod \"payload-processing-e2e-inf-a-f0fe13-7cb54c466b-s9889\" (UID: \"97e8a2e3-2092-4e54-8621-c33b9404e163\") " pod="openshift-ingress/payload-processing-e2e-inf-a-f0fe13-7cb54c466b-s9889" Oct 30 21:14:18.841251 ip-10-0-138-149 kubenswrapper[5378]: E1030 21:14:18.841224 5378 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q6mx7" containerName="extractor" Oct 30 21:14:18.944417 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:14:18.944341 5378 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-f0fe13-7447d4f599-smmk4" Oct 30 21:14:18.975857 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:14:18.975830 5378 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-f0fe13-7cb54c466b-s9889" Oct 30 21:14:19.626229 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:14:19.622894 5378 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-f0fe13-7cb54c466b-s9889"] Oct 30 21:14:19.671123 ip-10-0-138-149 kubenswrapper[5378]: W1030 21:14:19.671093 5378 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podef431fc4_98f7_4a6f_85c1_5989dc6665f6.slice/crio-538f498ed66d6d1a568281a092abf914f6d782248b917cd9c70841b78bdf35ff WatchSource:0}: Error finding container 538f498ed66d6d1a568281a092abf914f6d782248b917cd9c70841b78bdf35ff: Status 404 returned error can't find the container with id 538f498ed66d6d1a568281a092abf914f6d782248b917cd9c70841b78bdf35ff Oct 30 21:14:19.671838 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:14:19.671809 5378 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-f0fe13-7447d4f599-smmk4"] Oct 30 21:14:19.674078 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:14:19.674050 5378 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-f0fe13-7cb54c466b-s9889" event={"ID":"97e8a2e3-2092-4e54-8621-c33b9404e163","Type":"ContainerStarted","Data":"d3927d574116af81a83b53fa83770ceeb2a995fe549823f83250b66e65c765a6"} Oct 30 21:14:19.889208 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:14:19.889114 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:14:20.226252 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:14:20.226148 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:14:20.424858 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:14:20.424818 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:14:20.683444 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:14:20.683405 5378 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-f0fe13-7447d4f599-smmk4" event={"ID":"ef431fc4-98f7-4a6f-85c1-5989dc6665f6","Type":"ContainerStarted","Data":"cfa15a4128ca7c145d9e98b59f50ed566c296132662c645d33961cbfe7323012"} Oct 30 21:14:20.683630 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:14:20.683611 5378 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-f0fe13-7447d4f599-smmk4" Oct 30 21:14:20.683710 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:14:20.683659 5378 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-f0fe13-7447d4f599-smmk4" event={"ID":"ef431fc4-98f7-4a6f-85c1-5989dc6665f6","Type":"ContainerStarted","Data":"538f498ed66d6d1a568281a092abf914f6d782248b917cd9c70841b78bdf35ff"} Oct 30 21:14:20.685093 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:14:20.685070 5378 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-f0fe13-7cb54c466b-s9889" event={"ID":"97e8a2e3-2092-4e54-8621-c33b9404e163","Type":"ContainerStarted","Data":"93857c45ffc0ff65342d61c5ea09a58d8e1479acabb10b769ea9d46ac713259f"} Oct 30 21:14:20.685205 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:14:20.685191 5378 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-a-f0fe13-7cb54c466b-s9889" Oct 30 21:14:20.804759 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:14:20.804713 5378 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-f0fe13-7447d4f599-smmk4" podStartSLOduration=2.804701575 podStartE2EDuration="2.804701575s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:14:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:14:20.802905137 +0000 UTC m=+2151.231019069" watchObservedRunningTime="2026-10-30 21:14:20.804701575 +0000 UTC m=+2151.232815530" Oct 30 21:14:20.968487 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:14:20.968366 5378 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-a-f0fe13-7cb54c466b-s9889" podStartSLOduration=2.968342049 podStartE2EDuration="2.968342049s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:14:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:14:20.955483254 +0000 UTC m=+2151.383597173" watchObservedRunningTime="2026-10-30 21:14:20.968342049 +0000 UTC m=+2151.396455972" Oct 30 21:14:20.978265 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:14:20.973882 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:14:23.076186 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:14:23.076151 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:14:23.236999 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:14:23.236944 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:14:23.476541 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:14:23.476505 5378 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-4d1110-openshift-default-6576975b6c-gvzcm"] Oct 30 21:14:23.476757 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:14:23.476732 5378 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w1-4d1110-openshift-default-6576975b6c-gvzcm" podUID="1b385f61-b9d9-4f5d-8e71-7d86cd0d867d" containerName="istio-proxy" containerID="cri-o://048a3802a64f4473e97950a182b7b9278ad0397bbbb2b485c86e47964adaf321" gracePeriod=30 Oct 30 21:14:23.678659 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:14:23.678615 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:14:23.887032 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:14:23.886998 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:14:25.830851 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:14:25.830819 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:14:27.911205 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:14:27.911171 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:14:28.053172 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:14:28.053119 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:14:28.054090 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:14:28.054060 5378 patch_prober.go:29] interesting pod/e2e-shared-a-w1-4d1110-openshift-default-6576975b6c-gvzcm container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 21:14:28.054238 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:14:28.054156 5378 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-w1-4d1110-openshift-default-6576975b6c-gvzcm" podUID="1b385f61-b9d9-4f5d-8e71-7d86cd0d867d" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 21:14:28.267451 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:14:28.267359 5378 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-255f05-openshift-default-7db8bf49f-6bfml"] Oct 30 21:14:28.267645 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:14:28.267584 5378 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w2-255f05-openshift-default-7db8bf49f-6bfml" podUID="4955d910-8544-4a2b-a1fd-b414734aed56" containerName="istio-proxy" containerID="cri-o://54e9e27c0e5662cdc7708cccc54fa81ed8abb32708b17e78bee0d401d7486463" gracePeriod=30 Oct 30 21:14:28.273313 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:14:28.273281 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:14:28.758830 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:14:28.758793 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:14:28.790538 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:14:28.790461 5378 generic.go:356] "Generic (PLEG): container finished" podID="1b385f61-b9d9-4f5d-8e71-7d86cd0d867d" containerID="048a3802a64f4473e97950a182b7b9278ad0397bbbb2b485c86e47964adaf321" exitCode=0 Oct 30 21:14:28.790538 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:14:28.790517 5378 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-4d1110-openshift-default-6576975b6c-gvzcm" event={"ID":"1b385f61-b9d9-4f5d-8e71-7d86cd0d867d","Type":"ContainerDied","Data":"048a3802a64f4473e97950a182b7b9278ad0397bbbb2b485c86e47964adaf321"} Oct 30 21:14:28.790773 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:14:28.790577 5378 scope.go:122] "RemoveContainer" containerID="b75ef819ca3ba0cace281a1b6a7a8378147dfa4b114d39242abfe2a7d5c9bb44" Oct 30 21:14:29.189180 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:14:29.189157 5378 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w1-4d1110-openshift-default-6576975b6c-gvzcm" Oct 30 21:14:29.300800 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:14:29.300769 5378 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1b385f61-b9d9-4f5d-8e71-7d86cd0d867d-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1b385f61-b9d9-4f5d-8e71-7d86cd0d867d-istiod-ca-cert\") pod \"1b385f61-b9d9-4f5d-8e71-7d86cd0d867d\" (UID: \"1b385f61-b9d9-4f5d-8e71-7d86cd0d867d\") " Oct 30 21:14:29.300992 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:14:29.300928 5378 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1b385f61-b9d9-4f5d-8e71-7d86cd0d867d-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1b385f61-b9d9-4f5d-8e71-7d86cd0d867d-istio-data\") pod \"1b385f61-b9d9-4f5d-8e71-7d86cd0d867d\" (UID: \"1b385f61-b9d9-4f5d-8e71-7d86cd0d867d\") " Oct 30 21:14:29.300992 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:14:29.300956 5378 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1b385f61-b9d9-4f5d-8e71-7d86cd0d867d-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1b385f61-b9d9-4f5d-8e71-7d86cd0d867d-credential-socket\") pod \"1b385f61-b9d9-4f5d-8e71-7d86cd0d867d\" (UID: \"1b385f61-b9d9-4f5d-8e71-7d86cd0d867d\") " Oct 30 21:14:29.300992 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:14:29.300974 5378 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1b385f61-b9d9-4f5d-8e71-7d86cd0d867d-kube-api-access-vgfxz\" (UniqueName: \"kubernetes.io/projected/1b385f61-b9d9-4f5d-8e71-7d86cd0d867d-kube-api-access-vgfxz\") pod \"1b385f61-b9d9-4f5d-8e71-7d86cd0d867d\" (UID: \"1b385f61-b9d9-4f5d-8e71-7d86cd0d867d\") " Oct 30 21:14:29.301165 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:14:29.301011 5378 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1b385f61-b9d9-4f5d-8e71-7d86cd0d867d-istio-token\" (UniqueName: \"kubernetes.io/projected/1b385f61-b9d9-4f5d-8e71-7d86cd0d867d-istio-token\") pod \"1b385f61-b9d9-4f5d-8e71-7d86cd0d867d\" (UID: \"1b385f61-b9d9-4f5d-8e71-7d86cd0d867d\") " Oct 30 21:14:29.301165 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:14:29.301042 5378 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1b385f61-b9d9-4f5d-8e71-7d86cd0d867d-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1b385f61-b9d9-4f5d-8e71-7d86cd0d867d-istio-envoy\") pod \"1b385f61-b9d9-4f5d-8e71-7d86cd0d867d\" (UID: \"1b385f61-b9d9-4f5d-8e71-7d86cd0d867d\") " Oct 30 21:14:29.301282 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:14:29.301163 5378 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1b385f61-b9d9-4f5d-8e71-7d86cd0d867d-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1b385f61-b9d9-4f5d-8e71-7d86cd0d867d-workload-certs\") pod \"1b385f61-b9d9-4f5d-8e71-7d86cd0d867d\" (UID: \"1b385f61-b9d9-4f5d-8e71-7d86cd0d867d\") " Oct 30 21:14:29.301282 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:14:29.301211 5378 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/1b385f61-b9d9-4f5d-8e71-7d86cd0d867d-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1b385f61-b9d9-4f5d-8e71-7d86cd0d867d-istio-podinfo\") pod \"1b385f61-b9d9-4f5d-8e71-7d86cd0d867d\" (UID: \"1b385f61-b9d9-4f5d-8e71-7d86cd0d867d\") " Oct 30 21:14:29.301282 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:14:29.301227 5378 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1b385f61-b9d9-4f5d-8e71-7d86cd0d867d-istio-data" pod "1b385f61-b9d9-4f5d-8e71-7d86cd0d867d" (UID: "1b385f61-b9d9-4f5d-8e71-7d86cd0d867d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:14:29.301282 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:14:29.301246 5378 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1b385f61-b9d9-4f5d-8e71-7d86cd0d867d-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1b385f61-b9d9-4f5d-8e71-7d86cd0d867d-workload-socket\") pod \"1b385f61-b9d9-4f5d-8e71-7d86cd0d867d\" (UID: \"1b385f61-b9d9-4f5d-8e71-7d86cd0d867d\") " Oct 30 21:14:29.301282 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:14:29.301252 5378 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1b385f61-b9d9-4f5d-8e71-7d86cd0d867d-credential-socket" pod "1b385f61-b9d9-4f5d-8e71-7d86cd0d867d" (UID: "1b385f61-b9d9-4f5d-8e71-7d86cd0d867d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:14:29.301282 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:14:29.301261 5378 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1b385f61-b9d9-4f5d-8e71-7d86cd0d867d-istiod-ca-cert" pod "1b385f61-b9d9-4f5d-8e71-7d86cd0d867d" (UID: "1b385f61-b9d9-4f5d-8e71-7d86cd0d867d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:14:29.301628 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:14:29.301583 5378 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1b385f61-b9d9-4f5d-8e71-7d86cd0d867d-workload-socket" pod "1b385f61-b9d9-4f5d-8e71-7d86cd0d867d" (UID: "1b385f61-b9d9-4f5d-8e71-7d86cd0d867d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:14:29.302158 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:14:29.302077 5378 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1b385f61-b9d9-4f5d-8e71-7d86cd0d867d-istiod-ca-cert\") on node \"ip-10-0-138-149.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:14:29.302158 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:14:29.302104 5378 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1b385f61-b9d9-4f5d-8e71-7d86cd0d867d-istio-data\") on node \"ip-10-0-138-149.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:14:29.302158 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:14:29.302119 5378 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1b385f61-b9d9-4f5d-8e71-7d86cd0d867d-credential-socket\") on node \"ip-10-0-138-149.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:14:29.302158 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:14:29.302134 5378 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1b385f61-b9d9-4f5d-8e71-7d86cd0d867d-workload-socket\") on node \"ip-10-0-138-149.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:14:29.302554 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:14:29.302457 5378 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1b385f61-b9d9-4f5d-8e71-7d86cd0d867d-workload-certs" pod "1b385f61-b9d9-4f5d-8e71-7d86cd0d867d" (UID: "1b385f61-b9d9-4f5d-8e71-7d86cd0d867d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:14:29.303748 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:14:29.303707 5378 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1b385f61-b9d9-4f5d-8e71-7d86cd0d867d-kube-api-access-vgfxz" pod "1b385f61-b9d9-4f5d-8e71-7d86cd0d867d" (UID: "1b385f61-b9d9-4f5d-8e71-7d86cd0d867d"). InnerVolumeSpecName "kube-api-access-vgfxz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:14:29.303860 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:14:29.303823 5378 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1b385f61-b9d9-4f5d-8e71-7d86cd0d867d-istio-envoy" pod "1b385f61-b9d9-4f5d-8e71-7d86cd0d867d" (UID: "1b385f61-b9d9-4f5d-8e71-7d86cd0d867d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:14:29.303975 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:14:29.303954 5378 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/1b385f61-b9d9-4f5d-8e71-7d86cd0d867d-istio-podinfo" pod "1b385f61-b9d9-4f5d-8e71-7d86cd0d867d" (UID: "1b385f61-b9d9-4f5d-8e71-7d86cd0d867d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 21:14:29.304022 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:14:29.303977 5378 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1b385f61-b9d9-4f5d-8e71-7d86cd0d867d-istio-token" pod "1b385f61-b9d9-4f5d-8e71-7d86cd0d867d" (UID: "1b385f61-b9d9-4f5d-8e71-7d86cd0d867d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:14:29.403456 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:14:29.403412 5378 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vgfxz\" (UniqueName: \"kubernetes.io/projected/1b385f61-b9d9-4f5d-8e71-7d86cd0d867d-kube-api-access-vgfxz\") on node \"ip-10-0-138-149.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:14:29.403456 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:14:29.403450 5378 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1b385f61-b9d9-4f5d-8e71-7d86cd0d867d-istio-token\") on node \"ip-10-0-138-149.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:14:29.403706 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:14:29.403466 5378 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1b385f61-b9d9-4f5d-8e71-7d86cd0d867d-istio-envoy\") on node \"ip-10-0-138-149.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:14:29.403706 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:14:29.403478 5378 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1b385f61-b9d9-4f5d-8e71-7d86cd0d867d-workload-certs\") on node \"ip-10-0-138-149.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:14:29.403706 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:14:29.403490 5378 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1b385f61-b9d9-4f5d-8e71-7d86cd0d867d-istio-podinfo\") on node \"ip-10-0-138-149.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:14:29.799344 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:14:29.799311 5378 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w1-4d1110-openshift-default-6576975b6c-gvzcm" Oct 30 21:14:29.799344 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:14:29.799337 5378 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-4d1110-openshift-default-6576975b6c-gvzcm" event={"ID":"1b385f61-b9d9-4f5d-8e71-7d86cd0d867d","Type":"ContainerDied","Data":"89b9de5fc01269735421ccbfcc29bec60d14af2f60651411e270d4c5a4eba5aa"} Oct 30 21:14:29.799544 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:14:29.799391 5378 scope.go:122] "RemoveContainer" containerID="048a3802a64f4473e97950a182b7b9278ad0397bbbb2b485c86e47964adaf321" Oct 30 21:14:30.026622 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:14:30.026531 5378 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-4d1110-openshift-default-6576975b6c-gvzcm"] Oct 30 21:14:30.076302 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:14:30.076270 5378 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-4d1110-openshift-default-6576975b6c-gvzcm"] Oct 30 21:14:30.214190 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:14:30.214159 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:14:31.144938 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:14:31.144906 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:14:31.694198 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:14:31.694165 5378 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-a-f0fe13-7cb54c466b-s9889" Oct 30 21:14:31.694699 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:14:31.694242 5378 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-f0fe13-7447d4f599-smmk4" Oct 30 21:14:31.743258 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:14:31.743195 5378 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1b385f61-b9d9-4f5d-8e71-7d86cd0d867d" path="/var/lib/kubelet/pods/1b385f61-b9d9-4f5d-8e71-7d86cd0d867d/volumes" Oct 30 21:14:33.537177 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:14:33.537118 5378 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-255f05-openshift-default-7db8bf49f-6bfml" Oct 30 21:14:33.664184 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:14:33.664153 5378 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4955d910-8544-4a2b-a1fd-b414734aed56-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4955d910-8544-4a2b-a1fd-b414734aed56-istio-data\") pod \"4955d910-8544-4a2b-a1fd-b414734aed56\" (UID: \"4955d910-8544-4a2b-a1fd-b414734aed56\") " Oct 30 21:14:33.664358 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:14:33.664215 5378 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/4955d910-8544-4a2b-a1fd-b414734aed56-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4955d910-8544-4a2b-a1fd-b414734aed56-istio-podinfo\") pod \"4955d910-8544-4a2b-a1fd-b414734aed56\" (UID: \"4955d910-8544-4a2b-a1fd-b414734aed56\") " Oct 30 21:14:33.664358 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:14:33.664236 5378 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4955d910-8544-4a2b-a1fd-b414734aed56-kube-api-access-9mtg6\" (UniqueName: \"kubernetes.io/projected/4955d910-8544-4a2b-a1fd-b414734aed56-kube-api-access-9mtg6\") pod \"4955d910-8544-4a2b-a1fd-b414734aed56\" (UID: \"4955d910-8544-4a2b-a1fd-b414734aed56\") " Oct 30 21:14:33.664483 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:14:33.664391 5378 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4955d910-8544-4a2b-a1fd-b414734aed56-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4955d910-8544-4a2b-a1fd-b414734aed56-workload-socket\") pod \"4955d910-8544-4a2b-a1fd-b414734aed56\" (UID: \"4955d910-8544-4a2b-a1fd-b414734aed56\") " Oct 30 21:14:33.664483 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:14:33.664470 5378 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4955d910-8544-4a2b-a1fd-b414734aed56-istio-data" pod "4955d910-8544-4a2b-a1fd-b414734aed56" (UID: "4955d910-8544-4a2b-a1fd-b414734aed56"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:14:33.664603 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:14:33.664529 5378 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4955d910-8544-4a2b-a1fd-b414734aed56-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4955d910-8544-4a2b-a1fd-b414734aed56-istio-envoy\") pod \"4955d910-8544-4a2b-a1fd-b414734aed56\" (UID: \"4955d910-8544-4a2b-a1fd-b414734aed56\") " Oct 30 21:14:33.664603 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:14:33.664587 5378 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4955d910-8544-4a2b-a1fd-b414734aed56-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4955d910-8544-4a2b-a1fd-b414734aed56-credential-socket\") pod \"4955d910-8544-4a2b-a1fd-b414734aed56\" (UID: \"4955d910-8544-4a2b-a1fd-b414734aed56\") " Oct 30 21:14:33.664730 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:14:33.664686 5378 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4955d910-8544-4a2b-a1fd-b414734aed56-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4955d910-8544-4a2b-a1fd-b414734aed56-istiod-ca-cert\") pod \"4955d910-8544-4a2b-a1fd-b414734aed56\" (UID: \"4955d910-8544-4a2b-a1fd-b414734aed56\") " Oct 30 21:14:33.664730 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:14:33.664691 5378 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4955d910-8544-4a2b-a1fd-b414734aed56-workload-socket" pod "4955d910-8544-4a2b-a1fd-b414734aed56" (UID: "4955d910-8544-4a2b-a1fd-b414734aed56"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:14:33.664730 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:14:33.664719 5378 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4955d910-8544-4a2b-a1fd-b414734aed56-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4955d910-8544-4a2b-a1fd-b414734aed56-workload-certs\") pod \"4955d910-8544-4a2b-a1fd-b414734aed56\" (UID: \"4955d910-8544-4a2b-a1fd-b414734aed56\") " Oct 30 21:14:33.664892 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:14:33.664749 5378 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4955d910-8544-4a2b-a1fd-b414734aed56-istio-token\" (UniqueName: \"kubernetes.io/projected/4955d910-8544-4a2b-a1fd-b414734aed56-istio-token\") pod \"4955d910-8544-4a2b-a1fd-b414734aed56\" (UID: \"4955d910-8544-4a2b-a1fd-b414734aed56\") " Oct 30 21:14:33.665057 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:14:33.665034 5378 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4955d910-8544-4a2b-a1fd-b414734aed56-workload-certs" pod "4955d910-8544-4a2b-a1fd-b414734aed56" (UID: "4955d910-8544-4a2b-a1fd-b414734aed56"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:14:33.665057 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:14:33.665052 5378 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4955d910-8544-4a2b-a1fd-b414734aed56-istiod-ca-cert" pod "4955d910-8544-4a2b-a1fd-b414734aed56" (UID: "4955d910-8544-4a2b-a1fd-b414734aed56"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:14:33.665357 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:14:33.665330 5378 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4955d910-8544-4a2b-a1fd-b414734aed56-credential-socket" pod "4955d910-8544-4a2b-a1fd-b414734aed56" (UID: "4955d910-8544-4a2b-a1fd-b414734aed56"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:14:33.666657 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:14:33.666160 5378 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4955d910-8544-4a2b-a1fd-b414734aed56-istio-data\") on node \"ip-10-0-138-149.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:14:33.666657 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:14:33.666189 5378 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4955d910-8544-4a2b-a1fd-b414734aed56-workload-socket\") on node \"ip-10-0-138-149.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:14:33.666657 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:14:33.666237 5378 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4955d910-8544-4a2b-a1fd-b414734aed56-istiod-ca-cert\") on node \"ip-10-0-138-149.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:14:33.666657 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:14:33.666258 5378 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4955d910-8544-4a2b-a1fd-b414734aed56-workload-certs\") on node \"ip-10-0-138-149.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:14:33.670500 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:14:33.666957 5378 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/4955d910-8544-4a2b-a1fd-b414734aed56-istio-podinfo" pod "4955d910-8544-4a2b-a1fd-b414734aed56" (UID: "4955d910-8544-4a2b-a1fd-b414734aed56"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 21:14:33.670500 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:14:33.666998 5378 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4955d910-8544-4a2b-a1fd-b414734aed56-istio-envoy" pod "4955d910-8544-4a2b-a1fd-b414734aed56" (UID: "4955d910-8544-4a2b-a1fd-b414734aed56"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:14:33.670500 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:14:33.667525 5378 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4955d910-8544-4a2b-a1fd-b414734aed56-kube-api-access-9mtg6" pod "4955d910-8544-4a2b-a1fd-b414734aed56" (UID: "4955d910-8544-4a2b-a1fd-b414734aed56"). InnerVolumeSpecName "kube-api-access-9mtg6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:14:33.671507 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:14:33.671480 5378 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4955d910-8544-4a2b-a1fd-b414734aed56-istio-token" pod "4955d910-8544-4a2b-a1fd-b414734aed56" (UID: "4955d910-8544-4a2b-a1fd-b414734aed56"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:14:33.749842 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:14:33.749813 5378 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-d061d0/distinct-w2-d061d0-kserve-679c7cd7c4-8czrv"] Oct 30 21:14:33.767253 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:14:33.767226 5378 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4955d910-8544-4a2b-a1fd-b414734aed56-istio-podinfo\") on node \"ip-10-0-138-149.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:14:33.767253 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:14:33.767254 5378 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9mtg6\" (UniqueName: \"kubernetes.io/projected/4955d910-8544-4a2b-a1fd-b414734aed56-kube-api-access-9mtg6\") on node \"ip-10-0-138-149.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:14:33.767421 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:14:33.767265 5378 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4955d910-8544-4a2b-a1fd-b414734aed56-istio-envoy\") on node \"ip-10-0-138-149.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:14:33.767421 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:14:33.767274 5378 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4955d910-8544-4a2b-a1fd-b414734aed56-credential-socket\") on node \"ip-10-0-138-149.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:14:33.767421 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:14:33.767282 5378 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4955d910-8544-4a2b-a1fd-b414734aed56-istio-token\") on node \"ip-10-0-138-149.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:14:33.776812 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:14:33.776782 5378 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-d061d0/simulator-w2-d061d0-kserve-8484f9678b-lnszm"] Oct 30 21:14:33.830476 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:14:33.830405 5378 generic.go:356] "Generic (PLEG): container finished" podID="4955d910-8544-4a2b-a1fd-b414734aed56" containerID="54e9e27c0e5662cdc7708cccc54fa81ed8abb32708b17e78bee0d401d7486463" exitCode=0 Oct 30 21:14:33.830640 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:14:33.830477 5378 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-255f05-openshift-default-7db8bf49f-6bfml" Oct 30 21:14:33.830640 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:14:33.830477 5378 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-255f05-openshift-default-7db8bf49f-6bfml" event={"ID":"4955d910-8544-4a2b-a1fd-b414734aed56","Type":"ContainerDied","Data":"54e9e27c0e5662cdc7708cccc54fa81ed8abb32708b17e78bee0d401d7486463"} Oct 30 21:14:33.830734 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:14:33.830641 5378 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-255f05-openshift-default-7db8bf49f-6bfml" event={"ID":"4955d910-8544-4a2b-a1fd-b414734aed56","Type":"ContainerDied","Data":"60f7f7e917a7eacc2f8b89615553e3b3091aa25dbcb5915b630b570d9cc3b751"} Oct 30 21:14:33.830734 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:14:33.830673 5378 scope.go:122] "RemoveContainer" containerID="54e9e27c0e5662cdc7708cccc54fa81ed8abb32708b17e78bee0d401d7486463" Oct 30 21:14:33.840762 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:14:33.840743 5378 scope.go:122] "RemoveContainer" containerID="f436cafc6a27c4e22cde96a1784d61e2c40ce352ae7696fc0974d6435e60a1bf" Oct 30 21:14:33.851058 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:14:33.851037 5378 scope.go:122] "RemoveContainer" containerID="54e9e27c0e5662cdc7708cccc54fa81ed8abb32708b17e78bee0d401d7486463" Oct 30 21:14:33.851339 ip-10-0-138-149 kubenswrapper[5378]: E1030 21:14:33.851315 5378 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"54e9e27c0e5662cdc7708cccc54fa81ed8abb32708b17e78bee0d401d7486463\": container with ID starting with 54e9e27c0e5662cdc7708cccc54fa81ed8abb32708b17e78bee0d401d7486463 not found: ID does not exist" containerID="54e9e27c0e5662cdc7708cccc54fa81ed8abb32708b17e78bee0d401d7486463" Oct 30 21:14:33.851442 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:14:33.851345 5378 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"54e9e27c0e5662cdc7708cccc54fa81ed8abb32708b17e78bee0d401d7486463"} err="failed to get container status \"54e9e27c0e5662cdc7708cccc54fa81ed8abb32708b17e78bee0d401d7486463\": rpc error: code = NotFound desc = could not find container \"54e9e27c0e5662cdc7708cccc54fa81ed8abb32708b17e78bee0d401d7486463\": container with ID starting with 54e9e27c0e5662cdc7708cccc54fa81ed8abb32708b17e78bee0d401d7486463 not found: ID does not exist" Oct 30 21:14:33.851442 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:14:33.851361 5378 scope.go:122] "RemoveContainer" containerID="f436cafc6a27c4e22cde96a1784d61e2c40ce352ae7696fc0974d6435e60a1bf" Oct 30 21:14:33.851670 ip-10-0-138-149 kubenswrapper[5378]: E1030 21:14:33.851642 5378 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f436cafc6a27c4e22cde96a1784d61e2c40ce352ae7696fc0974d6435e60a1bf\": container with ID starting with f436cafc6a27c4e22cde96a1784d61e2c40ce352ae7696fc0974d6435e60a1bf not found: ID does not exist" containerID="f436cafc6a27c4e22cde96a1784d61e2c40ce352ae7696fc0974d6435e60a1bf" Oct 30 21:14:33.851727 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:14:33.851675 5378 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f436cafc6a27c4e22cde96a1784d61e2c40ce352ae7696fc0974d6435e60a1bf"} err="failed to get container status \"f436cafc6a27c4e22cde96a1784d61e2c40ce352ae7696fc0974d6435e60a1bf\": rpc error: code = NotFound desc = could not find container \"f436cafc6a27c4e22cde96a1784d61e2c40ce352ae7696fc0974d6435e60a1bf\": container with ID starting with f436cafc6a27c4e22cde96a1784d61e2c40ce352ae7696fc0974d6435e60a1bf not found: ID does not exist" Oct 30 21:14:33.875782 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:14:33.875755 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:14:34.117635 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:14:34.117604 5378 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-255f05-openshift-default-7db8bf49f-6bfml"] Oct 30 21:14:34.120549 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:14:34.120528 5378 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-255f05-openshift-default-7db8bf49f-6bfml"] Oct 30 21:14:35.743095 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:14:35.743061 5378 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4955d910-8544-4a2b-a1fd-b414734aed56" path="/var/lib/kubelet/pods/4955d910-8544-4a2b-a1fd-b414734aed56/volumes" Oct 30 21:14:36.019165 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:14:36.019091 5378 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w1-677b45-78946fddc-s5lpc"] Oct 30 21:14:36.276324 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:14:36.276240 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:14:36.427927 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:14:36.427897 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:14:36.566435 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:14:36.566342 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:14:41.022499 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:14:41.022022 5378 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-677b45-78946fddc-s5lpc" podUID="7ad59559-fd63-4973-9037-43c23ce150db" containerName="payload-pre-processing" containerID="cri-o://1af25aba1bb88880623f1c6caf9a8ff2772ff41ec6acf3b30ff542fb59e119cd" gracePeriod=25 Oct 30 21:14:41.160806 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:14:41.160771 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:14:41.402592 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:14:41.401132 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:14:41.434633 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:14:41.433855 5378 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-677b45-78946fddc-s5lpc" Oct 30 21:14:41.512772 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:14:41.512736 5378 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7ad59559-fd63-4973-9037-43c23ce150db-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7ad59559-fd63-4973-9037-43c23ce150db-plugins-config-volume\") pod \"7ad59559-fd63-4973-9037-43c23ce150db\" (UID: \"7ad59559-fd63-4973-9037-43c23ce150db\") " Oct 30 21:14:41.513173 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:14:41.513146 5378 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7ad59559-fd63-4973-9037-43c23ce150db-plugins-config-volume" pod "7ad59559-fd63-4973-9037-43c23ce150db" (UID: "7ad59559-fd63-4973-9037-43c23ce150db"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:14:41.513311 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:14:41.513286 5378 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7ad59559-fd63-4973-9037-43c23ce150db-kube-api-access-b4lgs\" (UniqueName: \"kubernetes.io/projected/7ad59559-fd63-4973-9037-43c23ce150db-kube-api-access-b4lgs\") pod \"7ad59559-fd63-4973-9037-43c23ce150db\" (UID: \"7ad59559-fd63-4973-9037-43c23ce150db\") " Oct 30 21:14:41.514143 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:14:41.514116 5378 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7ad59559-fd63-4973-9037-43c23ce150db-plugins-config-volume\") on node \"ip-10-0-138-149.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:14:41.515853 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:14:41.515821 5378 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7ad59559-fd63-4973-9037-43c23ce150db-kube-api-access-b4lgs" pod "7ad59559-fd63-4973-9037-43c23ce150db" (UID: "7ad59559-fd63-4973-9037-43c23ce150db"). InnerVolumeSpecName "kube-api-access-b4lgs". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:14:41.616019 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:14:41.615931 5378 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b4lgs\" (UniqueName: \"kubernetes.io/projected/7ad59559-fd63-4973-9037-43c23ce150db-kube-api-access-b4lgs\") on node \"ip-10-0-138-149.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:14:41.933048 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:14:41.932956 5378 generic.go:356] "Generic (PLEG): container finished" podID="7ad59559-fd63-4973-9037-43c23ce150db" containerID="1af25aba1bb88880623f1c6caf9a8ff2772ff41ec6acf3b30ff542fb59e119cd" exitCode=0 Oct 30 21:14:41.933048 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:14:41.933019 5378 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-677b45-78946fddc-s5lpc" Oct 30 21:14:41.933278 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:14:41.933020 5378 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-677b45-78946fddc-s5lpc" event={"ID":"7ad59559-fd63-4973-9037-43c23ce150db","Type":"ContainerDied","Data":"1af25aba1bb88880623f1c6caf9a8ff2772ff41ec6acf3b30ff542fb59e119cd"} Oct 30 21:14:41.933278 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:14:41.933177 5378 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-677b45-78946fddc-s5lpc" event={"ID":"7ad59559-fd63-4973-9037-43c23ce150db","Type":"ContainerDied","Data":"8171e2e8ab4acfe77b47fe63cbc39ca332c6133a24a74ff57257b19b41229b0e"} Oct 30 21:14:41.933278 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:14:41.933208 5378 scope.go:122] "RemoveContainer" containerID="1af25aba1bb88880623f1c6caf9a8ff2772ff41ec6acf3b30ff542fb59e119cd" Oct 30 21:14:41.943322 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:14:41.943301 5378 scope.go:122] "RemoveContainer" containerID="1af25aba1bb88880623f1c6caf9a8ff2772ff41ec6acf3b30ff542fb59e119cd" Oct 30 21:14:41.943625 ip-10-0-138-149 kubenswrapper[5378]: E1030 21:14:41.943603 5378 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1af25aba1bb88880623f1c6caf9a8ff2772ff41ec6acf3b30ff542fb59e119cd\": container with ID starting with 1af25aba1bb88880623f1c6caf9a8ff2772ff41ec6acf3b30ff542fb59e119cd not found: ID does not exist" containerID="1af25aba1bb88880623f1c6caf9a8ff2772ff41ec6acf3b30ff542fb59e119cd" Oct 30 21:14:41.943702 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:14:41.943632 5378 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1af25aba1bb88880623f1c6caf9a8ff2772ff41ec6acf3b30ff542fb59e119cd"} err="failed to get container status \"1af25aba1bb88880623f1c6caf9a8ff2772ff41ec6acf3b30ff542fb59e119cd\": rpc error: code = NotFound desc = could not find container \"1af25aba1bb88880623f1c6caf9a8ff2772ff41ec6acf3b30ff542fb59e119cd\": container with ID starting with 1af25aba1bb88880623f1c6caf9a8ff2772ff41ec6acf3b30ff542fb59e119cd not found: ID does not exist" Oct 30 21:14:42.103000 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:14:42.102970 5378 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w1-677b45-78946fddc-s5lpc"] Oct 30 21:14:42.152916 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:14:42.152885 5378 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w1-677b45-78946fddc-s5lpc"] Oct 30 21:14:42.178697 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:14:42.178666 5378 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-2f9eff-8594f457db-5m52x"] Oct 30 21:14:42.179307 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:14:42.179294 5378 cpu_manager.go:414] "containerMap: removing container" podUID="1b385f61-b9d9-4f5d-8e71-7d86cd0d867d" containerName="istio-proxy" Oct 30 21:14:42.179351 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:14:42.179309 5378 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1b385f61-b9d9-4f5d-8e71-7d86cd0d867d" containerName="istio-proxy" Oct 30 21:14:42.179351 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:14:42.179349 5378 cpu_manager.go:414] "containerMap: removing container" podUID="1b385f61-b9d9-4f5d-8e71-7d86cd0d867d" containerName="istio-proxy" Oct 30 21:14:42.179423 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:14:42.179355 5378 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1b385f61-b9d9-4f5d-8e71-7d86cd0d867d" containerName="istio-proxy" Oct 30 21:14:42.179423 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:14:42.179366 5378 cpu_manager.go:414] "containerMap: removing container" podUID="7ad59559-fd63-4973-9037-43c23ce150db" containerName="payload-pre-processing" Oct 30 21:14:42.179423 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:14:42.179371 5378 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7ad59559-fd63-4973-9037-43c23ce150db" containerName="payload-pre-processing" Oct 30 21:14:42.179423 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:14:42.179380 5378 cpu_manager.go:414] "containerMap: removing container" podUID="4955d910-8544-4a2b-a1fd-b414734aed56" containerName="istio-proxy" Oct 30 21:14:42.179423 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:14:42.179385 5378 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4955d910-8544-4a2b-a1fd-b414734aed56" containerName="istio-proxy" Oct 30 21:14:42.179423 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:14:42.179405 5378 cpu_manager.go:414] "containerMap: removing container" podUID="4955d910-8544-4a2b-a1fd-b414734aed56" containerName="istio-proxy" Oct 30 21:14:42.179423 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:14:42.179410 5378 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4955d910-8544-4a2b-a1fd-b414734aed56" containerName="istio-proxy" Oct 30 21:14:42.179648 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:14:42.179505 5378 memory_manager.go:365] "RemoveStaleState removing state" podUID="4955d910-8544-4a2b-a1fd-b414734aed56" containerName="istio-proxy" Oct 30 21:14:42.179648 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:14:42.179518 5378 memory_manager.go:365] "RemoveStaleState removing state" podUID="1b385f61-b9d9-4f5d-8e71-7d86cd0d867d" containerName="istio-proxy" Oct 30 21:14:42.179648 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:14:42.179524 5378 memory_manager.go:365] "RemoveStaleState removing state" podUID="7ad59559-fd63-4973-9037-43c23ce150db" containerName="payload-pre-processing" Oct 30 21:14:42.184505 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:14:42.184454 5378 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-2f9eff-8594f457db-5m52x" Oct 30 21:14:42.224451 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:14:42.224424 5378 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fd6d4723-fe0b-4eae-94fb-c7b61de829c5-metrics-certs\") pod \"maas-api-e2e-inf-a-2f9eff-8594f457db-5m52x\" (UID: \"fd6d4723-fe0b-4eae-94fb-c7b61de829c5\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-2f9eff-8594f457db-5m52x" Oct 30 21:14:42.224599 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:14:42.224465 5378 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fd6d4723-fe0b-4eae-94fb-c7b61de829c5-openshift-service-ca\") pod \"maas-api-e2e-inf-a-2f9eff-8594f457db-5m52x\" (UID: \"fd6d4723-fe0b-4eae-94fb-c7b61de829c5\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-2f9eff-8594f457db-5m52x" Oct 30 21:14:42.224651 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:14:42.224631 5378 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fd6d4723-fe0b-4eae-94fb-c7b61de829c5-maas-api-tls\") pod \"maas-api-e2e-inf-a-2f9eff-8594f457db-5m52x\" (UID: \"fd6d4723-fe0b-4eae-94fb-c7b61de829c5\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-2f9eff-8594f457db-5m52x" Oct 30 21:14:42.224709 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:14:42.224694 5378 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g8bqr\" (UniqueName: \"kubernetes.io/projected/fd6d4723-fe0b-4eae-94fb-c7b61de829c5-kube-api-access-g8bqr\") pod \"maas-api-e2e-inf-a-2f9eff-8594f457db-5m52x\" (UID: \"fd6d4723-fe0b-4eae-94fb-c7b61de829c5\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-2f9eff-8594f457db-5m52x" Oct 30 21:14:42.238073 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:14:42.238047 5378 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-a-2f9eff-serving-cert\"" Oct 30 21:14:42.242624 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:14:42.242594 5378 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-2f9eff-8594f457db-5m52x"] Oct 30 21:14:42.328023 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:14:42.327987 5378 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fd6d4723-fe0b-4eae-94fb-c7b61de829c5-metrics-certs\") pod \"maas-api-e2e-inf-a-2f9eff-8594f457db-5m52x\" (UID: \"fd6d4723-fe0b-4eae-94fb-c7b61de829c5\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-2f9eff-8594f457db-5m52x" Oct 30 21:14:42.328208 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:14:42.328053 5378 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fd6d4723-fe0b-4eae-94fb-c7b61de829c5-openshift-service-ca\") pod \"maas-api-e2e-inf-a-2f9eff-8594f457db-5m52x\" (UID: \"fd6d4723-fe0b-4eae-94fb-c7b61de829c5\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-2f9eff-8594f457db-5m52x" Oct 30 21:14:42.328208 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:14:42.328173 5378 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fd6d4723-fe0b-4eae-94fb-c7b61de829c5-maas-api-tls\") pod \"maas-api-e2e-inf-a-2f9eff-8594f457db-5m52x\" (UID: \"fd6d4723-fe0b-4eae-94fb-c7b61de829c5\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-2f9eff-8594f457db-5m52x" Oct 30 21:14:42.328328 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:14:42.328275 5378 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g8bqr\" (UniqueName: \"kubernetes.io/projected/fd6d4723-fe0b-4eae-94fb-c7b61de829c5-kube-api-access-g8bqr\") pod \"maas-api-e2e-inf-a-2f9eff-8594f457db-5m52x\" (UID: \"fd6d4723-fe0b-4eae-94fb-c7b61de829c5\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-2f9eff-8594f457db-5m52x" Oct 30 21:14:42.328606 ip-10-0-138-149 kubenswrapper[5378]: E1030 21:14:42.328550 5378 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-inf-a-2f9eff-serving-cert: secret "maas-api-e2e-inf-a-2f9eff-serving-cert" not found Oct 30 21:14:42.328701 ip-10-0-138-149 kubenswrapper[5378]: E1030 21:14:42.328655 5378 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fd6d4723-fe0b-4eae-94fb-c7b61de829c5-maas-api-tls podName:fd6d4723-fe0b-4eae-94fb-c7b61de829c5 nodeName:}" failed. No retries permitted until 2026-10-30 21:14:42.828635523 +0000 UTC m=+2173.256749422 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/fd6d4723-fe0b-4eae-94fb-c7b61de829c5-maas-api-tls") pod "maas-api-e2e-inf-a-2f9eff-8594f457db-5m52x" (UID: "fd6d4723-fe0b-4eae-94fb-c7b61de829c5") : secret "maas-api-e2e-inf-a-2f9eff-serving-cert" not found Oct 30 21:14:42.329168 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:14:42.329126 5378 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fd6d4723-fe0b-4eae-94fb-c7b61de829c5-openshift-service-ca\") pod \"maas-api-e2e-inf-a-2f9eff-8594f457db-5m52x\" (UID: \"fd6d4723-fe0b-4eae-94fb-c7b61de829c5\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-2f9eff-8594f457db-5m52x" Oct 30 21:14:42.333418 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:14:42.333385 5378 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fd6d4723-fe0b-4eae-94fb-c7b61de829c5-metrics-certs\") pod \"maas-api-e2e-inf-a-2f9eff-8594f457db-5m52x\" (UID: \"fd6d4723-fe0b-4eae-94fb-c7b61de829c5\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-2f9eff-8594f457db-5m52x" Oct 30 21:14:42.389526 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:14:42.389498 5378 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g8bqr\" (UniqueName: \"kubernetes.io/projected/fd6d4723-fe0b-4eae-94fb-c7b61de829c5-kube-api-access-g8bqr\") pod \"maas-api-e2e-inf-a-2f9eff-8594f457db-5m52x\" (UID: \"fd6d4723-fe0b-4eae-94fb-c7b61de829c5\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-2f9eff-8594f457db-5m52x" Oct 30 21:14:42.390305 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:14:42.390282 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:14:42.573246 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:14:42.570459 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:14:42.733587 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:14:42.729488 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:14:42.842640 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:14:42.842520 5378 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fd6d4723-fe0b-4eae-94fb-c7b61de829c5-maas-api-tls\") pod \"maas-api-e2e-inf-a-2f9eff-8594f457db-5m52x\" (UID: \"fd6d4723-fe0b-4eae-94fb-c7b61de829c5\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-2f9eff-8594f457db-5m52x" Oct 30 21:14:42.878584 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:14:42.875184 5378 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fd6d4723-fe0b-4eae-94fb-c7b61de829c5-maas-api-tls\") pod \"maas-api-e2e-inf-a-2f9eff-8594f457db-5m52x\" (UID: \"fd6d4723-fe0b-4eae-94fb-c7b61de829c5\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-2f9eff-8594f457db-5m52x" Oct 30 21:14:43.098272 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:14:43.098244 5378 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-2f9eff-8594f457db-5m52x" Oct 30 21:14:43.479111 ip-10-0-138-149 kubenswrapper[5378]: W1030 21:14:43.479075 5378 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfd6d4723_fe0b_4eae_94fb_c7b61de829c5.slice/crio-5e62c404cc16fba8ae0c69eaa518abdee924fda4c327c7656c1d7bf10442a113 WatchSource:0}: Error finding container 5e62c404cc16fba8ae0c69eaa518abdee924fda4c327c7656c1d7bf10442a113: Status 404 returned error can't find the container with id 5e62c404cc16fba8ae0c69eaa518abdee924fda4c327c7656c1d7bf10442a113 Oct 30 21:14:43.479646 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:14:43.479589 5378 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-2f9eff-8594f457db-5m52x"] Oct 30 21:14:43.745942 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:14:43.745866 5378 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7ad59559-fd63-4973-9037-43c23ce150db" path="/var/lib/kubelet/pods/7ad59559-fd63-4973-9037-43c23ce150db/volumes" Oct 30 21:14:43.746977 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:14:43.746955 5378 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-b888d0-openshift-default-65d785fbfc-vjfvl"] Oct 30 21:14:43.747151 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:14:43.747132 5378 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-a-b888d0-openshift-default-65d785fbfc-vjfvl" podUID="072e9c4b-6360-4cdd-8594-517158d18132" containerName="istio-proxy" containerID="cri-o://bd617042ca38094008c7ae1180bcd1226f139d12c879f98fe53eede7f73a6f85" gracePeriod=30 Oct 30 21:14:43.960123 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:14:43.960086 5378 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-2f9eff-8594f457db-5m52x" event={"ID":"fd6d4723-fe0b-4eae-94fb-c7b61de829c5","Type":"ContainerStarted","Data":"5e62c404cc16fba8ae0c69eaa518abdee924fda4c327c7656c1d7bf10442a113"} Oct 30 21:14:44.968734 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:14:44.968690 5378 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-2f9eff-8594f457db-5m52x" event={"ID":"fd6d4723-fe0b-4eae-94fb-c7b61de829c5","Type":"ContainerStarted","Data":"c893ee8fd320efc00243b332c70b50f6222eb24333264493df8692a72fc3f461"} Oct 30 21:14:44.969154 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:14:44.968768 5378 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-2f9eff-8594f457db-5m52x" Oct 30 21:14:45.086183 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:14:45.086128 5378 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-2f9eff-8594f457db-5m52x" podStartSLOduration=2.320343899 podStartE2EDuration="3.086113571s" totalImagesPullingTime="765.769672ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:14:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:14:45.085671972 +0000 UTC m=+2175.513785882" watchObservedRunningTime="2026-10-30 21:14:45.086113571 +0000 UTC m=+2175.514227487" Oct 30 21:14:47.945763 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:14:47.945725 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:14:48.453540 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:14:48.453495 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:14:48.778038 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:14:48.778004 5378 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w2-d061d0/distinct-w2-d061d0-kserve-679c7cd7c4-8czrv" podUID="2c139b9f-c7df-4675-b431-95d4136095e5" containerName="main" containerID="cri-o://c7558118e9ce72affc672e1d18263ca42d790131c00a7dac78fd384bcd70f1aa" gracePeriod=45 Oct 30 21:14:48.797931 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:14:48.797903 5378 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w2-d061d0/simulator-w2-d061d0-kserve-8484f9678b-lnszm" podUID="525fac02-b21c-4dd5-b553-72bcc36062c5" containerName="main" containerID="cri-o://d9dc8acba58ff31b072e5ad4cf311194b0404765220cf6074ef3265a4246dd5c" gracePeriod=45 Oct 30 21:14:48.798484 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:14:48.798460 5378 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-d061d0/distinct-w2-d061d0-kserve-679c7cd7c4-8czrv" podUID="2c139b9f-c7df-4675-b431-95d4136095e5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/ready\": dial tcp 10.133.0.69:8000: connect: connection refused" Oct 30 21:14:48.838344 ip-10-0-138-149 kubenswrapper[5378]: E1030 21:14:48.838198 5378 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q6mx7" containerName="extractor" Oct 30 21:14:49.005644 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:14:49.005543 5378 generic.go:356] "Generic (PLEG): container finished" podID="2c139b9f-c7df-4675-b431-95d4136095e5" containerID="c7558118e9ce72affc672e1d18263ca42d790131c00a7dac78fd384bcd70f1aa" exitCode=0 Oct 30 21:14:49.005644 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:14:49.005594 5378 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-d061d0/distinct-w2-d061d0-kserve-679c7cd7c4-8czrv" event={"ID":"2c139b9f-c7df-4675-b431-95d4136095e5","Type":"ContainerDied","Data":"c7558118e9ce72affc672e1d18263ca42d790131c00a7dac78fd384bcd70f1aa"} Oct 30 21:14:49.007606 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:14:49.007520 5378 generic.go:356] "Generic (PLEG): container finished" podID="525fac02-b21c-4dd5-b553-72bcc36062c5" containerID="d9dc8acba58ff31b072e5ad4cf311194b0404765220cf6074ef3265a4246dd5c" exitCode=0 Oct 30 21:14:49.007694 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:14:49.007617 5378 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-d061d0/simulator-w2-d061d0-kserve-8484f9678b-lnszm" event={"ID":"525fac02-b21c-4dd5-b553-72bcc36062c5","Type":"ContainerDied","Data":"d9dc8acba58ff31b072e5ad4cf311194b0404765220cf6074ef3265a4246dd5c"} Oct 30 21:14:49.009529 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:14:49.009508 5378 generic.go:356] "Generic (PLEG): container finished" podID="072e9c4b-6360-4cdd-8594-517158d18132" containerID="bd617042ca38094008c7ae1180bcd1226f139d12c879f98fe53eede7f73a6f85" exitCode=0 Oct 30 21:14:49.009631 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:14:49.009614 5378 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-b888d0-openshift-default-65d785fbfc-vjfvl" event={"ID":"072e9c4b-6360-4cdd-8594-517158d18132","Type":"ContainerDied","Data":"bd617042ca38094008c7ae1180bcd1226f139d12c879f98fe53eede7f73a6f85"} Oct 30 21:14:49.108484 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:14:49.108454 5378 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-d061d0/distinct-w2-d061d0-kserve-679c7cd7c4-8czrv" Oct 30 21:14:49.133222 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:14:49.133195 5378 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-b888d0-openshift-default-65d785fbfc-vjfvl" Oct 30 21:14:49.136705 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:14:49.136684 5378 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-d061d0/simulator-w2-d061d0-kserve-8484f9678b-lnszm" Oct 30 21:14:49.249838 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:14:49.249042 5378 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/525fac02-b21c-4dd5-b553-72bcc36062c5-home\" (UniqueName: \"kubernetes.io/empty-dir/525fac02-b21c-4dd5-b553-72bcc36062c5-home\") pod \"525fac02-b21c-4dd5-b553-72bcc36062c5\" (UID: \"525fac02-b21c-4dd5-b553-72bcc36062c5\") " Oct 30 21:14:49.249838 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:14:49.249082 5378 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2c139b9f-c7df-4675-b431-95d4136095e5-tls-certs\" (UniqueName: \"kubernetes.io/secret/2c139b9f-c7df-4675-b431-95d4136095e5-tls-certs\") pod \"2c139b9f-c7df-4675-b431-95d4136095e5\" (UID: \"2c139b9f-c7df-4675-b431-95d4136095e5\") " Oct 30 21:14:49.249838 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:14:49.249127 5378 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/072e9c4b-6360-4cdd-8594-517158d18132-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/072e9c4b-6360-4cdd-8594-517158d18132-istiod-ca-cert\") pod \"072e9c4b-6360-4cdd-8594-517158d18132\" (UID: \"072e9c4b-6360-4cdd-8594-517158d18132\") " Oct 30 21:14:49.249838 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:14:49.249192 5378 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2c139b9f-c7df-4675-b431-95d4136095e5-kube-api-access-b5tz4\" (UniqueName: \"kubernetes.io/projected/2c139b9f-c7df-4675-b431-95d4136095e5-kube-api-access-b5tz4\") pod \"2c139b9f-c7df-4675-b431-95d4136095e5\" (UID: \"2c139b9f-c7df-4675-b431-95d4136095e5\") " Oct 30 21:14:49.249838 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:14:49.249238 5378 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2c139b9f-c7df-4675-b431-95d4136095e5-home\" (UniqueName: \"kubernetes.io/empty-dir/2c139b9f-c7df-4675-b431-95d4136095e5-home\") pod \"2c139b9f-c7df-4675-b431-95d4136095e5\" (UID: \"2c139b9f-c7df-4675-b431-95d4136095e5\") " Oct 30 21:14:49.249838 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:14:49.249307 5378 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/525fac02-b21c-4dd5-b553-72bcc36062c5-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/525fac02-b21c-4dd5-b553-72bcc36062c5-model-cache\") pod \"525fac02-b21c-4dd5-b553-72bcc36062c5\" (UID: \"525fac02-b21c-4dd5-b553-72bcc36062c5\") " Oct 30 21:14:49.249838 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:14:49.249331 5378 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/525fac02-b21c-4dd5-b553-72bcc36062c5-home" pod "525fac02-b21c-4dd5-b553-72bcc36062c5" (UID: "525fac02-b21c-4dd5-b553-72bcc36062c5"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:14:49.249838 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:14:49.249347 5378 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/072e9c4b-6360-4cdd-8594-517158d18132-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/072e9c4b-6360-4cdd-8594-517158d18132-istio-podinfo\") pod \"072e9c4b-6360-4cdd-8594-517158d18132\" (UID: \"072e9c4b-6360-4cdd-8594-517158d18132\") " Oct 30 21:14:49.249838 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:14:49.249379 5378 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/072e9c4b-6360-4cdd-8594-517158d18132-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/072e9c4b-6360-4cdd-8594-517158d18132-istio-envoy\") pod \"072e9c4b-6360-4cdd-8594-517158d18132\" (UID: \"072e9c4b-6360-4cdd-8594-517158d18132\") " Oct 30 21:14:49.249838 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:14:49.249477 5378 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2c139b9f-c7df-4675-b431-95d4136095e5-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2c139b9f-c7df-4675-b431-95d4136095e5-model-cache\") pod \"2c139b9f-c7df-4675-b431-95d4136095e5\" (UID: \"2c139b9f-c7df-4675-b431-95d4136095e5\") " Oct 30 21:14:49.249838 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:14:49.249508 5378 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/525fac02-b21c-4dd5-b553-72bcc36062c5-kube-api-access-n2nfj\" (UniqueName: \"kubernetes.io/projected/525fac02-b21c-4dd5-b553-72bcc36062c5-kube-api-access-n2nfj\") pod \"525fac02-b21c-4dd5-b553-72bcc36062c5\" (UID: \"525fac02-b21c-4dd5-b553-72bcc36062c5\") " Oct 30 21:14:49.249838 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:14:49.249545 5378 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2c139b9f-c7df-4675-b431-95d4136095e5-home" pod "2c139b9f-c7df-4675-b431-95d4136095e5" (UID: "2c139b9f-c7df-4675-b431-95d4136095e5"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:14:49.249838 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:14:49.249550 5378 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/525fac02-b21c-4dd5-b553-72bcc36062c5-dshm\" (UniqueName: \"kubernetes.io/empty-dir/525fac02-b21c-4dd5-b553-72bcc36062c5-dshm\") pod \"525fac02-b21c-4dd5-b553-72bcc36062c5\" (UID: \"525fac02-b21c-4dd5-b553-72bcc36062c5\") " Oct 30 21:14:49.249838 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:14:49.249619 5378 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/072e9c4b-6360-4cdd-8594-517158d18132-istiod-ca-cert" pod "072e9c4b-6360-4cdd-8594-517158d18132" (UID: "072e9c4b-6360-4cdd-8594-517158d18132"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:14:49.249838 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:14:49.249630 5378 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2c139b9f-c7df-4675-b431-95d4136095e5-dshm\" (UniqueName: \"kubernetes.io/empty-dir/2c139b9f-c7df-4675-b431-95d4136095e5-dshm\") pod \"2c139b9f-c7df-4675-b431-95d4136095e5\" (UID: \"2c139b9f-c7df-4675-b431-95d4136095e5\") " Oct 30 21:14:49.249838 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:14:49.249667 5378 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/072e9c4b-6360-4cdd-8594-517158d18132-kube-api-access-jd4hw\" (UniqueName: \"kubernetes.io/projected/072e9c4b-6360-4cdd-8594-517158d18132-kube-api-access-jd4hw\") pod \"072e9c4b-6360-4cdd-8594-517158d18132\" (UID: \"072e9c4b-6360-4cdd-8594-517158d18132\") " Oct 30 21:14:49.249838 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:14:49.249702 5378 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/072e9c4b-6360-4cdd-8594-517158d18132-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/072e9c4b-6360-4cdd-8594-517158d18132-workload-certs\") pod \"072e9c4b-6360-4cdd-8594-517158d18132\" (UID: \"072e9c4b-6360-4cdd-8594-517158d18132\") " Oct 30 21:14:49.251031 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:14:49.250165 5378 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2c139b9f-c7df-4675-b431-95d4136095e5-model-cache" pod "2c139b9f-c7df-4675-b431-95d4136095e5" (UID: "2c139b9f-c7df-4675-b431-95d4136095e5"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:14:49.251031 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:14:49.250459 5378 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/525fac02-b21c-4dd5-b553-72bcc36062c5-model-cache" pod "525fac02-b21c-4dd5-b553-72bcc36062c5" (UID: "525fac02-b21c-4dd5-b553-72bcc36062c5"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:14:49.251031 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:14:49.249737 5378 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/525fac02-b21c-4dd5-b553-72bcc36062c5-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/525fac02-b21c-4dd5-b553-72bcc36062c5-tmp-dir\") pod \"525fac02-b21c-4dd5-b553-72bcc36062c5\" (UID: \"525fac02-b21c-4dd5-b553-72bcc36062c5\") " Oct 30 21:14:49.251200 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:14:49.251056 5378 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/072e9c4b-6360-4cdd-8594-517158d18132-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/072e9c4b-6360-4cdd-8594-517158d18132-credential-socket\") pod \"072e9c4b-6360-4cdd-8594-517158d18132\" (UID: \"072e9c4b-6360-4cdd-8594-517158d18132\") " Oct 30 21:14:49.251200 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:14:49.251126 5378 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/072e9c4b-6360-4cdd-8594-517158d18132-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/072e9c4b-6360-4cdd-8594-517158d18132-workload-socket\") pod \"072e9c4b-6360-4cdd-8594-517158d18132\" (UID: \"072e9c4b-6360-4cdd-8594-517158d18132\") " Oct 30 21:14:49.251200 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:14:49.251162 5378 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2c139b9f-c7df-4675-b431-95d4136095e5-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2c139b9f-c7df-4675-b431-95d4136095e5-tmp-dir\") pod \"2c139b9f-c7df-4675-b431-95d4136095e5\" (UID: \"2c139b9f-c7df-4675-b431-95d4136095e5\") " Oct 30 21:14:49.251441 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:14:49.251235 5378 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/072e9c4b-6360-4cdd-8594-517158d18132-istio-token\" (UniqueName: \"kubernetes.io/projected/072e9c4b-6360-4cdd-8594-517158d18132-istio-token\") pod \"072e9c4b-6360-4cdd-8594-517158d18132\" (UID: \"072e9c4b-6360-4cdd-8594-517158d18132\") " Oct 30 21:14:49.251441 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:14:49.251286 5378 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/525fac02-b21c-4dd5-b553-72bcc36062c5-tls-certs\" (UniqueName: \"kubernetes.io/secret/525fac02-b21c-4dd5-b553-72bcc36062c5-tls-certs\") pod \"525fac02-b21c-4dd5-b553-72bcc36062c5\" (UID: \"525fac02-b21c-4dd5-b553-72bcc36062c5\") " Oct 30 21:14:49.251441 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:14:49.251334 5378 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/072e9c4b-6360-4cdd-8594-517158d18132-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/072e9c4b-6360-4cdd-8594-517158d18132-istio-data\") pod \"072e9c4b-6360-4cdd-8594-517158d18132\" (UID: \"072e9c4b-6360-4cdd-8594-517158d18132\") " Oct 30 21:14:49.252628 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:14:49.252293 5378 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/525fac02-b21c-4dd5-b553-72bcc36062c5-home\") on node \"ip-10-0-138-149.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:14:49.252628 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:14:49.252322 5378 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/072e9c4b-6360-4cdd-8594-517158d18132-istiod-ca-cert\") on node \"ip-10-0-138-149.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:14:49.252628 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:14:49.252338 5378 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2c139b9f-c7df-4675-b431-95d4136095e5-home\") on node \"ip-10-0-138-149.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:14:49.252628 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:14:49.252352 5378 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/525fac02-b21c-4dd5-b553-72bcc36062c5-model-cache\") on node \"ip-10-0-138-149.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:14:49.252628 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:14:49.252365 5378 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2c139b9f-c7df-4675-b431-95d4136095e5-model-cache\") on node \"ip-10-0-138-149.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:14:49.252983 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:14:49.252639 5378 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/072e9c4b-6360-4cdd-8594-517158d18132-istio-data" pod "072e9c4b-6360-4cdd-8594-517158d18132" (UID: "072e9c4b-6360-4cdd-8594-517158d18132"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:14:49.252983 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:14:49.252824 5378 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/072e9c4b-6360-4cdd-8594-517158d18132-workload-certs" pod "072e9c4b-6360-4cdd-8594-517158d18132" (UID: "072e9c4b-6360-4cdd-8594-517158d18132"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:14:49.253113 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:14:49.252997 5378 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/525fac02-b21c-4dd5-b553-72bcc36062c5-tmp-dir" pod "525fac02-b21c-4dd5-b553-72bcc36062c5" (UID: "525fac02-b21c-4dd5-b553-72bcc36062c5"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:14:49.253113 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:14:49.253018 5378 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2c139b9f-c7df-4675-b431-95d4136095e5-kube-api-access-b5tz4" pod "2c139b9f-c7df-4675-b431-95d4136095e5" (UID: "2c139b9f-c7df-4675-b431-95d4136095e5"). InnerVolumeSpecName "kube-api-access-b5tz4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:14:49.253226 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:14:49.253186 5378 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/072e9c4b-6360-4cdd-8594-517158d18132-credential-socket" pod "072e9c4b-6360-4cdd-8594-517158d18132" (UID: "072e9c4b-6360-4cdd-8594-517158d18132"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:14:49.253390 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:14:49.253366 5378 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/072e9c4b-6360-4cdd-8594-517158d18132-workload-socket" pod "072e9c4b-6360-4cdd-8594-517158d18132" (UID: "072e9c4b-6360-4cdd-8594-517158d18132"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:14:49.253665 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:14:49.253642 5378 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2c139b9f-c7df-4675-b431-95d4136095e5-tmp-dir" pod "2c139b9f-c7df-4675-b431-95d4136095e5" (UID: "2c139b9f-c7df-4675-b431-95d4136095e5"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:14:49.254451 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:14:49.254424 5378 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/072e9c4b-6360-4cdd-8594-517158d18132-istio-podinfo" pod "072e9c4b-6360-4cdd-8594-517158d18132" (UID: "072e9c4b-6360-4cdd-8594-517158d18132"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 21:14:49.254646 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:14:49.254616 5378 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2c139b9f-c7df-4675-b431-95d4136095e5-tls-certs" pod "2c139b9f-c7df-4675-b431-95d4136095e5" (UID: "2c139b9f-c7df-4675-b431-95d4136095e5"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:14:49.254764 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:14:49.254724 5378 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/072e9c4b-6360-4cdd-8594-517158d18132-istio-envoy" pod "072e9c4b-6360-4cdd-8594-517158d18132" (UID: "072e9c4b-6360-4cdd-8594-517158d18132"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:14:49.255410 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:14:49.255333 5378 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/525fac02-b21c-4dd5-b553-72bcc36062c5-kube-api-access-n2nfj" pod "525fac02-b21c-4dd5-b553-72bcc36062c5" (UID: "525fac02-b21c-4dd5-b553-72bcc36062c5"). InnerVolumeSpecName "kube-api-access-n2nfj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:14:49.255410 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:14:49.255379 5378 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2c139b9f-c7df-4675-b431-95d4136095e5-dshm" pod "2c139b9f-c7df-4675-b431-95d4136095e5" (UID: "2c139b9f-c7df-4675-b431-95d4136095e5"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:14:49.255559 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:14:49.255454 5378 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/072e9c4b-6360-4cdd-8594-517158d18132-kube-api-access-jd4hw" pod "072e9c4b-6360-4cdd-8594-517158d18132" (UID: "072e9c4b-6360-4cdd-8594-517158d18132"). InnerVolumeSpecName "kube-api-access-jd4hw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:14:49.255657 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:14:49.255617 5378 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/525fac02-b21c-4dd5-b553-72bcc36062c5-dshm" pod "525fac02-b21c-4dd5-b553-72bcc36062c5" (UID: "525fac02-b21c-4dd5-b553-72bcc36062c5"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:14:49.256248 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:14:49.256226 5378 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/525fac02-b21c-4dd5-b553-72bcc36062c5-tls-certs" pod "525fac02-b21c-4dd5-b553-72bcc36062c5" (UID: "525fac02-b21c-4dd5-b553-72bcc36062c5"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:14:49.256992 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:14:49.256955 5378 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/072e9c4b-6360-4cdd-8594-517158d18132-istio-token" pod "072e9c4b-6360-4cdd-8594-517158d18132" (UID: "072e9c4b-6360-4cdd-8594-517158d18132"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:14:49.355703 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:14:49.354824 5378 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/072e9c4b-6360-4cdd-8594-517158d18132-istio-token\") on node \"ip-10-0-138-149.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:14:49.355703 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:14:49.354858 5378 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/525fac02-b21c-4dd5-b553-72bcc36062c5-tls-certs\") on node \"ip-10-0-138-149.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:14:49.355703 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:14:49.354872 5378 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/072e9c4b-6360-4cdd-8594-517158d18132-istio-data\") on node \"ip-10-0-138-149.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:14:49.355703 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:14:49.354884 5378 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2c139b9f-c7df-4675-b431-95d4136095e5-tls-certs\") on node \"ip-10-0-138-149.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:14:49.355703 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:14:49.354907 5378 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b5tz4\" (UniqueName: \"kubernetes.io/projected/2c139b9f-c7df-4675-b431-95d4136095e5-kube-api-access-b5tz4\") on node \"ip-10-0-138-149.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:14:49.355703 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:14:49.354920 5378 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/072e9c4b-6360-4cdd-8594-517158d18132-istio-podinfo\") on node \"ip-10-0-138-149.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:14:49.355703 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:14:49.354932 5378 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/072e9c4b-6360-4cdd-8594-517158d18132-istio-envoy\") on node \"ip-10-0-138-149.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:14:49.355703 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:14:49.354945 5378 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n2nfj\" (UniqueName: \"kubernetes.io/projected/525fac02-b21c-4dd5-b553-72bcc36062c5-kube-api-access-n2nfj\") on node \"ip-10-0-138-149.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:14:49.355703 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:14:49.354957 5378 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/525fac02-b21c-4dd5-b553-72bcc36062c5-dshm\") on node \"ip-10-0-138-149.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:14:49.355703 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:14:49.354969 5378 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2c139b9f-c7df-4675-b431-95d4136095e5-dshm\") on node \"ip-10-0-138-149.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:14:49.355703 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:14:49.354980 5378 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jd4hw\" (UniqueName: \"kubernetes.io/projected/072e9c4b-6360-4cdd-8594-517158d18132-kube-api-access-jd4hw\") on node \"ip-10-0-138-149.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:14:49.355703 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:14:49.354992 5378 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/072e9c4b-6360-4cdd-8594-517158d18132-workload-certs\") on node \"ip-10-0-138-149.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:14:49.355703 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:14:49.355003 5378 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/525fac02-b21c-4dd5-b553-72bcc36062c5-tmp-dir\") on node \"ip-10-0-138-149.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:14:49.355703 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:14:49.355015 5378 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/072e9c4b-6360-4cdd-8594-517158d18132-credential-socket\") on node \"ip-10-0-138-149.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:14:49.355703 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:14:49.355026 5378 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/072e9c4b-6360-4cdd-8594-517158d18132-workload-socket\") on node \"ip-10-0-138-149.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:14:49.355703 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:14:49.355039 5378 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2c139b9f-c7df-4675-b431-95d4136095e5-tmp-dir\") on node \"ip-10-0-138-149.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:14:50.017345 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:14:50.017317 5378 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-d061d0/distinct-w2-d061d0-kserve-679c7cd7c4-8czrv" Oct 30 21:14:50.017345 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:14:50.017327 5378 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-d061d0/distinct-w2-d061d0-kserve-679c7cd7c4-8czrv" event={"ID":"2c139b9f-c7df-4675-b431-95d4136095e5","Type":"ContainerDied","Data":"3c41ff3842d6b2c8544d235e3321d4bb093903d35c776332295c8263f40d4606"} Oct 30 21:14:50.017916 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:14:50.017376 5378 scope.go:122] "RemoveContainer" containerID="c7558118e9ce72affc672e1d18263ca42d790131c00a7dac78fd384bcd70f1aa" Oct 30 21:14:50.019025 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:14:50.019001 5378 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-d061d0/simulator-w2-d061d0-kserve-8484f9678b-lnszm" Oct 30 21:14:50.019161 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:14:50.019003 5378 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-d061d0/simulator-w2-d061d0-kserve-8484f9678b-lnszm" event={"ID":"525fac02-b21c-4dd5-b553-72bcc36062c5","Type":"ContainerDied","Data":"4a5c2994f6a916c441c0b7b26adee925ceaa6169372f6cd5ca3e911d57fb23fe"} Oct 30 21:14:50.020716 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:14:50.020696 5378 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-b888d0-openshift-default-65d785fbfc-vjfvl" Oct 30 21:14:50.020716 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:14:50.020696 5378 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-b888d0-openshift-default-65d785fbfc-vjfvl" event={"ID":"072e9c4b-6360-4cdd-8594-517158d18132","Type":"ContainerDied","Data":"9f764b2dc467c8542cfeef6131960aea2a7730887c7d89458185c4becde22aaa"} Oct 30 21:14:50.029164 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:14:50.029142 5378 scope.go:122] "RemoveContainer" containerID="d9dc8acba58ff31b072e5ad4cf311194b0404765220cf6074ef3265a4246dd5c" Oct 30 21:14:50.037855 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:14:50.037838 5378 scope.go:122] "RemoveContainer" containerID="bd617042ca38094008c7ae1180bcd1226f139d12c879f98fe53eede7f73a6f85" Oct 30 21:14:50.221388 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:14:50.221352 5378 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-d061d0/simulator-w2-d061d0-kserve-8484f9678b-lnszm"] Oct 30 21:14:50.227664 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:14:50.227630 5378 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w2-d061d0/simulator-w2-d061d0-kserve-8484f9678b-lnszm"] Oct 30 21:14:50.339480 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:14:50.339389 5378 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-d061d0/distinct-w2-d061d0-kserve-679c7cd7c4-8czrv"] Oct 30 21:14:50.343371 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:14:50.343344 5378 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w2-d061d0/distinct-w2-d061d0-kserve-679c7cd7c4-8czrv"] Oct 30 21:14:50.470134 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:14:50.470099 5378 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-b888d0-openshift-default-65d785fbfc-vjfvl"] Oct 30 21:14:50.484144 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:14:50.484117 5378 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-a-b888d0-openshift-default-65d785fbfc-vjfvl"] Oct 30 21:14:50.981455 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:14:50.981430 5378 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-2f9eff-8594f457db-5m52x" Oct 30 21:14:51.744823 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:14:51.744788 5378 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="072e9c4b-6360-4cdd-8594-517158d18132" path="/var/lib/kubelet/pods/072e9c4b-6360-4cdd-8594-517158d18132/volumes" Oct 30 21:14:51.745328 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:14:51.745313 5378 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2c139b9f-c7df-4675-b431-95d4136095e5" path="/var/lib/kubelet/pods/2c139b9f-c7df-4675-b431-95d4136095e5/volumes" Oct 30 21:14:51.745781 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:14:51.745765 5378 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="525fac02-b21c-4dd5-b553-72bcc36062c5" path="/var/lib/kubelet/pods/525fac02-b21c-4dd5-b553-72bcc36062c5/volumes" Oct 30 21:14:52.446580 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:14:52.446528 5378 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-r9g59"] Oct 30 21:14:53.385069 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:14:53.385036 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:14:53.508671 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:14:53.508627 5378 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-2b425d-openshift-default-f547dbcbf-2dmdq"] Oct 30 21:14:53.508912 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:14:53.508883 5378 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w5-2b425d-openshift-default-f547dbcbf-2dmdq" podUID="0359a851-6211-4192-9569-255a5b3dc518" containerName="istio-proxy" containerID="cri-o://e9fd800b4f60a2b6e715d2da7ae20fc8df38a0a3782046ca707f2856ac9d78f0" gracePeriod=30 Oct 30 21:14:58.316156 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:14:58.316110 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:14:58.825973 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:14:58.825948 5378 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-2b425d-openshift-default-f547dbcbf-2dmdq" Oct 30 21:14:58.941903 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:14:58.941831 5378 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0359a851-6211-4192-9569-255a5b3dc518-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0359a851-6211-4192-9569-255a5b3dc518-workload-certs\") pod \"0359a851-6211-4192-9569-255a5b3dc518\" (UID: \"0359a851-6211-4192-9569-255a5b3dc518\") " Oct 30 21:14:58.942085 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:14:58.941920 5378 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0359a851-6211-4192-9569-255a5b3dc518-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0359a851-6211-4192-9569-255a5b3dc518-workload-socket\") pod \"0359a851-6211-4192-9569-255a5b3dc518\" (UID: \"0359a851-6211-4192-9569-255a5b3dc518\") " Oct 30 21:14:58.942085 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:14:58.941973 5378 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0359a851-6211-4192-9569-255a5b3dc518-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0359a851-6211-4192-9569-255a5b3dc518-istio-data\") pod \"0359a851-6211-4192-9569-255a5b3dc518\" (UID: \"0359a851-6211-4192-9569-255a5b3dc518\") " Oct 30 21:14:58.942085 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:14:58.942007 5378 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0359a851-6211-4192-9569-255a5b3dc518-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0359a851-6211-4192-9569-255a5b3dc518-credential-socket\") pod \"0359a851-6211-4192-9569-255a5b3dc518\" (UID: \"0359a851-6211-4192-9569-255a5b3dc518\") " Oct 30 21:14:58.942264 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:14:58.942118 5378 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0359a851-6211-4192-9569-255a5b3dc518-workload-certs" pod "0359a851-6211-4192-9569-255a5b3dc518" (UID: "0359a851-6211-4192-9569-255a5b3dc518"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:14:58.942264 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:14:58.942195 5378 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0359a851-6211-4192-9569-255a5b3dc518-istio-token\" (UniqueName: \"kubernetes.io/projected/0359a851-6211-4192-9569-255a5b3dc518-istio-token\") pod \"0359a851-6211-4192-9569-255a5b3dc518\" (UID: \"0359a851-6211-4192-9569-255a5b3dc518\") " Oct 30 21:14:58.942264 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:14:58.942232 5378 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0359a851-6211-4192-9569-255a5b3dc518-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0359a851-6211-4192-9569-255a5b3dc518-istiod-ca-cert\") pod \"0359a851-6211-4192-9569-255a5b3dc518\" (UID: \"0359a851-6211-4192-9569-255a5b3dc518\") " Oct 30 21:14:58.942435 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:14:58.942296 5378 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0359a851-6211-4192-9569-255a5b3dc518-kube-api-access-m4wv2\" (UniqueName: \"kubernetes.io/projected/0359a851-6211-4192-9569-255a5b3dc518-kube-api-access-m4wv2\") pod \"0359a851-6211-4192-9569-255a5b3dc518\" (UID: \"0359a851-6211-4192-9569-255a5b3dc518\") " Oct 30 21:14:58.942435 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:14:58.942306 5378 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0359a851-6211-4192-9569-255a5b3dc518-workload-socket" pod "0359a851-6211-4192-9569-255a5b3dc518" (UID: "0359a851-6211-4192-9569-255a5b3dc518"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:14:58.942435 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:14:58.942315 5378 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0359a851-6211-4192-9569-255a5b3dc518-istio-data" pod "0359a851-6211-4192-9569-255a5b3dc518" (UID: "0359a851-6211-4192-9569-255a5b3dc518"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:14:58.942435 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:14:58.942324 5378 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/0359a851-6211-4192-9569-255a5b3dc518-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0359a851-6211-4192-9569-255a5b3dc518-istio-podinfo\") pod \"0359a851-6211-4192-9569-255a5b3dc518\" (UID: \"0359a851-6211-4192-9569-255a5b3dc518\") " Oct 30 21:14:58.942435 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:14:58.942359 5378 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0359a851-6211-4192-9569-255a5b3dc518-credential-socket" pod "0359a851-6211-4192-9569-255a5b3dc518" (UID: "0359a851-6211-4192-9569-255a5b3dc518"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:14:58.942435 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:14:58.942366 5378 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0359a851-6211-4192-9569-255a5b3dc518-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0359a851-6211-4192-9569-255a5b3dc518-istio-envoy\") pod \"0359a851-6211-4192-9569-255a5b3dc518\" (UID: \"0359a851-6211-4192-9569-255a5b3dc518\") " Oct 30 21:14:58.942762 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:14:58.942676 5378 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0359a851-6211-4192-9569-255a5b3dc518-istiod-ca-cert" pod "0359a851-6211-4192-9569-255a5b3dc518" (UID: "0359a851-6211-4192-9569-255a5b3dc518"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:14:58.943077 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:14:58.943055 5378 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0359a851-6211-4192-9569-255a5b3dc518-workload-certs\") on node \"ip-10-0-138-149.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:14:58.943143 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:14:58.943083 5378 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0359a851-6211-4192-9569-255a5b3dc518-workload-socket\") on node \"ip-10-0-138-149.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:14:58.943143 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:14:58.943099 5378 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0359a851-6211-4192-9569-255a5b3dc518-istio-data\") on node \"ip-10-0-138-149.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:14:58.943143 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:14:58.943113 5378 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0359a851-6211-4192-9569-255a5b3dc518-credential-socket\") on node \"ip-10-0-138-149.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:14:58.943143 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:14:58.943129 5378 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0359a851-6211-4192-9569-255a5b3dc518-istiod-ca-cert\") on node \"ip-10-0-138-149.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:14:58.944591 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:14:58.944536 5378 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/0359a851-6211-4192-9569-255a5b3dc518-istio-podinfo" pod "0359a851-6211-4192-9569-255a5b3dc518" (UID: "0359a851-6211-4192-9569-255a5b3dc518"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 21:14:58.944712 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:14:58.944659 5378 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0359a851-6211-4192-9569-255a5b3dc518-kube-api-access-m4wv2" pod "0359a851-6211-4192-9569-255a5b3dc518" (UID: "0359a851-6211-4192-9569-255a5b3dc518"). InnerVolumeSpecName "kube-api-access-m4wv2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:14:58.944895 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:14:58.944875 5378 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0359a851-6211-4192-9569-255a5b3dc518-istio-envoy" pod "0359a851-6211-4192-9569-255a5b3dc518" (UID: "0359a851-6211-4192-9569-255a5b3dc518"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:14:58.945049 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:14:58.945027 5378 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0359a851-6211-4192-9569-255a5b3dc518-istio-token" pod "0359a851-6211-4192-9569-255a5b3dc518" (UID: "0359a851-6211-4192-9569-255a5b3dc518"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:14:59.044542 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:14:59.044502 5378 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m4wv2\" (UniqueName: \"kubernetes.io/projected/0359a851-6211-4192-9569-255a5b3dc518-kube-api-access-m4wv2\") on node \"ip-10-0-138-149.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:14:59.044542 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:14:59.044541 5378 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0359a851-6211-4192-9569-255a5b3dc518-istio-podinfo\") on node \"ip-10-0-138-149.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:14:59.044750 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:14:59.044556 5378 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0359a851-6211-4192-9569-255a5b3dc518-istio-envoy\") on node \"ip-10-0-138-149.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:14:59.044750 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:14:59.044588 5378 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0359a851-6211-4192-9569-255a5b3dc518-istio-token\") on node \"ip-10-0-138-149.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:14:59.114295 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:14:59.114260 5378 generic.go:356] "Generic (PLEG): container finished" podID="0359a851-6211-4192-9569-255a5b3dc518" containerID="e9fd800b4f60a2b6e715d2da7ae20fc8df38a0a3782046ca707f2856ac9d78f0" exitCode=0 Oct 30 21:14:59.114514 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:14:59.114309 5378 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-2b425d-openshift-default-f547dbcbf-2dmdq" Oct 30 21:14:59.114514 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:14:59.114362 5378 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-2b425d-openshift-default-f547dbcbf-2dmdq" event={"ID":"0359a851-6211-4192-9569-255a5b3dc518","Type":"ContainerDied","Data":"e9fd800b4f60a2b6e715d2da7ae20fc8df38a0a3782046ca707f2856ac9d78f0"} Oct 30 21:14:59.114514 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:14:59.114432 5378 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-2b425d-openshift-default-f547dbcbf-2dmdq" event={"ID":"0359a851-6211-4192-9569-255a5b3dc518","Type":"ContainerDied","Data":"148e8811b1cbdf509d183a5f30c129066a57ea490904c4ff483622d0111fddad"} Oct 30 21:14:59.114514 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:14:59.114463 5378 scope.go:122] "RemoveContainer" containerID="e9fd800b4f60a2b6e715d2da7ae20fc8df38a0a3782046ca707f2856ac9d78f0" Oct 30 21:14:59.134687 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:14:59.134667 5378 scope.go:122] "RemoveContainer" containerID="e9fd800b4f60a2b6e715d2da7ae20fc8df38a0a3782046ca707f2856ac9d78f0" Oct 30 21:14:59.135022 ip-10-0-138-149 kubenswrapper[5378]: E1030 21:14:59.135000 5378 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e9fd800b4f60a2b6e715d2da7ae20fc8df38a0a3782046ca707f2856ac9d78f0\": container with ID starting with e9fd800b4f60a2b6e715d2da7ae20fc8df38a0a3782046ca707f2856ac9d78f0 not found: ID does not exist" containerID="e9fd800b4f60a2b6e715d2da7ae20fc8df38a0a3782046ca707f2856ac9d78f0" Oct 30 21:14:59.135122 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:14:59.135031 5378 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e9fd800b4f60a2b6e715d2da7ae20fc8df38a0a3782046ca707f2856ac9d78f0"} err="failed to get container status \"e9fd800b4f60a2b6e715d2da7ae20fc8df38a0a3782046ca707f2856ac9d78f0\": rpc error: code = NotFound desc = could not find container \"e9fd800b4f60a2b6e715d2da7ae20fc8df38a0a3782046ca707f2856ac9d78f0\": container with ID starting with e9fd800b4f60a2b6e715d2da7ae20fc8df38a0a3782046ca707f2856ac9d78f0 not found: ID does not exist" Oct 30 21:14:59.286338 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:14:59.286249 5378 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-2b425d-openshift-default-f547dbcbf-2dmdq"] Oct 30 21:14:59.298943 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:14:59.298910 5378 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w5-2b425d-openshift-default-f547dbcbf-2dmdq"] Oct 30 21:14:59.746574 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:14:59.746542 5378 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0359a851-6211-4192-9569-255a5b3dc518" path="/var/lib/kubelet/pods/0359a851-6211-4192-9569-255a5b3dc518/volumes" Oct 30 21:15:00.188940 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:15:00.188903 5378 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-7eaf74-29889915-g4gn9"] Oct 30 21:15:00.189646 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:15:00.189624 5378 cpu_manager.go:414] "containerMap: removing container" podUID="2c139b9f-c7df-4675-b431-95d4136095e5" containerName="main" Oct 30 21:15:00.189646 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:15:00.189645 5378 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2c139b9f-c7df-4675-b431-95d4136095e5" containerName="main" Oct 30 21:15:00.189839 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:15:00.189675 5378 cpu_manager.go:414] "containerMap: removing container" podUID="525fac02-b21c-4dd5-b553-72bcc36062c5" containerName="main" Oct 30 21:15:00.189839 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:15:00.189683 5378 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="525fac02-b21c-4dd5-b553-72bcc36062c5" containerName="main" Oct 30 21:15:00.189839 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:15:00.189702 5378 cpu_manager.go:414] "containerMap: removing container" podUID="0359a851-6211-4192-9569-255a5b3dc518" containerName="istio-proxy" Oct 30 21:15:00.189839 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:15:00.189708 5378 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0359a851-6211-4192-9569-255a5b3dc518" containerName="istio-proxy" Oct 30 21:15:00.189839 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:15:00.189725 5378 cpu_manager.go:414] "containerMap: removing container" podUID="072e9c4b-6360-4cdd-8594-517158d18132" containerName="istio-proxy" Oct 30 21:15:00.189839 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:15:00.189732 5378 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="072e9c4b-6360-4cdd-8594-517158d18132" containerName="istio-proxy" Oct 30 21:15:00.190147 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:15:00.189908 5378 memory_manager.go:365] "RemoveStaleState removing state" podUID="2c139b9f-c7df-4675-b431-95d4136095e5" containerName="main" Oct 30 21:15:00.190147 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:15:00.189925 5378 memory_manager.go:365] "RemoveStaleState removing state" podUID="525fac02-b21c-4dd5-b553-72bcc36062c5" containerName="main" Oct 30 21:15:00.190147 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:15:00.189935 5378 memory_manager.go:365] "RemoveStaleState removing state" podUID="4955d910-8544-4a2b-a1fd-b414734aed56" containerName="istio-proxy" Oct 30 21:15:00.190147 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:15:00.189946 5378 memory_manager.go:365] "RemoveStaleState removing state" podUID="0359a851-6211-4192-9569-255a5b3dc518" containerName="istio-proxy" Oct 30 21:15:00.190147 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:15:00.189957 5378 memory_manager.go:365] "RemoveStaleState removing state" podUID="072e9c4b-6360-4cdd-8594-517158d18132" containerName="istio-proxy" Oct 30 21:15:00.190147 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:15:00.189972 5378 memory_manager.go:365] "RemoveStaleState removing state" podUID="1b385f61-b9d9-4f5d-8e71-7d86cd0d867d" containerName="istio-proxy" Oct 30 21:15:00.195089 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:15:00.195065 5378 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-7eaf74-29889915-g4gn9" Oct 30 21:15:00.246074 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:15:00.246039 5378 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-781058-29889915-k2s9b"] Oct 30 21:15:00.250516 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:15:00.250488 5378 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-2b425d-29889915-hpg7c"] Oct 30 21:15:00.250697 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:15:00.250647 5378 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-781058-29889915-k2s9b" Oct 30 21:15:00.254462 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:15:00.254442 5378 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-2b425d-29889915-hpg7c" Oct 30 21:15:00.260002 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:15:00.259973 5378 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rh7rm\" (UniqueName: \"kubernetes.io/projected/cbd70c63-4eab-49cc-9b10-f039320c1fb0-kube-api-access-rh7rm\") pod \"maas-api-key-cleanup-e2e-shared-b-w6-7eaf74-29889915-g4gn9\" (UID: \"cbd70c63-4eab-49cc-9b10-f039320c1fb0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-7eaf74-29889915-g4gn9" Oct 30 21:15:00.260116 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:15:00.260080 5378 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cbd70c63-4eab-49cc-9b10-f039320c1fb0-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w6-7eaf74-29889915-g4gn9\" (UID: \"cbd70c63-4eab-49cc-9b10-f039320c1fb0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-7eaf74-29889915-g4gn9" Oct 30 21:15:00.297145 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:15:00.297116 5378 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-d061d0-29889915-d4fwx"] Oct 30 21:15:00.302121 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:15:00.302098 5378 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-7eaf74-29889915-g4gn9"] Oct 30 21:15:00.302247 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:15:00.302194 5378 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-d061d0-29889915-d4fwx" Oct 30 21:15:00.312551 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:15:00.312519 5378 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-1fdaa7-29889915-vfvtm"] Oct 30 21:15:00.322601 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:15:00.317808 5378 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-1fdaa7-29889915-vfvtm" Oct 30 21:15:00.353911 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:15:00.353878 5378 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-b888d0-29889915-2dftf"] Oct 30 21:15:00.359412 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:15:00.359388 5378 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-781058-29889915-k2s9b"] Oct 30 21:15:00.359635 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:15:00.359512 5378 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-b888d0-29889915-2dftf" Oct 30 21:15:00.364902 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:15:00.362498 5378 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-q2mgc\"" Oct 30 21:15:00.369817 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:15:00.369791 5378 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zbmz8\" (UniqueName: \"kubernetes.io/projected/789b0c2c-6150-4867-9c6b-8e7fffa5e17f-kube-api-access-zbmz8\") pod \"maas-api-key-cleanup-e2e-worker-w0-781058-29889915-k2s9b\" (UID: \"789b0c2c-6150-4867-9c6b-8e7fffa5e17f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-781058-29889915-k2s9b" Oct 30 21:15:00.370047 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:15:00.370023 5378 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-clr2q\" (UniqueName: \"kubernetes.io/projected/acab1aa9-cd67-4e6e-afc5-a504a8064563-kube-api-access-clr2q\") pod \"maas-api-key-cleanup-e2e-worker-w4-1fdaa7-29889915-vfvtm\" (UID: \"acab1aa9-cd67-4e6e-afc5-a504a8064563\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-1fdaa7-29889915-vfvtm" Oct 30 21:15:00.370308 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:15:00.370287 5378 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/acab1aa9-cd67-4e6e-afc5-a504a8064563-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-1fdaa7-29889915-vfvtm\" (UID: \"acab1aa9-cd67-4e6e-afc5-a504a8064563\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-1fdaa7-29889915-vfvtm" Oct 30 21:15:00.370536 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:15:00.370519 5378 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/03043216-5312-45de-9d5c-63b2cc34b320-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-d061d0-29889915-d4fwx\" (UID: \"03043216-5312-45de-9d5c-63b2cc34b320\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-d061d0-29889915-d4fwx" Oct 30 21:15:00.374302 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:15:00.370726 5378 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/789b0c2c-6150-4867-9c6b-8e7fffa5e17f-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-781058-29889915-k2s9b\" (UID: \"789b0c2c-6150-4867-9c6b-8e7fffa5e17f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-781058-29889915-k2s9b" Oct 30 21:15:00.374659 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:15:00.374638 5378 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gh9kx\" (UniqueName: \"kubernetes.io/projected/03043216-5312-45de-9d5c-63b2cc34b320-kube-api-access-gh9kx\") pod \"maas-api-key-cleanup-e2e-worker-w2-d061d0-29889915-d4fwx\" (UID: \"03043216-5312-45de-9d5c-63b2cc34b320\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-d061d0-29889915-d4fwx" Oct 30 21:15:00.374833 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:15:00.374815 5378 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/503a9e15-f5df-4fde-8ca1-ea1946fe581b-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w5-2b425d-29889915-hpg7c\" (UID: \"503a9e15-f5df-4fde-8ca1-ea1946fe581b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-2b425d-29889915-hpg7c" Oct 30 21:15:00.375067 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:15:00.375047 5378 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ptg2r\" (UniqueName: \"kubernetes.io/projected/503a9e15-f5df-4fde-8ca1-ea1946fe581b-kube-api-access-ptg2r\") pod \"maas-api-key-cleanup-e2e-worker-w5-2b425d-29889915-hpg7c\" (UID: \"503a9e15-f5df-4fde-8ca1-ea1946fe581b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-2b425d-29889915-hpg7c" Oct 30 21:15:00.375239 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:15:00.375217 5378 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rh7rm\" (UniqueName: \"kubernetes.io/projected/cbd70c63-4eab-49cc-9b10-f039320c1fb0-kube-api-access-rh7rm\") pod \"maas-api-key-cleanup-e2e-shared-b-w6-7eaf74-29889915-g4gn9\" (UID: \"cbd70c63-4eab-49cc-9b10-f039320c1fb0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-7eaf74-29889915-g4gn9" Oct 30 21:15:00.375400 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:15:00.375380 5378 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cbd70c63-4eab-49cc-9b10-f039320c1fb0-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w6-7eaf74-29889915-g4gn9\" (UID: \"cbd70c63-4eab-49cc-9b10-f039320c1fb0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-7eaf74-29889915-g4gn9" Oct 30 21:15:00.376229 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:15:00.376187 5378 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cbd70c63-4eab-49cc-9b10-f039320c1fb0-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w6-7eaf74-29889915-g4gn9\" (UID: \"cbd70c63-4eab-49cc-9b10-f039320c1fb0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-7eaf74-29889915-g4gn9" Oct 30 21:15:00.376811 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:15:00.376779 5378 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-d061d0-29889915-d4fwx"] Oct 30 21:15:00.389052 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:15:00.389030 5378 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-2b425d-29889915-hpg7c"] Oct 30 21:15:00.395541 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:15:00.395159 5378 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-b888d0-29889915-2dftf"] Oct 30 21:15:00.425618 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:15:00.425551 5378 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-1fdaa7-29889915-vfvtm"] Oct 30 21:15:00.434074 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:15:00.434044 5378 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rh7rm\" (UniqueName: \"kubernetes.io/projected/cbd70c63-4eab-49cc-9b10-f039320c1fb0-kube-api-access-rh7rm\") pod \"maas-api-key-cleanup-e2e-shared-b-w6-7eaf74-29889915-g4gn9\" (UID: \"cbd70c63-4eab-49cc-9b10-f039320c1fb0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-7eaf74-29889915-g4gn9" Oct 30 21:15:00.477167 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:15:00.477083 5378 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zbmz8\" (UniqueName: \"kubernetes.io/projected/789b0c2c-6150-4867-9c6b-8e7fffa5e17f-kube-api-access-zbmz8\") pod \"maas-api-key-cleanup-e2e-worker-w0-781058-29889915-k2s9b\" (UID: \"789b0c2c-6150-4867-9c6b-8e7fffa5e17f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-781058-29889915-k2s9b" Oct 30 21:15:00.477167 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:15:00.477121 5378 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-clr2q\" (UniqueName: \"kubernetes.io/projected/acab1aa9-cd67-4e6e-afc5-a504a8064563-kube-api-access-clr2q\") pod \"maas-api-key-cleanup-e2e-worker-w4-1fdaa7-29889915-vfvtm\" (UID: \"acab1aa9-cd67-4e6e-afc5-a504a8064563\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-1fdaa7-29889915-vfvtm" Oct 30 21:15:00.477167 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:15:00.477158 5378 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tvjwh\" (UniqueName: \"kubernetes.io/projected/db796f69-9d96-4b13-930d-79276df5078d-kube-api-access-tvjwh\") pod \"maas-api-key-cleanup-e2e-api-a-b888d0-29889915-2dftf\" (UID: \"db796f69-9d96-4b13-930d-79276df5078d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-b888d0-29889915-2dftf" Oct 30 21:15:00.477407 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:15:00.477211 5378 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/db796f69-9d96-4b13-930d-79276df5078d-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-api-a-b888d0-29889915-2dftf\" (UID: \"db796f69-9d96-4b13-930d-79276df5078d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-b888d0-29889915-2dftf" Oct 30 21:15:00.477488 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:15:00.477458 5378 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/acab1aa9-cd67-4e6e-afc5-a504a8064563-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-1fdaa7-29889915-vfvtm\" (UID: \"acab1aa9-cd67-4e6e-afc5-a504a8064563\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-1fdaa7-29889915-vfvtm" Oct 30 21:15:00.477616 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:15:00.477596 5378 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/03043216-5312-45de-9d5c-63b2cc34b320-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-d061d0-29889915-d4fwx\" (UID: \"03043216-5312-45de-9d5c-63b2cc34b320\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-d061d0-29889915-d4fwx" Oct 30 21:15:00.477680 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:15:00.477644 5378 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/789b0c2c-6150-4867-9c6b-8e7fffa5e17f-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-781058-29889915-k2s9b\" (UID: \"789b0c2c-6150-4867-9c6b-8e7fffa5e17f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-781058-29889915-k2s9b" Oct 30 21:15:00.477724 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:15:00.477674 5378 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gh9kx\" (UniqueName: \"kubernetes.io/projected/03043216-5312-45de-9d5c-63b2cc34b320-kube-api-access-gh9kx\") pod \"maas-api-key-cleanup-e2e-worker-w2-d061d0-29889915-d4fwx\" (UID: \"03043216-5312-45de-9d5c-63b2cc34b320\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-d061d0-29889915-d4fwx" Oct 30 21:15:00.477768 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:15:00.477742 5378 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/503a9e15-f5df-4fde-8ca1-ea1946fe581b-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w5-2b425d-29889915-hpg7c\" (UID: \"503a9e15-f5df-4fde-8ca1-ea1946fe581b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-2b425d-29889915-hpg7c" Oct 30 21:15:00.477836 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:15:00.477820 5378 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ptg2r\" (UniqueName: \"kubernetes.io/projected/503a9e15-f5df-4fde-8ca1-ea1946fe581b-kube-api-access-ptg2r\") pod \"maas-api-key-cleanup-e2e-worker-w5-2b425d-29889915-hpg7c\" (UID: \"503a9e15-f5df-4fde-8ca1-ea1946fe581b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-2b425d-29889915-hpg7c" Oct 30 21:15:00.478259 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:15:00.478231 5378 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/acab1aa9-cd67-4e6e-afc5-a504a8064563-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-1fdaa7-29889915-vfvtm\" (UID: \"acab1aa9-cd67-4e6e-afc5-a504a8064563\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-1fdaa7-29889915-vfvtm" Oct 30 21:15:00.478383 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:15:00.478297 5378 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/03043216-5312-45de-9d5c-63b2cc34b320-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-d061d0-29889915-d4fwx\" (UID: \"03043216-5312-45de-9d5c-63b2cc34b320\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-d061d0-29889915-d4fwx" Oct 30 21:15:00.478383 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:15:00.478365 5378 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/789b0c2c-6150-4867-9c6b-8e7fffa5e17f-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-781058-29889915-k2s9b\" (UID: \"789b0c2c-6150-4867-9c6b-8e7fffa5e17f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-781058-29889915-k2s9b" Oct 30 21:15:00.478635 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:15:00.478611 5378 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/503a9e15-f5df-4fde-8ca1-ea1946fe581b-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w5-2b425d-29889915-hpg7c\" (UID: \"503a9e15-f5df-4fde-8ca1-ea1946fe581b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-2b425d-29889915-hpg7c" Oct 30 21:15:00.512304 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:15:00.512269 5378 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-7eaf74-29889915-g4gn9" Oct 30 21:15:00.536679 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:15:00.536648 5378 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zbmz8\" (UniqueName: \"kubernetes.io/projected/789b0c2c-6150-4867-9c6b-8e7fffa5e17f-kube-api-access-zbmz8\") pod \"maas-api-key-cleanup-e2e-worker-w0-781058-29889915-k2s9b\" (UID: \"789b0c2c-6150-4867-9c6b-8e7fffa5e17f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-781058-29889915-k2s9b" Oct 30 21:15:00.539016 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:15:00.538234 5378 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ptg2r\" (UniqueName: \"kubernetes.io/projected/503a9e15-f5df-4fde-8ca1-ea1946fe581b-kube-api-access-ptg2r\") pod \"maas-api-key-cleanup-e2e-worker-w5-2b425d-29889915-hpg7c\" (UID: \"503a9e15-f5df-4fde-8ca1-ea1946fe581b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-2b425d-29889915-hpg7c" Oct 30 21:15:00.539016 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:15:00.538234 5378 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-clr2q\" (UniqueName: \"kubernetes.io/projected/acab1aa9-cd67-4e6e-afc5-a504a8064563-kube-api-access-clr2q\") pod \"maas-api-key-cleanup-e2e-worker-w4-1fdaa7-29889915-vfvtm\" (UID: \"acab1aa9-cd67-4e6e-afc5-a504a8064563\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-1fdaa7-29889915-vfvtm" Oct 30 21:15:00.539016 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:15:00.538988 5378 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gh9kx\" (UniqueName: \"kubernetes.io/projected/03043216-5312-45de-9d5c-63b2cc34b320-kube-api-access-gh9kx\") pod \"maas-api-key-cleanup-e2e-worker-w2-d061d0-29889915-d4fwx\" (UID: \"03043216-5312-45de-9d5c-63b2cc34b320\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-d061d0-29889915-d4fwx" Oct 30 21:15:00.573126 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:15:00.571813 5378 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-781058-29889915-k2s9b" Oct 30 21:15:00.579486 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:15:00.579453 5378 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tvjwh\" (UniqueName: \"kubernetes.io/projected/db796f69-9d96-4b13-930d-79276df5078d-kube-api-access-tvjwh\") pod \"maas-api-key-cleanup-e2e-api-a-b888d0-29889915-2dftf\" (UID: \"db796f69-9d96-4b13-930d-79276df5078d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-b888d0-29889915-2dftf" Oct 30 21:15:00.579655 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:15:00.579506 5378 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/db796f69-9d96-4b13-930d-79276df5078d-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-api-a-b888d0-29889915-2dftf\" (UID: \"db796f69-9d96-4b13-930d-79276df5078d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-b888d0-29889915-2dftf" Oct 30 21:15:00.580454 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:15:00.580429 5378 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/db796f69-9d96-4b13-930d-79276df5078d-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-api-a-b888d0-29889915-2dftf\" (UID: \"db796f69-9d96-4b13-930d-79276df5078d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-b888d0-29889915-2dftf" Oct 30 21:15:00.590533 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:15:00.590511 5378 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-2b425d-29889915-hpg7c" Oct 30 21:15:00.623195 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:15:00.622658 5378 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-d061d0-29889915-d4fwx" Oct 30 21:15:00.633995 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:15:00.633963 5378 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-1fdaa7-29889915-vfvtm" Oct 30 21:15:00.644295 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:15:00.644219 5378 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tvjwh\" (UniqueName: \"kubernetes.io/projected/db796f69-9d96-4b13-930d-79276df5078d-kube-api-access-tvjwh\") pod \"maas-api-key-cleanup-e2e-api-a-b888d0-29889915-2dftf\" (UID: \"db796f69-9d96-4b13-930d-79276df5078d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-b888d0-29889915-2dftf" Oct 30 21:15:00.700622 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:15:00.700013 5378 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-b888d0-29889915-2dftf" Oct 30 21:15:00.948162 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:15:00.948125 5378 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-7eaf74-29889915-g4gn9"] Oct 30 21:15:00.953279 ip-10-0-138-149 kubenswrapper[5378]: W1030 21:15:00.953251 5378 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcbd70c63_4eab_49cc_9b10_f039320c1fb0.slice/crio-52bf73eac19f29e48f4b2d5def739526c1d3e3ad5376a5bd61e17d520119ad7e WatchSource:0}: Error finding container 52bf73eac19f29e48f4b2d5def739526c1d3e3ad5376a5bd61e17d520119ad7e: Status 404 returned error can't find the container with id 52bf73eac19f29e48f4b2d5def739526c1d3e3ad5376a5bd61e17d520119ad7e Oct 30 21:15:01.041032 ip-10-0-138-149 kubenswrapper[5378]: W1030 21:15:01.041007 5378 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod503a9e15_f5df_4fde_8ca1_ea1946fe581b.slice/crio-b85902fa94eebac2154132a1d6d5e4dc03992188a2ced57601b93618b538c5bc WatchSource:0}: Error finding container b85902fa94eebac2154132a1d6d5e4dc03992188a2ced57601b93618b538c5bc: Status 404 returned error can't find the container with id b85902fa94eebac2154132a1d6d5e4dc03992188a2ced57601b93618b538c5bc Oct 30 21:15:01.042767 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:15:01.042742 5378 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-2b425d-29889915-hpg7c"] Oct 30 21:15:01.048660 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:15:01.048638 5378 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-781058-29889915-k2s9b"] Oct 30 21:15:01.054938 ip-10-0-138-149 kubenswrapper[5378]: W1030 21:15:01.054910 5378 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod789b0c2c_6150_4867_9c6b_8e7fffa5e17f.slice/crio-ef7f232644193b170aec78d700ce912937f89b7e5c296b68d4c294a17d874183 WatchSource:0}: Error finding container ef7f232644193b170aec78d700ce912937f89b7e5c296b68d4c294a17d874183: Status 404 returned error can't find the container with id ef7f232644193b170aec78d700ce912937f89b7e5c296b68d4c294a17d874183 Oct 30 21:15:01.130965 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:15:01.130927 5378 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-781058-29889915-k2s9b" event={"ID":"789b0c2c-6150-4867-9c6b-8e7fffa5e17f","Type":"ContainerStarted","Data":"ef7f232644193b170aec78d700ce912937f89b7e5c296b68d4c294a17d874183"} Oct 30 21:15:01.132478 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:15:01.132450 5378 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-7eaf74-29889915-g4gn9" event={"ID":"cbd70c63-4eab-49cc-9b10-f039320c1fb0","Type":"ContainerStarted","Data":"52bf73eac19f29e48f4b2d5def739526c1d3e3ad5376a5bd61e17d520119ad7e"} Oct 30 21:15:01.133875 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:15:01.133846 5378 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-2b425d-29889915-hpg7c" event={"ID":"503a9e15-f5df-4fde-8ca1-ea1946fe581b","Type":"ContainerStarted","Data":"b85902fa94eebac2154132a1d6d5e4dc03992188a2ced57601b93618b538c5bc"} Oct 30 21:15:01.160683 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:15:01.160661 5378 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-1fdaa7-29889915-vfvtm"] Oct 30 21:15:01.173963 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:15:01.173945 5378 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-b888d0-29889915-2dftf"] Oct 30 21:15:01.261600 ip-10-0-138-149 kubenswrapper[5378]: W1030 21:15:01.261436 5378 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podacab1aa9_cd67_4e6e_afc5_a504a8064563.slice/crio-add2f61042a09daaf7c0f32efd84fe5de3857019c8bb0c304ff8d7f085ef3bb1 WatchSource:0}: Error finding container add2f61042a09daaf7c0f32efd84fe5de3857019c8bb0c304ff8d7f085ef3bb1: Status 404 returned error can't find the container with id add2f61042a09daaf7c0f32efd84fe5de3857019c8bb0c304ff8d7f085ef3bb1 Oct 30 21:15:01.262139 ip-10-0-138-149 kubenswrapper[5378]: W1030 21:15:01.261929 5378 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddb796f69_9d96_4b13_930d_79276df5078d.slice/crio-644861daed82c202fc066637db6a5e2e75d5134a9ea0348c487a815c2c7cd04a WatchSource:0}: Error finding container 644861daed82c202fc066637db6a5e2e75d5134a9ea0348c487a815c2c7cd04a: Status 404 returned error can't find the container with id 644861daed82c202fc066637db6a5e2e75d5134a9ea0348c487a815c2c7cd04a Oct 30 21:15:01.369375 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:15:01.369350 5378 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-d061d0-29889915-d4fwx"] Oct 30 21:15:01.457264 ip-10-0-138-149 kubenswrapper[5378]: W1030 21:15:01.457223 5378 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod03043216_5312_45de_9d5c_63b2cc34b320.slice/crio-5180363807b25f3ded03e0013780c330a742e6d1cd3a3b08b0bfe019fbfe9e07 WatchSource:0}: Error finding container 5180363807b25f3ded03e0013780c330a742e6d1cd3a3b08b0bfe019fbfe9e07: Status 404 returned error can't find the container with id 5180363807b25f3ded03e0013780c330a742e6d1cd3a3b08b0bfe019fbfe9e07 Oct 30 21:15:02.144171 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:15:02.144132 5378 generic.go:356] "Generic (PLEG): container finished" podID="789b0c2c-6150-4867-9c6b-8e7fffa5e17f" containerID="72109542f8afb8dfbb072e2a460650c718bb78896001e31c536cf3566368e4a1" exitCode=60 Oct 30 21:15:02.144616 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:15:02.144220 5378 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-781058-29889915-k2s9b" event={"ID":"789b0c2c-6150-4867-9c6b-8e7fffa5e17f","Type":"ContainerDied","Data":"72109542f8afb8dfbb072e2a460650c718bb78896001e31c536cf3566368e4a1"} Oct 30 21:15:02.144616 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:15:02.144514 5378 scope.go:122] "RemoveContainer" containerID="72109542f8afb8dfbb072e2a460650c718bb78896001e31c536cf3566368e4a1" Oct 30 21:15:02.146098 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:15:02.146067 5378 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-1fdaa7-29889915-vfvtm" event={"ID":"acab1aa9-cd67-4e6e-afc5-a504a8064563","Type":"ContainerStarted","Data":"741cea129a7b1c7d60615b502eb3e49c67208d1c7ff09cb550d633ac44435fd8"} Oct 30 21:15:02.146213 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:15:02.146133 5378 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-1fdaa7-29889915-vfvtm" event={"ID":"acab1aa9-cd67-4e6e-afc5-a504a8064563","Type":"ContainerStarted","Data":"add2f61042a09daaf7c0f32efd84fe5de3857019c8bb0c304ff8d7f085ef3bb1"} Oct 30 21:15:02.147923 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:15:02.147895 5378 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-d061d0-29889915-d4fwx" event={"ID":"03043216-5312-45de-9d5c-63b2cc34b320","Type":"ContainerStarted","Data":"dab379ec663f799678cf927408058b9c34397a96151aa645e4bbf95a06ed08cc"} Oct 30 21:15:02.148034 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:15:02.147950 5378 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-d061d0-29889915-d4fwx" event={"ID":"03043216-5312-45de-9d5c-63b2cc34b320","Type":"ContainerStarted","Data":"5180363807b25f3ded03e0013780c330a742e6d1cd3a3b08b0bfe019fbfe9e07"} Oct 30 21:15:02.149652 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:15:02.149620 5378 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-7eaf74-29889915-g4gn9" event={"ID":"cbd70c63-4eab-49cc-9b10-f039320c1fb0","Type":"ContainerStarted","Data":"6e74e046d332dcbf542e25430b89f7c056d81f765db1e44ff68ab9613e0654b3"} Oct 30 21:15:02.151416 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:15:02.151386 5378 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-b888d0-29889915-2dftf" event={"ID":"db796f69-9d96-4b13-930d-79276df5078d","Type":"ContainerStarted","Data":"0e8405babd92b8762b1210e428672d5ffba1f6ba1dc13198d2cde54893d74a00"} Oct 30 21:15:02.151517 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:15:02.151447 5378 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-b888d0-29889915-2dftf" event={"ID":"db796f69-9d96-4b13-930d-79276df5078d","Type":"ContainerStarted","Data":"644861daed82c202fc066637db6a5e2e75d5134a9ea0348c487a815c2c7cd04a"} Oct 30 21:15:02.153816 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:15:02.153118 5378 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-2b425d-29889915-hpg7c" event={"ID":"503a9e15-f5df-4fde-8ca1-ea1946fe581b","Type":"ContainerStarted","Data":"cd017f832e3a859e3271e89bfb1c335f993c192df9b58f3063b91769b9a80e81"} Oct 30 21:15:02.378348 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:15:02.378291 5378 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-7eaf74-29889915-g4gn9" podStartSLOduration=2.378274531 podStartE2EDuration="2.378274531s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:15:02.375483002 +0000 UTC m=+2192.803596921" watchObservedRunningTime="2026-10-30 21:15:02.378274531 +0000 UTC m=+2192.806388474" Oct 30 21:15:02.494274 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:15:02.494176 5378 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-b888d0-29889915-2dftf" podStartSLOduration=2.494160955 podStartE2EDuration="2.494160955s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:15:02.491416787 +0000 UTC m=+2192.919530709" watchObservedRunningTime="2026-10-30 21:15:02.494160955 +0000 UTC m=+2192.922274874" Oct 30 21:15:02.608502 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:15:02.608439 5378 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-d061d0-29889915-d4fwx" podStartSLOduration=2.608425155 podStartE2EDuration="2.608425155s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:15:02.606073455 +0000 UTC m=+2193.034187374" watchObservedRunningTime="2026-10-30 21:15:02.608425155 +0000 UTC m=+2193.036539073" Oct 30 21:15:02.726067 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:15:02.725998 5378 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-2b425d-29889915-hpg7c" podStartSLOduration=2.72598258 podStartE2EDuration="2.72598258s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:15:02.723617407 +0000 UTC m=+2193.151731323" watchObservedRunningTime="2026-10-30 21:15:02.72598258 +0000 UTC m=+2193.154096498" Oct 30 21:15:02.843243 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:15:02.843185 5378 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-1fdaa7-29889915-vfvtm" podStartSLOduration=2.843170336 podStartE2EDuration="2.843170336s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:15:02.83983537 +0000 UTC m=+2193.267949287" watchObservedRunningTime="2026-10-30 21:15:02.843170336 +0000 UTC m=+2193.271284260" Oct 30 21:15:02.861441 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:15:02.861359 5378 patch_prober.go:29] interesting pod/e2e-inf-a-390b71-openshift-default-df8b44c8c-5k4nj container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.146:15021/healthz/ready\": read tcp 10.133.0.2:37538->10.133.0.146:15021: read: connection reset by peer" start-of-body= Oct 30 21:15:02.861617 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:15:02.861485 5378 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-a-390b71-openshift-default-df8b44c8c-5k4nj" podUID="dd395160-55d3-4dfb-a2b1-6e68c62f32b5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.146:15021/healthz/ready\": read tcp 10.133.0.2:37538->10.133.0.146:15021: read: connection reset by peer" Oct 30 21:15:03.167403 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:15:03.167355 5378 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-781058-29889915-k2s9b" event={"ID":"789b0c2c-6150-4867-9c6b-8e7fffa5e17f","Type":"ContainerStarted","Data":"fec93ffdf1b1bad503151991eea25852463c70adc15a5ec5b750131dcd0e059e"} Oct 30 21:15:03.169037 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:15:03.169015 5378 generic.go:356] "Generic (PLEG): container finished" podID="acab1aa9-cd67-4e6e-afc5-a504a8064563" containerID="741cea129a7b1c7d60615b502eb3e49c67208d1c7ff09cb550d633ac44435fd8" exitCode=60 Oct 30 21:15:03.169177 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:15:03.169076 5378 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-1fdaa7-29889915-vfvtm" event={"ID":"acab1aa9-cd67-4e6e-afc5-a504a8064563","Type":"ContainerDied","Data":"741cea129a7b1c7d60615b502eb3e49c67208d1c7ff09cb550d633ac44435fd8"} Oct 30 21:15:03.169344 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:15:03.169328 5378 scope.go:122] "RemoveContainer" containerID="741cea129a7b1c7d60615b502eb3e49c67208d1c7ff09cb550d633ac44435fd8" Oct 30 21:15:03.170752 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:15:03.170732 5378 generic.go:356] "Generic (PLEG): container finished" podID="03043216-5312-45de-9d5c-63b2cc34b320" containerID="dab379ec663f799678cf927408058b9c34397a96151aa645e4bbf95a06ed08cc" exitCode=60 Oct 30 21:15:03.170866 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:15:03.170795 5378 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-d061d0-29889915-d4fwx" event={"ID":"03043216-5312-45de-9d5c-63b2cc34b320","Type":"ContainerDied","Data":"dab379ec663f799678cf927408058b9c34397a96151aa645e4bbf95a06ed08cc"} Oct 30 21:15:03.171075 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:15:03.171001 5378 scope.go:122] "RemoveContainer" containerID="dab379ec663f799678cf927408058b9c34397a96151aa645e4bbf95a06ed08cc" Oct 30 21:15:03.172650 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:15:03.172520 5378 generic.go:356] "Generic (PLEG): container finished" podID="cbd70c63-4eab-49cc-9b10-f039320c1fb0" containerID="6e74e046d332dcbf542e25430b89f7c056d81f765db1e44ff68ab9613e0654b3" exitCode=60 Oct 30 21:15:03.172650 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:15:03.172600 5378 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-7eaf74-29889915-g4gn9" event={"ID":"cbd70c63-4eab-49cc-9b10-f039320c1fb0","Type":"ContainerDied","Data":"6e74e046d332dcbf542e25430b89f7c056d81f765db1e44ff68ab9613e0654b3"} Oct 30 21:15:03.172941 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:15:03.172875 5378 scope.go:122] "RemoveContainer" containerID="6e74e046d332dcbf542e25430b89f7c056d81f765db1e44ff68ab9613e0654b3" Oct 30 21:15:03.174639 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:15:03.174617 5378 generic.go:356] "Generic (PLEG): container finished" podID="db796f69-9d96-4b13-930d-79276df5078d" containerID="0e8405babd92b8762b1210e428672d5ffba1f6ba1dc13198d2cde54893d74a00" exitCode=60 Oct 30 21:15:03.174726 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:15:03.174696 5378 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-b888d0-29889915-2dftf" event={"ID":"db796f69-9d96-4b13-930d-79276df5078d","Type":"ContainerDied","Data":"0e8405babd92b8762b1210e428672d5ffba1f6ba1dc13198d2cde54893d74a00"} Oct 30 21:15:03.174880 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:15:03.174864 5378 scope.go:122] "RemoveContainer" containerID="0e8405babd92b8762b1210e428672d5ffba1f6ba1dc13198d2cde54893d74a00" Oct 30 21:15:03.176314 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:15:03.176294 5378 generic.go:356] "Generic (PLEG): container finished" podID="503a9e15-f5df-4fde-8ca1-ea1946fe581b" containerID="cd017f832e3a859e3271e89bfb1c335f993c192df9b58f3063b91769b9a80e81" exitCode=60 Oct 30 21:15:03.176416 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:15:03.176367 5378 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-2b425d-29889915-hpg7c" event={"ID":"503a9e15-f5df-4fde-8ca1-ea1946fe581b","Type":"ContainerDied","Data":"cd017f832e3a859e3271e89bfb1c335f993c192df9b58f3063b91769b9a80e81"} Oct 30 21:15:03.176525 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:15:03.176510 5378 scope.go:122] "RemoveContainer" containerID="cd017f832e3a859e3271e89bfb1c335f993c192df9b58f3063b91769b9a80e81" Oct 30 21:15:03.178009 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:15:03.177991 5378 generic.go:356] "Generic (PLEG): container finished" podID="dd395160-55d3-4dfb-a2b1-6e68c62f32b5" containerID="28e80945bafbdf729da98e7b0864087040d0ba4695c109e85aebe64751bebafd" exitCode=0 Oct 30 21:15:03.178111 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:15:03.178054 5378 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-390b71-openshift-default-df8b44c8c-5k4nj" event={"ID":"dd395160-55d3-4dfb-a2b1-6e68c62f32b5","Type":"ContainerDied","Data":"28e80945bafbdf729da98e7b0864087040d0ba4695c109e85aebe64751bebafd"} Oct 30 21:15:03.178371 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:15:03.178357 5378 scope.go:122] "RemoveContainer" containerID="28e80945bafbdf729da98e7b0864087040d0ba4695c109e85aebe64751bebafd" Oct 30 21:15:03.180140 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:15:03.179850 5378 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 21:15:03.180140 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:15:03.179933 5378 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 21:15:03.180140 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:15:03.179985 5378 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 21:15:03.286855 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:15:03.286807 5378 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-781058-29889915-k2s9b" podStartSLOduration=3.286793612 podStartE2EDuration="3.286793612s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:15:03.284076018 +0000 UTC m=+2193.712189940" watchObservedRunningTime="2026-10-30 21:15:03.286793612 +0000 UTC m=+2193.714907530" Oct 30 21:15:03.415050 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:15:03.415013 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:15:04.190264 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:15:04.190221 5378 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-b888d0-29889915-2dftf" event={"ID":"db796f69-9d96-4b13-930d-79276df5078d","Type":"ContainerStarted","Data":"3f3c067c4e6bd908386e20cc72954c47626a32c428288ed1f23903bac2a63755"} Oct 30 21:15:04.192138 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:15:04.192107 5378 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-2b425d-29889915-hpg7c" event={"ID":"503a9e15-f5df-4fde-8ca1-ea1946fe581b","Type":"ContainerStarted","Data":"c32f209c216c64c3440b76764305d8eda3133260e3d04daff1c73a76ed1e429f"} Oct 30 21:15:04.194050 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:15:04.194023 5378 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-390b71-openshift-default-df8b44c8c-5k4nj" event={"ID":"dd395160-55d3-4dfb-a2b1-6e68c62f32b5","Type":"ContainerStarted","Data":"52716f3a2df2478b92010b966b4f1c27c7bc90c5b7fe7b0b16c961c15fc20bb3"} Oct 30 21:15:04.196122 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:15:04.196097 5378 generic.go:356] "Generic (PLEG): container finished" podID="789b0c2c-6150-4867-9c6b-8e7fffa5e17f" containerID="fec93ffdf1b1bad503151991eea25852463c70adc15a5ec5b750131dcd0e059e" exitCode=60 Oct 30 21:15:04.196205 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:15:04.196174 5378 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-781058-29889915-k2s9b" event={"ID":"789b0c2c-6150-4867-9c6b-8e7fffa5e17f","Type":"ContainerDied","Data":"fec93ffdf1b1bad503151991eea25852463c70adc15a5ec5b750131dcd0e059e"} Oct 30 21:15:04.196249 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:15:04.196227 5378 scope.go:122] "RemoveContainer" containerID="72109542f8afb8dfbb072e2a460650c718bb78896001e31c536cf3566368e4a1" Oct 30 21:15:04.196407 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:15:04.196385 5378 scope.go:122] "RemoveContainer" containerID="fec93ffdf1b1bad503151991eea25852463c70adc15a5ec5b750131dcd0e059e" Oct 30 21:15:04.196723 ip-10-0-138-149 kubenswrapper[5378]: E1030 21:15:04.196695 5378 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-781058-29889915-k2s9b_odh-ai-gateway-infra(789b0c2c-6150-4867-9c6b-8e7fffa5e17f)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-781058-29889915-k2s9b" podUID="789b0c2c-6150-4867-9c6b-8e7fffa5e17f" Oct 30 21:15:04.198063 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:15:04.198044 5378 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-1fdaa7-29889915-vfvtm" event={"ID":"acab1aa9-cd67-4e6e-afc5-a504a8064563","Type":"ContainerStarted","Data":"8c971bfd88838a1759a8ff13e047171c847467f44e616d353e73f89c7411a572"} Oct 30 21:15:04.199953 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:15:04.199934 5378 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-d061d0-29889915-d4fwx" event={"ID":"03043216-5312-45de-9d5c-63b2cc34b320","Type":"ContainerStarted","Data":"32b44f8b85a43a1be130efdf08aa35d2faf6143aed716d9fa794bb212553c408"} Oct 30 21:15:04.201741 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:15:04.201720 5378 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-7eaf74-29889915-g4gn9" event={"ID":"cbd70c63-4eab-49cc-9b10-f039320c1fb0","Type":"ContainerStarted","Data":"7fa1f615dd255664a6c767ca13047fa6a3eb57cb0d80fed56b915a51d79e8597"} Oct 30 21:15:04.290390 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:15:04.290362 5378 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-390b71-openshift-default-df8b44c8c-5k4nj" Oct 30 21:15:04.290611 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:15:04.290583 5378 patch_prober.go:29] interesting pod/e2e-inf-a-390b71-openshift-default-df8b44c8c-5k4nj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.146:15021/healthz/ready\": dial tcp 10.133.0.146:15021: connect: connection refused" start-of-body= Oct 30 21:15:04.290742 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:15:04.290640 5378 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-390b71-openshift-default-df8b44c8c-5k4nj" podUID="dd395160-55d3-4dfb-a2b1-6e68c62f32b5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.146:15021/healthz/ready\": dial tcp 10.133.0.146:15021: connect: connection refused" Oct 30 21:15:05.210156 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:15:05.210117 5378 generic.go:356] "Generic (PLEG): container finished" podID="db796f69-9d96-4b13-930d-79276df5078d" containerID="3f3c067c4e6bd908386e20cc72954c47626a32c428288ed1f23903bac2a63755" exitCode=60 Oct 30 21:15:05.210662 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:15:05.210195 5378 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-b888d0-29889915-2dftf" event={"ID":"db796f69-9d96-4b13-930d-79276df5078d","Type":"ContainerDied","Data":"3f3c067c4e6bd908386e20cc72954c47626a32c428288ed1f23903bac2a63755"} Oct 30 21:15:05.210662 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:15:05.210256 5378 scope.go:122] "RemoveContainer" containerID="0e8405babd92b8762b1210e428672d5ffba1f6ba1dc13198d2cde54893d74a00" Oct 30 21:15:05.210662 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:15:05.210649 5378 scope.go:122] "RemoveContainer" containerID="3f3c067c4e6bd908386e20cc72954c47626a32c428288ed1f23903bac2a63755" Oct 30 21:15:05.211077 ip-10-0-138-149 kubenswrapper[5378]: E1030 21:15:05.211048 5378 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-api-a-b888d0-29889915-2dftf_odh-ai-gateway-infra(db796f69-9d96-4b13-930d-79276df5078d)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-b888d0-29889915-2dftf" podUID="db796f69-9d96-4b13-930d-79276df5078d" Oct 30 21:15:05.212420 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:15:05.212396 5378 generic.go:356] "Generic (PLEG): container finished" podID="503a9e15-f5df-4fde-8ca1-ea1946fe581b" containerID="c32f209c216c64c3440b76764305d8eda3133260e3d04daff1c73a76ed1e429f" exitCode=60 Oct 30 21:15:05.212505 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:15:05.212466 5378 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-2b425d-29889915-hpg7c" event={"ID":"503a9e15-f5df-4fde-8ca1-ea1946fe581b","Type":"ContainerDied","Data":"c32f209c216c64c3440b76764305d8eda3133260e3d04daff1c73a76ed1e429f"} Oct 30 21:15:05.212781 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:15:05.212763 5378 scope.go:122] "RemoveContainer" containerID="c32f209c216c64c3440b76764305d8eda3133260e3d04daff1c73a76ed1e429f" Oct 30 21:15:05.213161 ip-10-0-138-149 kubenswrapper[5378]: E1030 21:15:05.213139 5378 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w5-2b425d-29889915-hpg7c_odh-ai-gateway-infra(503a9e15-f5df-4fde-8ca1-ea1946fe581b)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-2b425d-29889915-hpg7c" podUID="503a9e15-f5df-4fde-8ca1-ea1946fe581b" Oct 30 21:15:05.214745 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:15:05.214721 5378 scope.go:122] "RemoveContainer" containerID="fec93ffdf1b1bad503151991eea25852463c70adc15a5ec5b750131dcd0e059e" Oct 30 21:15:05.215074 ip-10-0-138-149 kubenswrapper[5378]: E1030 21:15:05.215052 5378 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-781058-29889915-k2s9b_odh-ai-gateway-infra(789b0c2c-6150-4867-9c6b-8e7fffa5e17f)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-781058-29889915-k2s9b" podUID="789b0c2c-6150-4867-9c6b-8e7fffa5e17f" Oct 30 21:15:05.216295 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:15:05.216274 5378 generic.go:356] "Generic (PLEG): container finished" podID="acab1aa9-cd67-4e6e-afc5-a504a8064563" containerID="8c971bfd88838a1759a8ff13e047171c847467f44e616d353e73f89c7411a572" exitCode=60 Oct 30 21:15:05.216375 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:15:05.216336 5378 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-1fdaa7-29889915-vfvtm" event={"ID":"acab1aa9-cd67-4e6e-afc5-a504a8064563","Type":"ContainerDied","Data":"8c971bfd88838a1759a8ff13e047171c847467f44e616d353e73f89c7411a572"} Oct 30 21:15:05.216658 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:15:05.216637 5378 scope.go:122] "RemoveContainer" containerID="8c971bfd88838a1759a8ff13e047171c847467f44e616d353e73f89c7411a572" Oct 30 21:15:05.217021 ip-10-0-138-149 kubenswrapper[5378]: E1030 21:15:05.216999 5378 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-1fdaa7-29889915-vfvtm_odh-ai-gateway-infra(acab1aa9-cd67-4e6e-afc5-a504a8064563)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-1fdaa7-29889915-vfvtm" podUID="acab1aa9-cd67-4e6e-afc5-a504a8064563" Oct 30 21:15:05.218418 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:15:05.218397 5378 generic.go:356] "Generic (PLEG): container finished" podID="03043216-5312-45de-9d5c-63b2cc34b320" containerID="32b44f8b85a43a1be130efdf08aa35d2faf6143aed716d9fa794bb212553c408" exitCode=60 Oct 30 21:15:05.218495 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:15:05.218472 5378 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-d061d0-29889915-d4fwx" event={"ID":"03043216-5312-45de-9d5c-63b2cc34b320","Type":"ContainerDied","Data":"32b44f8b85a43a1be130efdf08aa35d2faf6143aed716d9fa794bb212553c408"} Oct 30 21:15:05.218722 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:15:05.218706 5378 scope.go:122] "RemoveContainer" containerID="32b44f8b85a43a1be130efdf08aa35d2faf6143aed716d9fa794bb212553c408" Oct 30 21:15:05.219079 ip-10-0-138-149 kubenswrapper[5378]: E1030 21:15:05.219045 5378 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-d061d0-29889915-d4fwx_odh-ai-gateway-infra(03043216-5312-45de-9d5c-63b2cc34b320)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-d061d0-29889915-d4fwx" podUID="03043216-5312-45de-9d5c-63b2cc34b320" Oct 30 21:15:05.220270 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:15:05.220252 5378 generic.go:356] "Generic (PLEG): container finished" podID="cbd70c63-4eab-49cc-9b10-f039320c1fb0" containerID="7fa1f615dd255664a6c767ca13047fa6a3eb57cb0d80fed56b915a51d79e8597" exitCode=60 Oct 30 21:15:05.220335 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:15:05.220316 5378 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-7eaf74-29889915-g4gn9" event={"ID":"cbd70c63-4eab-49cc-9b10-f039320c1fb0","Type":"ContainerDied","Data":"7fa1f615dd255664a6c767ca13047fa6a3eb57cb0d80fed56b915a51d79e8597"} Oct 30 21:15:05.220508 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:15:05.220495 5378 scope.go:122] "RemoveContainer" containerID="7fa1f615dd255664a6c767ca13047fa6a3eb57cb0d80fed56b915a51d79e8597" Oct 30 21:15:05.220772 ip-10-0-138-149 kubenswrapper[5378]: E1030 21:15:05.220755 5378 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w6-7eaf74-29889915-g4gn9_odh-ai-gateway-infra(cbd70c63-4eab-49cc-9b10-f039320c1fb0)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-7eaf74-29889915-g4gn9" podUID="cbd70c63-4eab-49cc-9b10-f039320c1fb0" Oct 30 21:15:05.223362 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:15:05.223344 5378 scope.go:122] "RemoveContainer" containerID="cd017f832e3a859e3271e89bfb1c335f993c192df9b58f3063b91769b9a80e81" Oct 30 21:15:05.235256 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:15:05.235236 5378 scope.go:122] "RemoveContainer" containerID="741cea129a7b1c7d60615b502eb3e49c67208d1c7ff09cb550d633ac44435fd8" Oct 30 21:15:05.246762 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:15:05.246737 5378 scope.go:122] "RemoveContainer" containerID="dab379ec663f799678cf927408058b9c34397a96151aa645e4bbf95a06ed08cc" Oct 30 21:15:05.283859 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:15:05.283833 5378 scope.go:122] "RemoveContainer" containerID="6e74e046d332dcbf542e25430b89f7c056d81f765db1e44ff68ab9613e0654b3" Oct 30 21:15:05.290181 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:15:05.290148 5378 patch_prober.go:29] interesting pod/e2e-inf-a-390b71-openshift-default-df8b44c8c-5k4nj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.146:15021/healthz/ready\": dial tcp 10.133.0.146:15021: connect: connection refused" start-of-body= Oct 30 21:15:05.290322 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:15:05.290208 5378 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-390b71-openshift-default-df8b44c8c-5k4nj" podUID="dd395160-55d3-4dfb-a2b1-6e68c62f32b5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.146:15021/healthz/ready\": dial tcp 10.133.0.146:15021: connect: connection refused" Oct 30 21:15:06.229743 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:15:06.229710 5378 scope.go:122] "RemoveContainer" containerID="8c971bfd88838a1759a8ff13e047171c847467f44e616d353e73f89c7411a572" Oct 30 21:15:06.230215 ip-10-0-138-149 kubenswrapper[5378]: E1030 21:15:06.230121 5378 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-1fdaa7-29889915-vfvtm_odh-ai-gateway-infra(acab1aa9-cd67-4e6e-afc5-a504a8064563)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-1fdaa7-29889915-vfvtm" podUID="acab1aa9-cd67-4e6e-afc5-a504a8064563" Oct 30 21:15:06.231395 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:15:06.231369 5378 scope.go:122] "RemoveContainer" containerID="32b44f8b85a43a1be130efdf08aa35d2faf6143aed716d9fa794bb212553c408" Oct 30 21:15:06.231720 ip-10-0-138-149 kubenswrapper[5378]: E1030 21:15:06.231694 5378 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-d061d0-29889915-d4fwx_odh-ai-gateway-infra(03043216-5312-45de-9d5c-63b2cc34b320)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-d061d0-29889915-d4fwx" podUID="03043216-5312-45de-9d5c-63b2cc34b320" Oct 30 21:15:06.233173 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:15:06.233150 5378 scope.go:122] "RemoveContainer" containerID="7fa1f615dd255664a6c767ca13047fa6a3eb57cb0d80fed56b915a51d79e8597" Oct 30 21:15:06.233460 ip-10-0-138-149 kubenswrapper[5378]: E1030 21:15:06.233439 5378 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w6-7eaf74-29889915-g4gn9_odh-ai-gateway-infra(cbd70c63-4eab-49cc-9b10-f039320c1fb0)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-7eaf74-29889915-g4gn9" podUID="cbd70c63-4eab-49cc-9b10-f039320c1fb0" Oct 30 21:15:06.234745 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:15:06.234722 5378 scope.go:122] "RemoveContainer" containerID="3f3c067c4e6bd908386e20cc72954c47626a32c428288ed1f23903bac2a63755" Oct 30 21:15:06.235005 ip-10-0-138-149 kubenswrapper[5378]: E1030 21:15:06.234987 5378 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-api-a-b888d0-29889915-2dftf_odh-ai-gateway-infra(db796f69-9d96-4b13-930d-79276df5078d)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-b888d0-29889915-2dftf" podUID="db796f69-9d96-4b13-930d-79276df5078d" Oct 30 21:15:06.236235 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:15:06.236214 5378 scope.go:122] "RemoveContainer" containerID="c32f209c216c64c3440b76764305d8eda3133260e3d04daff1c73a76ed1e429f" Oct 30 21:15:06.236483 ip-10-0-138-149 kubenswrapper[5378]: E1030 21:15:06.236461 5378 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w5-2b425d-29889915-hpg7c_odh-ai-gateway-infra(503a9e15-f5df-4fde-8ca1-ea1946fe581b)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-2b425d-29889915-hpg7c" podUID="503a9e15-f5df-4fde-8ca1-ea1946fe581b" Oct 30 21:15:06.290368 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:15:06.290328 5378 patch_prober.go:29] interesting pod/e2e-inf-a-390b71-openshift-default-df8b44c8c-5k4nj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.146:15021/healthz/ready\": dial tcp 10.133.0.146:15021: connect: connection refused" start-of-body= Oct 30 21:15:06.290523 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:15:06.290385 5378 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-390b71-openshift-default-df8b44c8c-5k4nj" podUID="dd395160-55d3-4dfb-a2b1-6e68c62f32b5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.146:15021/healthz/ready\": dial tcp 10.133.0.146:15021: connect: connection refused" Oct 30 21:15:07.290710 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:15:07.290675 5378 patch_prober.go:29] interesting pod/e2e-inf-a-390b71-openshift-default-df8b44c8c-5k4nj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.146:15021/healthz/ready\": dial tcp 10.133.0.146:15021: connect: connection refused" start-of-body= Oct 30 21:15:07.291115 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:15:07.290757 5378 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-390b71-openshift-default-df8b44c8c-5k4nj" podUID="dd395160-55d3-4dfb-a2b1-6e68c62f32b5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.146:15021/healthz/ready\": dial tcp 10.133.0.146:15021: connect: connection refused" Oct 30 21:15:08.290295 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:15:08.290260 5378 patch_prober.go:29] interesting pod/e2e-inf-a-390b71-openshift-default-df8b44c8c-5k4nj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.146:15021/healthz/ready\": dial tcp 10.133.0.146:15021: connect: connection refused" start-of-body= Oct 30 21:15:08.290492 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:15:08.290312 5378 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-390b71-openshift-default-df8b44c8c-5k4nj" podUID="dd395160-55d3-4dfb-a2b1-6e68c62f32b5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.146:15021/healthz/ready\": dial tcp 10.133.0.146:15021: connect: connection refused" Oct 30 21:15:08.523814 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:15:08.523774 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:15:08.712521 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:15:08.712486 5378 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-1fdaa7-openshift-default-84586db7d5-zklmb"] Oct 30 21:15:08.712798 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:15:08.712766 5378 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-1fdaa7-openshift-default-84586db7d5-zklmb" podUID="f0685b8b-5dbc-4eb1-afcb-a390e5b464e7" containerName="istio-proxy" containerID="cri-o://71d242c0429a32ec01180957598143d6f2c6d3ab9c76ac0cf5f23a14dc188bcc" gracePeriod=30 Oct 30 21:15:09.290281 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:15:09.290246 5378 patch_prober.go:29] interesting pod/e2e-inf-a-390b71-openshift-default-df8b44c8c-5k4nj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.146:15021/healthz/ready\": dial tcp 10.133.0.146:15021: connect: connection refused" start-of-body= Oct 30 21:15:09.290477 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:15:09.290309 5378 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-390b71-openshift-default-df8b44c8c-5k4nj" podUID="dd395160-55d3-4dfb-a2b1-6e68c62f32b5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.146:15021/healthz/ready\": dial tcp 10.133.0.146:15021: connect: connection refused" Oct 30 21:15:10.295114 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:15:10.295077 5378 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-390b71-openshift-default-df8b44c8c-5k4nj" Oct 30 21:15:10.295660 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:15:10.295460 5378 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-390b71-openshift-default-df8b44c8c-5k4nj" Oct 30 21:15:10.296192 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:15:10.296173 5378 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-390b71-openshift-default-df8b44c8c-5k4nj" Oct 30 21:15:10.606069 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:15:10.605221 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:15:10.756553 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:15:10.756518 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:15:12.023935 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:15:12.023900 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:15:12.510832 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:15:12.510797 5378 patch_prober.go:29] interesting pod/e2e-worker-w4-1fdaa7-openshift-default-84586db7d5-zklmb container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 21:15:12.511006 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:15:12.510856 5378 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-1fdaa7-openshift-default-84586db7d5-zklmb" podUID="f0685b8b-5dbc-4eb1-afcb-a390e5b464e7" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 21:15:13.592923 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:15:13.592888 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:15:14.006205 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:15:14.006174 5378 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-1fdaa7-openshift-default-84586db7d5-zklmb" Oct 30 21:15:14.128183 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:15:14.128148 5378 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f0685b8b-5dbc-4eb1-afcb-a390e5b464e7-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f0685b8b-5dbc-4eb1-afcb-a390e5b464e7-workload-socket\") pod \"f0685b8b-5dbc-4eb1-afcb-a390e5b464e7\" (UID: \"f0685b8b-5dbc-4eb1-afcb-a390e5b464e7\") " Oct 30 21:15:14.128387 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:15:14.128206 5378 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f0685b8b-5dbc-4eb1-afcb-a390e5b464e7-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f0685b8b-5dbc-4eb1-afcb-a390e5b464e7-istio-envoy\") pod \"f0685b8b-5dbc-4eb1-afcb-a390e5b464e7\" (UID: \"f0685b8b-5dbc-4eb1-afcb-a390e5b464e7\") " Oct 30 21:15:14.128387 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:15:14.128233 5378 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f0685b8b-5dbc-4eb1-afcb-a390e5b464e7-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f0685b8b-5dbc-4eb1-afcb-a390e5b464e7-istiod-ca-cert\") pod \"f0685b8b-5dbc-4eb1-afcb-a390e5b464e7\" (UID: \"f0685b8b-5dbc-4eb1-afcb-a390e5b464e7\") " Oct 30 21:15:14.128512 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:15:14.128418 5378 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f0685b8b-5dbc-4eb1-afcb-a390e5b464e7-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f0685b8b-5dbc-4eb1-afcb-a390e5b464e7-istio-data\") pod \"f0685b8b-5dbc-4eb1-afcb-a390e5b464e7\" (UID: \"f0685b8b-5dbc-4eb1-afcb-a390e5b464e7\") " Oct 30 21:15:14.128512 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:15:14.128454 5378 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f0685b8b-5dbc-4eb1-afcb-a390e5b464e7-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f0685b8b-5dbc-4eb1-afcb-a390e5b464e7-workload-certs\") pod \"f0685b8b-5dbc-4eb1-afcb-a390e5b464e7\" (UID: \"f0685b8b-5dbc-4eb1-afcb-a390e5b464e7\") " Oct 30 21:15:14.128512 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:15:14.128496 5378 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f0685b8b-5dbc-4eb1-afcb-a390e5b464e7-workload-socket" pod "f0685b8b-5dbc-4eb1-afcb-a390e5b464e7" (UID: "f0685b8b-5dbc-4eb1-afcb-a390e5b464e7"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:15:14.128690 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:15:14.128518 5378 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f0685b8b-5dbc-4eb1-afcb-a390e5b464e7-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f0685b8b-5dbc-4eb1-afcb-a390e5b464e7-credential-socket\") pod \"f0685b8b-5dbc-4eb1-afcb-a390e5b464e7\" (UID: \"f0685b8b-5dbc-4eb1-afcb-a390e5b464e7\") " Oct 30 21:15:14.128690 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:15:14.128667 5378 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/f0685b8b-5dbc-4eb1-afcb-a390e5b464e7-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f0685b8b-5dbc-4eb1-afcb-a390e5b464e7-istio-podinfo\") pod \"f0685b8b-5dbc-4eb1-afcb-a390e5b464e7\" (UID: \"f0685b8b-5dbc-4eb1-afcb-a390e5b464e7\") " Oct 30 21:15:14.128802 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:15:14.128708 5378 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f0685b8b-5dbc-4eb1-afcb-a390e5b464e7-istio-data" pod "f0685b8b-5dbc-4eb1-afcb-a390e5b464e7" (UID: "f0685b8b-5dbc-4eb1-afcb-a390e5b464e7"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:15:14.128802 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:15:14.128738 5378 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f0685b8b-5dbc-4eb1-afcb-a390e5b464e7-istio-token\" (UniqueName: \"kubernetes.io/projected/f0685b8b-5dbc-4eb1-afcb-a390e5b464e7-istio-token\") pod \"f0685b8b-5dbc-4eb1-afcb-a390e5b464e7\" (UID: \"f0685b8b-5dbc-4eb1-afcb-a390e5b464e7\") " Oct 30 21:15:14.128802 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:15:14.128771 5378 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f0685b8b-5dbc-4eb1-afcb-a390e5b464e7-istiod-ca-cert" pod "f0685b8b-5dbc-4eb1-afcb-a390e5b464e7" (UID: "f0685b8b-5dbc-4eb1-afcb-a390e5b464e7"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:15:14.128802 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:15:14.128778 5378 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f0685b8b-5dbc-4eb1-afcb-a390e5b464e7-workload-certs" pod "f0685b8b-5dbc-4eb1-afcb-a390e5b464e7" (UID: "f0685b8b-5dbc-4eb1-afcb-a390e5b464e7"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:15:14.128802 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:15:14.128789 5378 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f0685b8b-5dbc-4eb1-afcb-a390e5b464e7-credential-socket" pod "f0685b8b-5dbc-4eb1-afcb-a390e5b464e7" (UID: "f0685b8b-5dbc-4eb1-afcb-a390e5b464e7"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:15:14.128802 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:15:14.128800 5378 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f0685b8b-5dbc-4eb1-afcb-a390e5b464e7-kube-api-access-9jmjb\" (UniqueName: \"kubernetes.io/projected/f0685b8b-5dbc-4eb1-afcb-a390e5b464e7-kube-api-access-9jmjb\") pod \"f0685b8b-5dbc-4eb1-afcb-a390e5b464e7\" (UID: \"f0685b8b-5dbc-4eb1-afcb-a390e5b464e7\") " Oct 30 21:15:14.129501 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:15:14.129479 5378 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f0685b8b-5dbc-4eb1-afcb-a390e5b464e7-istio-data\") on node \"ip-10-0-138-149.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:15:14.129501 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:15:14.129500 5378 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f0685b8b-5dbc-4eb1-afcb-a390e5b464e7-workload-certs\") on node \"ip-10-0-138-149.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:15:14.129681 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:15:14.129514 5378 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f0685b8b-5dbc-4eb1-afcb-a390e5b464e7-credential-socket\") on node \"ip-10-0-138-149.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:15:14.129681 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:15:14.129527 5378 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f0685b8b-5dbc-4eb1-afcb-a390e5b464e7-workload-socket\") on node \"ip-10-0-138-149.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:15:14.129681 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:15:14.129539 5378 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f0685b8b-5dbc-4eb1-afcb-a390e5b464e7-istiod-ca-cert\") on node \"ip-10-0-138-149.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:15:14.130851 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:15:14.130823 5378 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f0685b8b-5dbc-4eb1-afcb-a390e5b464e7-istio-podinfo" pod "f0685b8b-5dbc-4eb1-afcb-a390e5b464e7" (UID: "f0685b8b-5dbc-4eb1-afcb-a390e5b464e7"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 21:15:14.130947 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:15:14.130872 5378 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f0685b8b-5dbc-4eb1-afcb-a390e5b464e7-kube-api-access-9jmjb" pod "f0685b8b-5dbc-4eb1-afcb-a390e5b464e7" (UID: "f0685b8b-5dbc-4eb1-afcb-a390e5b464e7"). InnerVolumeSpecName "kube-api-access-9jmjb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:15:14.131104 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:15:14.131087 5378 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f0685b8b-5dbc-4eb1-afcb-a390e5b464e7-istio-token" pod "f0685b8b-5dbc-4eb1-afcb-a390e5b464e7" (UID: "f0685b8b-5dbc-4eb1-afcb-a390e5b464e7"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:15:14.131156 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:15:14.131089 5378 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f0685b8b-5dbc-4eb1-afcb-a390e5b464e7-istio-envoy" pod "f0685b8b-5dbc-4eb1-afcb-a390e5b464e7" (UID: "f0685b8b-5dbc-4eb1-afcb-a390e5b464e7"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:15:14.231086 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:15:14.231002 5378 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f0685b8b-5dbc-4eb1-afcb-a390e5b464e7-istio-podinfo\") on node \"ip-10-0-138-149.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:15:14.231086 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:15:14.231046 5378 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f0685b8b-5dbc-4eb1-afcb-a390e5b464e7-istio-token\") on node \"ip-10-0-138-149.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:15:14.231086 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:15:14.231062 5378 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9jmjb\" (UniqueName: \"kubernetes.io/projected/f0685b8b-5dbc-4eb1-afcb-a390e5b464e7-kube-api-access-9jmjb\") on node \"ip-10-0-138-149.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:15:14.231086 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:15:14.231076 5378 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f0685b8b-5dbc-4eb1-afcb-a390e5b464e7-istio-envoy\") on node \"ip-10-0-138-149.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:15:14.328031 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:15:14.327995 5378 generic.go:356] "Generic (PLEG): container finished" podID="f0685b8b-5dbc-4eb1-afcb-a390e5b464e7" containerID="71d242c0429a32ec01180957598143d6f2c6d3ab9c76ac0cf5f23a14dc188bcc" exitCode=0 Oct 30 21:15:14.328303 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:15:14.328065 5378 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-1fdaa7-openshift-default-84586db7d5-zklmb" Oct 30 21:15:14.328460 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:15:14.328080 5378 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-1fdaa7-openshift-default-84586db7d5-zklmb" event={"ID":"f0685b8b-5dbc-4eb1-afcb-a390e5b464e7","Type":"ContainerDied","Data":"71d242c0429a32ec01180957598143d6f2c6d3ab9c76ac0cf5f23a14dc188bcc"} Oct 30 21:15:14.328460 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:15:14.328415 5378 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-1fdaa7-openshift-default-84586db7d5-zklmb" event={"ID":"f0685b8b-5dbc-4eb1-afcb-a390e5b464e7","Type":"ContainerDied","Data":"b5cde2dbb7d40b0676bb4a45ffa7742033470cdc6ede2057aafaa7584121480f"} Oct 30 21:15:14.328625 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:15:14.328461 5378 scope.go:122] "RemoveContainer" containerID="71d242c0429a32ec01180957598143d6f2c6d3ab9c76ac0cf5f23a14dc188bcc" Oct 30 21:15:14.344061 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:15:14.344036 5378 scope.go:122] "RemoveContainer" containerID="71d242c0429a32ec01180957598143d6f2c6d3ab9c76ac0cf5f23a14dc188bcc" Oct 30 21:15:14.344455 ip-10-0-138-149 kubenswrapper[5378]: E1030 21:15:14.344430 5378 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"71d242c0429a32ec01180957598143d6f2c6d3ab9c76ac0cf5f23a14dc188bcc\": container with ID starting with 71d242c0429a32ec01180957598143d6f2c6d3ab9c76ac0cf5f23a14dc188bcc not found: ID does not exist" containerID="71d242c0429a32ec01180957598143d6f2c6d3ab9c76ac0cf5f23a14dc188bcc" Oct 30 21:15:14.344613 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:15:14.344461 5378 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"71d242c0429a32ec01180957598143d6f2c6d3ab9c76ac0cf5f23a14dc188bcc"} err="failed to get container status \"71d242c0429a32ec01180957598143d6f2c6d3ab9c76ac0cf5f23a14dc188bcc\": rpc error: code = NotFound desc = could not find container \"71d242c0429a32ec01180957598143d6f2c6d3ab9c76ac0cf5f23a14dc188bcc\": container with ID starting with 71d242c0429a32ec01180957598143d6f2c6d3ab9c76ac0cf5f23a14dc188bcc not found: ID does not exist" Oct 30 21:15:14.509528 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:15:14.509440 5378 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-1fdaa7-openshift-default-84586db7d5-zklmb"] Oct 30 21:15:14.512384 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:15:14.512361 5378 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-1fdaa7-openshift-default-84586db7d5-zklmb"] Oct 30 21:15:14.735285 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:15:14.735257 5378 scope.go:122] "RemoveContainer" containerID="8c971bfd88838a1759a8ff13e047171c847467f44e616d353e73f89c7411a572" Oct 30 21:15:15.338511 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:15:15.338438 5378 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-1fdaa7-29889915-vfvtm" event={"ID":"acab1aa9-cd67-4e6e-afc5-a504a8064563","Type":"ContainerStarted","Data":"7d6f3e18cfc1fc4d36233549536c0c0e3524480c6fa3a27d55d57d78ac8e348f"} Oct 30 21:15:15.734625 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:15:15.734596 5378 scope.go:122] "RemoveContainer" containerID="3f3c067c4e6bd908386e20cc72954c47626a32c428288ed1f23903bac2a63755" Oct 30 21:15:15.746341 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:15:15.746296 5378 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f0685b8b-5dbc-4eb1-afcb-a390e5b464e7" path="/var/lib/kubelet/pods/f0685b8b-5dbc-4eb1-afcb-a390e5b464e7/volumes" Oct 30 21:15:16.311526 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:15:16.311484 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:15:16.348945 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:15:16.348917 5378 generic.go:356] "Generic (PLEG): container finished" podID="acab1aa9-cd67-4e6e-afc5-a504a8064563" containerID="7d6f3e18cfc1fc4d36233549536c0c0e3524480c6fa3a27d55d57d78ac8e348f" exitCode=60 Oct 30 21:15:16.349138 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:15:16.348990 5378 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-1fdaa7-29889915-vfvtm" event={"ID":"acab1aa9-cd67-4e6e-afc5-a504a8064563","Type":"ContainerDied","Data":"7d6f3e18cfc1fc4d36233549536c0c0e3524480c6fa3a27d55d57d78ac8e348f"} Oct 30 21:15:16.349138 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:15:16.349050 5378 scope.go:122] "RemoveContainer" containerID="8c971bfd88838a1759a8ff13e047171c847467f44e616d353e73f89c7411a572" Oct 30 21:15:16.349515 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:15:16.349494 5378 scope.go:122] "RemoveContainer" containerID="7d6f3e18cfc1fc4d36233549536c0c0e3524480c6fa3a27d55d57d78ac8e348f" Oct 30 21:15:16.349952 ip-10-0-138-149 kubenswrapper[5378]: E1030 21:15:16.349925 5378 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-1fdaa7-29889915-vfvtm_odh-ai-gateway-infra(acab1aa9-cd67-4e6e-afc5-a504a8064563)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-1fdaa7-29889915-vfvtm" podUID="acab1aa9-cd67-4e6e-afc5-a504a8064563" Oct 30 21:15:16.351452 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:15:16.351426 5378 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-b888d0-29889915-2dftf" event={"ID":"db796f69-9d96-4b13-930d-79276df5078d","Type":"ContainerStarted","Data":"6410d90971488b13f75f03857d0f464941792f064253f3199c8ac6ecd0471af1"} Oct 30 21:15:16.506530 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:15:16.506495 5378 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-1fdaa7-29889915-vfvtm"] Oct 30 21:15:16.548505 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:15:16.548476 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:15:16.734991 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:15:16.734896 5378 scope.go:122] "RemoveContainer" containerID="7fa1f615dd255664a6c767ca13047fa6a3eb57cb0d80fed56b915a51d79e8597" Oct 30 21:15:16.735158 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:15:16.735096 5378 scope.go:122] "RemoveContainer" containerID="32b44f8b85a43a1be130efdf08aa35d2faf6143aed716d9fa794bb212553c408" Oct 30 21:15:17.367750 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:15:17.367667 5378 generic.go:356] "Generic (PLEG): container finished" podID="db796f69-9d96-4b13-930d-79276df5078d" containerID="6410d90971488b13f75f03857d0f464941792f064253f3199c8ac6ecd0471af1" exitCode=60 Oct 30 21:15:17.367750 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:15:17.367737 5378 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-b888d0-29889915-2dftf" event={"ID":"db796f69-9d96-4b13-930d-79276df5078d","Type":"ContainerDied","Data":"6410d90971488b13f75f03857d0f464941792f064253f3199c8ac6ecd0471af1"} Oct 30 21:15:17.368295 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:15:17.367801 5378 scope.go:122] "RemoveContainer" containerID="3f3c067c4e6bd908386e20cc72954c47626a32c428288ed1f23903bac2a63755" Oct 30 21:15:17.368295 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:15:17.368243 5378 scope.go:122] "RemoveContainer" containerID="6410d90971488b13f75f03857d0f464941792f064253f3199c8ac6ecd0471af1" Oct 30 21:15:17.368803 ip-10-0-138-149 kubenswrapper[5378]: E1030 21:15:17.368762 5378 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-api-a-b888d0-29889915-2dftf_odh-ai-gateway-infra(db796f69-9d96-4b13-930d-79276df5078d)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-b888d0-29889915-2dftf" podUID="db796f69-9d96-4b13-930d-79276df5078d" Oct 30 21:15:17.376317 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:15:17.376289 5378 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-d061d0-29889915-d4fwx" event={"ID":"03043216-5312-45de-9d5c-63b2cc34b320","Type":"ContainerStarted","Data":"f8072653bb2eed67d3688c84eeebd9a4efd74b0560439e87898cc884ba76d605"} Oct 30 21:15:17.384329 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:15:17.383152 5378 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-7eaf74-29889915-g4gn9" event={"ID":"cbd70c63-4eab-49cc-9b10-f039320c1fb0","Type":"ContainerStarted","Data":"ebe6b9680811ea3df85da960729d72319bc67dd0c112a8161ef67e21b9c2de7d"} Oct 30 21:15:17.571657 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:15:17.571442 5378 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-1fdaa7-29889915-vfvtm" Oct 30 21:15:17.621015 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:15:17.620931 5378 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-b888d0-29889915-2dftf"] Oct 30 21:15:17.698645 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:15:17.698617 5378 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/acab1aa9-cd67-4e6e-afc5-a504a8064563-kube-api-access-clr2q\" (UniqueName: \"kubernetes.io/projected/acab1aa9-cd67-4e6e-afc5-a504a8064563-kube-api-access-clr2q\") pod \"acab1aa9-cd67-4e6e-afc5-a504a8064563\" (UID: \"acab1aa9-cd67-4e6e-afc5-a504a8064563\") " Oct 30 21:15:17.698816 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:15:17.698726 5378 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/acab1aa9-cd67-4e6e-afc5-a504a8064563-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/acab1aa9-cd67-4e6e-afc5-a504a8064563-maas-api-service-ca\") pod \"acab1aa9-cd67-4e6e-afc5-a504a8064563\" (UID: \"acab1aa9-cd67-4e6e-afc5-a504a8064563\") " Oct 30 21:15:17.699068 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:15:17.699042 5378 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/acab1aa9-cd67-4e6e-afc5-a504a8064563-maas-api-service-ca" pod "acab1aa9-cd67-4e6e-afc5-a504a8064563" (UID: "acab1aa9-cd67-4e6e-afc5-a504a8064563"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:15:17.699246 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:15:17.699229 5378 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/acab1aa9-cd67-4e6e-afc5-a504a8064563-maas-api-service-ca\") on node \"ip-10-0-138-149.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:15:17.700804 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:15:17.700782 5378 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/acab1aa9-cd67-4e6e-afc5-a504a8064563-kube-api-access-clr2q" pod "acab1aa9-cd67-4e6e-afc5-a504a8064563" (UID: "acab1aa9-cd67-4e6e-afc5-a504a8064563"). InnerVolumeSpecName "kube-api-access-clr2q". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:15:17.734699 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:15:17.734667 5378 scope.go:122] "RemoveContainer" containerID="c32f209c216c64c3440b76764305d8eda3133260e3d04daff1c73a76ed1e429f" Oct 30 21:15:17.735017 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:15:17.734992 5378 scope.go:122] "RemoveContainer" containerID="fec93ffdf1b1bad503151991eea25852463c70adc15a5ec5b750131dcd0e059e" Oct 30 21:15:17.801833 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:15:17.801508 5378 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-clr2q\" (UniqueName: \"kubernetes.io/projected/acab1aa9-cd67-4e6e-afc5-a504a8064563-kube-api-access-clr2q\") on node \"ip-10-0-138-149.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:15:18.394932 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:15:18.394898 5378 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-2b425d-29889915-hpg7c" event={"ID":"503a9e15-f5df-4fde-8ca1-ea1946fe581b","Type":"ContainerStarted","Data":"c1c189cf728893a603903eda337835d42b5f3545a5649f9d03121e510f73dc2f"} Oct 30 21:15:18.397059 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:15:18.397030 5378 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-781058-29889915-k2s9b" event={"ID":"789b0c2c-6150-4867-9c6b-8e7fffa5e17f","Type":"ContainerStarted","Data":"3e0dd538ecd637daf51ed40154f92c639a960b63ae6af310bc43fc5794f79359"} Oct 30 21:15:18.398409 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:15:18.398378 5378 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-1fdaa7-29889915-vfvtm" Oct 30 21:15:18.398535 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:15:18.398381 5378 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-1fdaa7-29889915-vfvtm" event={"ID":"acab1aa9-cd67-4e6e-afc5-a504a8064563","Type":"ContainerDied","Data":"add2f61042a09daaf7c0f32efd84fe5de3857019c8bb0c304ff8d7f085ef3bb1"} Oct 30 21:15:18.398535 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:15:18.398512 5378 scope.go:122] "RemoveContainer" containerID="7d6f3e18cfc1fc4d36233549536c0c0e3524480c6fa3a27d55d57d78ac8e348f" Oct 30 21:15:18.400652 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:15:18.400555 5378 generic.go:356] "Generic (PLEG): container finished" podID="03043216-5312-45de-9d5c-63b2cc34b320" containerID="f8072653bb2eed67d3688c84eeebd9a4efd74b0560439e87898cc884ba76d605" exitCode=60 Oct 30 21:15:18.400789 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:15:18.400699 5378 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-d061d0-29889915-d4fwx" event={"ID":"03043216-5312-45de-9d5c-63b2cc34b320","Type":"ContainerDied","Data":"f8072653bb2eed67d3688c84eeebd9a4efd74b0560439e87898cc884ba76d605"} Oct 30 21:15:18.401180 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:15:18.401157 5378 scope.go:122] "RemoveContainer" containerID="f8072653bb2eed67d3688c84eeebd9a4efd74b0560439e87898cc884ba76d605" Oct 30 21:15:18.401512 ip-10-0-138-149 kubenswrapper[5378]: E1030 21:15:18.401485 5378 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-d061d0-29889915-d4fwx_odh-ai-gateway-infra(03043216-5312-45de-9d5c-63b2cc34b320)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-d061d0-29889915-d4fwx" podUID="03043216-5312-45de-9d5c-63b2cc34b320" Oct 30 21:15:18.403088 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:15:18.403060 5378 generic.go:356] "Generic (PLEG): container finished" podID="cbd70c63-4eab-49cc-9b10-f039320c1fb0" containerID="ebe6b9680811ea3df85da960729d72319bc67dd0c112a8161ef67e21b9c2de7d" exitCode=60 Oct 30 21:15:18.403215 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:15:18.403134 5378 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-7eaf74-29889915-g4gn9" event={"ID":"cbd70c63-4eab-49cc-9b10-f039320c1fb0","Type":"ContainerDied","Data":"ebe6b9680811ea3df85da960729d72319bc67dd0c112a8161ef67e21b9c2de7d"} Oct 30 21:15:18.403506 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:15:18.403474 5378 scope.go:122] "RemoveContainer" containerID="ebe6b9680811ea3df85da960729d72319bc67dd0c112a8161ef67e21b9c2de7d" Oct 30 21:15:18.403755 ip-10-0-138-149 kubenswrapper[5378]: E1030 21:15:18.403736 5378 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w6-7eaf74-29889915-g4gn9_odh-ai-gateway-infra(cbd70c63-4eab-49cc-9b10-f039320c1fb0)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-7eaf74-29889915-g4gn9" podUID="cbd70c63-4eab-49cc-9b10-f039320c1fb0" Oct 30 21:15:18.411081 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:15:18.411058 5378 scope.go:122] "RemoveContainer" containerID="32b44f8b85a43a1be130efdf08aa35d2faf6143aed716d9fa794bb212553c408" Oct 30 21:15:18.423042 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:15:18.423021 5378 scope.go:122] "RemoveContainer" containerID="7fa1f615dd255664a6c767ca13047fa6a3eb57cb0d80fed56b915a51d79e8597" Oct 30 21:15:18.582421 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:15:18.582393 5378 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-b888d0-29889915-2dftf" Oct 30 21:15:18.636406 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:15:18.636372 5378 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/db796f69-9d96-4b13-930d-79276df5078d-kube-api-access-tvjwh\" (UniqueName: \"kubernetes.io/projected/db796f69-9d96-4b13-930d-79276df5078d-kube-api-access-tvjwh\") pod \"db796f69-9d96-4b13-930d-79276df5078d\" (UID: \"db796f69-9d96-4b13-930d-79276df5078d\") " Oct 30 21:15:18.636623 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:15:18.636445 5378 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/db796f69-9d96-4b13-930d-79276df5078d-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/db796f69-9d96-4b13-930d-79276df5078d-maas-api-service-ca\") pod \"db796f69-9d96-4b13-930d-79276df5078d\" (UID: \"db796f69-9d96-4b13-930d-79276df5078d\") " Oct 30 21:15:18.636871 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:15:18.636846 5378 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/db796f69-9d96-4b13-930d-79276df5078d-maas-api-service-ca" pod "db796f69-9d96-4b13-930d-79276df5078d" (UID: "db796f69-9d96-4b13-930d-79276df5078d"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:15:18.637031 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:15:18.637018 5378 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/db796f69-9d96-4b13-930d-79276df5078d-maas-api-service-ca\") on node \"ip-10-0-138-149.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:15:18.638615 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:15:18.638597 5378 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/db796f69-9d96-4b13-930d-79276df5078d-kube-api-access-tvjwh" pod "db796f69-9d96-4b13-930d-79276df5078d" (UID: "db796f69-9d96-4b13-930d-79276df5078d"). InnerVolumeSpecName "kube-api-access-tvjwh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:15:18.738808 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:15:18.738778 5378 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tvjwh\" (UniqueName: \"kubernetes.io/projected/db796f69-9d96-4b13-930d-79276df5078d-kube-api-access-tvjwh\") on node \"ip-10-0-138-149.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:15:18.740226 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:15:18.740204 5378 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-7eaf74-29889915-g4gn9"] Oct 30 21:15:18.762674 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:15:18.762644 5378 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-1fdaa7-29889915-vfvtm"] Oct 30 21:15:18.770228 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:15:18.770201 5378 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-1fdaa7-29889915-vfvtm"] Oct 30 21:15:18.828827 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:15:18.828748 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:15:18.846672 ip-10-0-138-149 kubenswrapper[5378]: E1030 21:15:18.846638 5378 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q6mx7" containerName="extractor" Oct 30 21:15:18.894761 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:15:18.894720 5378 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-d061d0-29889915-d4fwx"] Oct 30 21:15:19.159413 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:15:19.159374 5378 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-390b71-484bv"] Oct 30 21:15:19.160509 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:15:19.160485 5378 cpu_manager.go:414] "containerMap: removing container" podUID="db796f69-9d96-4b13-930d-79276df5078d" containerName="cleanup" Oct 30 21:15:19.160575 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:15:19.160514 5378 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="db796f69-9d96-4b13-930d-79276df5078d" containerName="cleanup" Oct 30 21:15:19.160575 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:15:19.160537 5378 cpu_manager.go:414] "containerMap: removing container" podUID="db796f69-9d96-4b13-930d-79276df5078d" containerName="cleanup" Oct 30 21:15:19.160575 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:15:19.160545 5378 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="db796f69-9d96-4b13-930d-79276df5078d" containerName="cleanup" Oct 30 21:15:19.160681 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:15:19.160598 5378 cpu_manager.go:414] "containerMap: removing container" podUID="acab1aa9-cd67-4e6e-afc5-a504a8064563" containerName="cleanup" Oct 30 21:15:19.160681 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:15:19.160607 5378 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="acab1aa9-cd67-4e6e-afc5-a504a8064563" containerName="cleanup" Oct 30 21:15:19.160681 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:15:19.160637 5378 cpu_manager.go:414] "containerMap: removing container" podUID="acab1aa9-cd67-4e6e-afc5-a504a8064563" containerName="cleanup" Oct 30 21:15:19.160681 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:15:19.160644 5378 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="acab1aa9-cd67-4e6e-afc5-a504a8064563" containerName="cleanup" Oct 30 21:15:19.160681 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:15:19.160657 5378 cpu_manager.go:414] "containerMap: removing container" podUID="acab1aa9-cd67-4e6e-afc5-a504a8064563" containerName="cleanup" Oct 30 21:15:19.160681 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:15:19.160664 5378 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="acab1aa9-cd67-4e6e-afc5-a504a8064563" containerName="cleanup" Oct 30 21:15:19.160857 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:15:19.160689 5378 cpu_manager.go:414] "containerMap: removing container" podUID="f0685b8b-5dbc-4eb1-afcb-a390e5b464e7" containerName="istio-proxy" Oct 30 21:15:19.160857 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:15:19.160697 5378 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f0685b8b-5dbc-4eb1-afcb-a390e5b464e7" containerName="istio-proxy" Oct 30 21:15:19.160922 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:15:19.160876 5378 memory_manager.go:365] "RemoveStaleState removing state" podUID="acab1aa9-cd67-4e6e-afc5-a504a8064563" containerName="cleanup" Oct 30 21:15:19.160922 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:15:19.160889 5378 memory_manager.go:365] "RemoveStaleState removing state" podUID="db796f69-9d96-4b13-930d-79276df5078d" containerName="cleanup" Oct 30 21:15:19.160922 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:15:19.160899 5378 memory_manager.go:365] "RemoveStaleState removing state" podUID="acab1aa9-cd67-4e6e-afc5-a504a8064563" containerName="cleanup" Oct 30 21:15:19.160922 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:15:19.160909 5378 memory_manager.go:365] "RemoveStaleState removing state" podUID="acab1aa9-cd67-4e6e-afc5-a504a8064563" containerName="cleanup" Oct 30 21:15:19.160922 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:15:19.160920 5378 memory_manager.go:365] "RemoveStaleState removing state" podUID="f0685b8b-5dbc-4eb1-afcb-a390e5b464e7" containerName="istio-proxy" Oct 30 21:15:19.161060 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:15:19.160939 5378 memory_manager.go:365] "RemoveStaleState removing state" podUID="db796f69-9d96-4b13-930d-79276df5078d" containerName="cleanup" Oct 30 21:15:19.161060 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:15:19.160950 5378 memory_manager.go:365] "RemoveStaleState removing state" podUID="db796f69-9d96-4b13-930d-79276df5078d" containerName="cleanup" Oct 30 21:15:19.164963 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:15:19.164935 5378 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-390b71-484bv" Oct 30 21:15:19.220601 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:15:19.220535 5378 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 21:15:19.225169 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:15:19.225132 5378 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-390b71-484bv"] Oct 30 21:15:19.246325 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:15:19.246292 5378 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b24b6671-0dce-4274-8cd1-af908fc9e591-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-390b71-484bv\" (UID: \"b24b6671-0dce-4274-8cd1-af908fc9e591\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-390b71-484bv" Oct 30 21:15:19.246515 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:15:19.246481 5378 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b24b6671-0dce-4274-8cd1-af908fc9e591-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-390b71-484bv\" (UID: \"b24b6671-0dce-4274-8cd1-af908fc9e591\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-390b71-484bv" Oct 30 21:15:19.246714 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:15:19.246694 5378 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h2927\" (UniqueName: \"kubernetes.io/projected/b24b6671-0dce-4274-8cd1-af908fc9e591-kube-api-access-h2927\") pod \"maas-api-revoke-keys-e2e-inf-a-390b71-484bv\" (UID: \"b24b6671-0dce-4274-8cd1-af908fc9e591\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-390b71-484bv" Oct 30 21:15:19.347894 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:15:19.347848 5378 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b24b6671-0dce-4274-8cd1-af908fc9e591-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-390b71-484bv\" (UID: \"b24b6671-0dce-4274-8cd1-af908fc9e591\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-390b71-484bv" Oct 30 21:15:19.348088 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:15:19.348015 5378 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h2927\" (UniqueName: \"kubernetes.io/projected/b24b6671-0dce-4274-8cd1-af908fc9e591-kube-api-access-h2927\") pod \"maas-api-revoke-keys-e2e-inf-a-390b71-484bv\" (UID: \"b24b6671-0dce-4274-8cd1-af908fc9e591\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-390b71-484bv" Oct 30 21:15:19.348088 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:15:19.348074 5378 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b24b6671-0dce-4274-8cd1-af908fc9e591-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-390b71-484bv\" (UID: \"b24b6671-0dce-4274-8cd1-af908fc9e591\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-390b71-484bv" Oct 30 21:15:19.348555 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:15:19.348527 5378 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b24b6671-0dce-4274-8cd1-af908fc9e591-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-390b71-484bv\" (UID: \"b24b6671-0dce-4274-8cd1-af908fc9e591\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-390b71-484bv" Oct 30 21:15:19.350861 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:15:19.350824 5378 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b24b6671-0dce-4274-8cd1-af908fc9e591-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-390b71-484bv\" (UID: \"b24b6671-0dce-4274-8cd1-af908fc9e591\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-390b71-484bv" Oct 30 21:15:19.408233 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:15:19.408203 5378 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h2927\" (UniqueName: \"kubernetes.io/projected/b24b6671-0dce-4274-8cd1-af908fc9e591-kube-api-access-h2927\") pod \"maas-api-revoke-keys-e2e-inf-a-390b71-484bv\" (UID: \"b24b6671-0dce-4274-8cd1-af908fc9e591\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-390b71-484bv" Oct 30 21:15:19.418529 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:15:19.418433 5378 generic.go:356] "Generic (PLEG): container finished" podID="503a9e15-f5df-4fde-8ca1-ea1946fe581b" containerID="c1c189cf728893a603903eda337835d42b5f3545a5649f9d03121e510f73dc2f" exitCode=60 Oct 30 21:15:19.418529 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:15:19.418469 5378 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-2b425d-29889915-hpg7c" event={"ID":"503a9e15-f5df-4fde-8ca1-ea1946fe581b","Type":"ContainerDied","Data":"c1c189cf728893a603903eda337835d42b5f3545a5649f9d03121e510f73dc2f"} Oct 30 21:15:19.418794 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:15:19.418559 5378 scope.go:122] "RemoveContainer" containerID="c32f209c216c64c3440b76764305d8eda3133260e3d04daff1c73a76ed1e429f" Oct 30 21:15:19.419051 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:15:19.419031 5378 scope.go:122] "RemoveContainer" containerID="c1c189cf728893a603903eda337835d42b5f3545a5649f9d03121e510f73dc2f" Oct 30 21:15:19.419484 ip-10-0-138-149 kubenswrapper[5378]: E1030 21:15:19.419455 5378 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w5-2b425d-29889915-hpg7c_odh-ai-gateway-infra(503a9e15-f5df-4fde-8ca1-ea1946fe581b)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-2b425d-29889915-hpg7c" podUID="503a9e15-f5df-4fde-8ca1-ea1946fe581b" Oct 30 21:15:19.423660 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:15:19.423169 5378 generic.go:356] "Generic (PLEG): container finished" podID="789b0c2c-6150-4867-9c6b-8e7fffa5e17f" containerID="3e0dd538ecd637daf51ed40154f92c639a960b63ae6af310bc43fc5794f79359" exitCode=60 Oct 30 21:15:19.423660 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:15:19.423273 5378 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-781058-29889915-k2s9b" event={"ID":"789b0c2c-6150-4867-9c6b-8e7fffa5e17f","Type":"ContainerDied","Data":"3e0dd538ecd637daf51ed40154f92c639a960b63ae6af310bc43fc5794f79359"} Oct 30 21:15:19.424579 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:15:19.424094 5378 scope.go:122] "RemoveContainer" containerID="3e0dd538ecd637daf51ed40154f92c639a960b63ae6af310bc43fc5794f79359" Oct 30 21:15:19.424579 ip-10-0-138-149 kubenswrapper[5378]: E1030 21:15:19.424497 5378 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-781058-29889915-k2s9b_odh-ai-gateway-infra(789b0c2c-6150-4867-9c6b-8e7fffa5e17f)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-781058-29889915-k2s9b" podUID="789b0c2c-6150-4867-9c6b-8e7fffa5e17f" Oct 30 21:15:19.443247 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:15:19.440550 5378 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-b888d0-29889915-2dftf" Oct 30 21:15:19.443247 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:15:19.440822 5378 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-b888d0-29889915-2dftf" event={"ID":"db796f69-9d96-4b13-930d-79276df5078d","Type":"ContainerDied","Data":"644861daed82c202fc066637db6a5e2e75d5134a9ea0348c487a815c2c7cd04a"} Oct 30 21:15:19.467150 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:15:19.467124 5378 scope.go:122] "RemoveContainer" containerID="fec93ffdf1b1bad503151991eea25852463c70adc15a5ec5b750131dcd0e059e" Oct 30 21:15:19.483051 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:15:19.483019 5378 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-390b71-484bv" Oct 30 21:15:19.492037 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:15:19.491906 5378 scope.go:122] "RemoveContainer" containerID="6410d90971488b13f75f03857d0f464941792f064253f3199c8ac6ecd0471af1" Oct 30 21:15:19.542757 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:15:19.541500 5378 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-2b425d-29889915-hpg7c"] Oct 30 21:15:19.638072 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:15:19.637980 5378 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-d061d0-29889915-d4fwx" Oct 30 21:15:19.725322 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:15:19.725286 5378 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-7eaf74-29889915-g4gn9" Oct 30 21:15:19.748846 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:15:19.747736 5378 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="acab1aa9-cd67-4e6e-afc5-a504a8064563" path="/var/lib/kubelet/pods/acab1aa9-cd67-4e6e-afc5-a504a8064563/volumes" Oct 30 21:15:19.766148 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:15:19.754119 5378 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/03043216-5312-45de-9d5c-63b2cc34b320-kube-api-access-gh9kx\" (UniqueName: \"kubernetes.io/projected/03043216-5312-45de-9d5c-63b2cc34b320-kube-api-access-gh9kx\") pod \"03043216-5312-45de-9d5c-63b2cc34b320\" (UID: \"03043216-5312-45de-9d5c-63b2cc34b320\") " Oct 30 21:15:19.766148 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:15:19.754261 5378 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/03043216-5312-45de-9d5c-63b2cc34b320-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/03043216-5312-45de-9d5c-63b2cc34b320-maas-api-service-ca\") pod \"03043216-5312-45de-9d5c-63b2cc34b320\" (UID: \"03043216-5312-45de-9d5c-63b2cc34b320\") " Oct 30 21:15:19.766148 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:15:19.755745 5378 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/03043216-5312-45de-9d5c-63b2cc34b320-maas-api-service-ca" pod "03043216-5312-45de-9d5c-63b2cc34b320" (UID: "03043216-5312-45de-9d5c-63b2cc34b320"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:15:19.766148 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:15:19.762791 5378 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/03043216-5312-45de-9d5c-63b2cc34b320-kube-api-access-gh9kx" pod "03043216-5312-45de-9d5c-63b2cc34b320" (UID: "03043216-5312-45de-9d5c-63b2cc34b320"). InnerVolumeSpecName "kube-api-access-gh9kx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:15:19.855709 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:15:19.855660 5378 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cbd70c63-4eab-49cc-9b10-f039320c1fb0-kube-api-access-rh7rm\" (UniqueName: \"kubernetes.io/projected/cbd70c63-4eab-49cc-9b10-f039320c1fb0-kube-api-access-rh7rm\") pod \"cbd70c63-4eab-49cc-9b10-f039320c1fb0\" (UID: \"cbd70c63-4eab-49cc-9b10-f039320c1fb0\") " Oct 30 21:15:19.856015 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:15:19.855793 5378 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cbd70c63-4eab-49cc-9b10-f039320c1fb0-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cbd70c63-4eab-49cc-9b10-f039320c1fb0-maas-api-service-ca\") pod \"cbd70c63-4eab-49cc-9b10-f039320c1fb0\" (UID: \"cbd70c63-4eab-49cc-9b10-f039320c1fb0\") " Oct 30 21:15:19.856240 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:15:19.856210 5378 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cbd70c63-4eab-49cc-9b10-f039320c1fb0-maas-api-service-ca" pod "cbd70c63-4eab-49cc-9b10-f039320c1fb0" (UID: "cbd70c63-4eab-49cc-9b10-f039320c1fb0"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:15:19.856479 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:15:19.856458 5378 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gh9kx\" (UniqueName: \"kubernetes.io/projected/03043216-5312-45de-9d5c-63b2cc34b320-kube-api-access-gh9kx\") on node \"ip-10-0-138-149.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:15:19.856479 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:15:19.856478 5378 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cbd70c63-4eab-49cc-9b10-f039320c1fb0-maas-api-service-ca\") on node \"ip-10-0-138-149.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:15:19.856669 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:15:19.856488 5378 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/03043216-5312-45de-9d5c-63b2cc34b320-maas-api-service-ca\") on node \"ip-10-0-138-149.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:15:19.858312 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:15:19.858279 5378 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cbd70c63-4eab-49cc-9b10-f039320c1fb0-kube-api-access-rh7rm" pod "cbd70c63-4eab-49cc-9b10-f039320c1fb0" (UID: "cbd70c63-4eab-49cc-9b10-f039320c1fb0"). InnerVolumeSpecName "kube-api-access-rh7rm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:15:19.925011 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:15:19.924975 5378 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-b888d0-29889915-2dftf"] Oct 30 21:15:19.927558 ip-10-0-138-149 kubenswrapper[5378]: W1030 21:15:19.927511 5378 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb24b6671_0dce_4274_8cd1_af908fc9e591.slice/crio-e8be4e894fbaa3c9a0e0c1726787bcb70a9955d58f5f8f83c8a0278d8504d646 WatchSource:0}: Error finding container e8be4e894fbaa3c9a0e0c1726787bcb70a9955d58f5f8f83c8a0278d8504d646: Status 404 returned error can't find the container with id e8be4e894fbaa3c9a0e0c1726787bcb70a9955d58f5f8f83c8a0278d8504d646 Oct 30 21:15:19.929864 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:15:19.929751 5378 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-390b71-484bv"] Oct 30 21:15:19.933931 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:15:19.933904 5378 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-b888d0-29889915-2dftf"] Oct 30 21:15:19.940381 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:15:19.940357 5378 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-781058-29889915-k2s9b"] Oct 30 21:15:19.957482 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:15:19.957448 5378 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rh7rm\" (UniqueName: \"kubernetes.io/projected/cbd70c63-4eab-49cc-9b10-f039320c1fb0-kube-api-access-rh7rm\") on node \"ip-10-0-138-149.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:15:20.465538 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:15:20.465252 5378 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-390b71-484bv" event={"ID":"b24b6671-0dce-4274-8cd1-af908fc9e591","Type":"ContainerStarted","Data":"65d9e00da5785d63948239191de1cb71acf1464768dbdf42ee5d168029863d49"} Oct 30 21:15:20.465538 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:15:20.465364 5378 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-390b71-484bv" event={"ID":"b24b6671-0dce-4274-8cd1-af908fc9e591","Type":"ContainerStarted","Data":"e8be4e894fbaa3c9a0e0c1726787bcb70a9955d58f5f8f83c8a0278d8504d646"} Oct 30 21:15:20.469189 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:15:20.469168 5378 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-d061d0-29889915-d4fwx" Oct 30 21:15:20.469304 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:15:20.469170 5378 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-d061d0-29889915-d4fwx" event={"ID":"03043216-5312-45de-9d5c-63b2cc34b320","Type":"ContainerDied","Data":"5180363807b25f3ded03e0013780c330a742e6d1cd3a3b08b0bfe019fbfe9e07"} Oct 30 21:15:20.469304 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:15:20.469297 5378 scope.go:122] "RemoveContainer" containerID="f8072653bb2eed67d3688c84eeebd9a4efd74b0560439e87898cc884ba76d605" Oct 30 21:15:20.471245 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:15:20.471210 5378 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-7eaf74-29889915-g4gn9" Oct 30 21:15:20.471678 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:15:20.471258 5378 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-7eaf74-29889915-g4gn9" event={"ID":"cbd70c63-4eab-49cc-9b10-f039320c1fb0","Type":"ContainerDied","Data":"52bf73eac19f29e48f4b2d5def739526c1d3e3ad5376a5bd61e17d520119ad7e"} Oct 30 21:15:20.482500 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:15:20.482403 5378 scope.go:122] "RemoveContainer" containerID="ebe6b9680811ea3df85da960729d72319bc67dd0c112a8161ef67e21b9c2de7d" Oct 30 21:15:20.623096 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:15:20.623064 5378 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-2b425d-29889915-hpg7c" Oct 30 21:15:20.663158 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:15:20.663129 5378 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-781058-29889915-k2s9b" Oct 30 21:15:20.772383 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:15:20.772354 5378 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/503a9e15-f5df-4fde-8ca1-ea1946fe581b-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/503a9e15-f5df-4fde-8ca1-ea1946fe581b-maas-api-service-ca\") pod \"503a9e15-f5df-4fde-8ca1-ea1946fe581b\" (UID: \"503a9e15-f5df-4fde-8ca1-ea1946fe581b\") " Oct 30 21:15:20.772527 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:15:20.772486 5378 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/789b0c2c-6150-4867-9c6b-8e7fffa5e17f-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/789b0c2c-6150-4867-9c6b-8e7fffa5e17f-maas-api-service-ca\") pod \"789b0c2c-6150-4867-9c6b-8e7fffa5e17f\" (UID: \"789b0c2c-6150-4867-9c6b-8e7fffa5e17f\") " Oct 30 21:15:20.772604 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:15:20.772530 5378 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/503a9e15-f5df-4fde-8ca1-ea1946fe581b-kube-api-access-ptg2r\" (UniqueName: \"kubernetes.io/projected/503a9e15-f5df-4fde-8ca1-ea1946fe581b-kube-api-access-ptg2r\") pod \"503a9e15-f5df-4fde-8ca1-ea1946fe581b\" (UID: \"503a9e15-f5df-4fde-8ca1-ea1946fe581b\") " Oct 30 21:15:20.772604 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:15:20.772552 5378 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/789b0c2c-6150-4867-9c6b-8e7fffa5e17f-kube-api-access-zbmz8\" (UniqueName: \"kubernetes.io/projected/789b0c2c-6150-4867-9c6b-8e7fffa5e17f-kube-api-access-zbmz8\") pod \"789b0c2c-6150-4867-9c6b-8e7fffa5e17f\" (UID: \"789b0c2c-6150-4867-9c6b-8e7fffa5e17f\") " Oct 30 21:15:20.772721 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:15:20.772686 5378 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/503a9e15-f5df-4fde-8ca1-ea1946fe581b-maas-api-service-ca" pod "503a9e15-f5df-4fde-8ca1-ea1946fe581b" (UID: "503a9e15-f5df-4fde-8ca1-ea1946fe581b"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:15:20.772897 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:15:20.772875 5378 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/789b0c2c-6150-4867-9c6b-8e7fffa5e17f-maas-api-service-ca" pod "789b0c2c-6150-4867-9c6b-8e7fffa5e17f" (UID: "789b0c2c-6150-4867-9c6b-8e7fffa5e17f"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:15:20.773099 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:15:20.773084 5378 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/503a9e15-f5df-4fde-8ca1-ea1946fe581b-maas-api-service-ca\") on node \"ip-10-0-138-149.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:15:20.773164 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:15:20.773101 5378 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/789b0c2c-6150-4867-9c6b-8e7fffa5e17f-maas-api-service-ca\") on node \"ip-10-0-138-149.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:15:20.774739 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:15:20.774689 5378 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/503a9e15-f5df-4fde-8ca1-ea1946fe581b-kube-api-access-ptg2r" pod "503a9e15-f5df-4fde-8ca1-ea1946fe581b" (UID: "503a9e15-f5df-4fde-8ca1-ea1946fe581b"). InnerVolumeSpecName "kube-api-access-ptg2r". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:15:20.774739 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:15:20.774704 5378 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/789b0c2c-6150-4867-9c6b-8e7fffa5e17f-kube-api-access-zbmz8" pod "789b0c2c-6150-4867-9c6b-8e7fffa5e17f" (UID: "789b0c2c-6150-4867-9c6b-8e7fffa5e17f"). InnerVolumeSpecName "kube-api-access-zbmz8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:15:20.821532 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:15:20.821481 5378 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-390b71-484bv" podStartSLOduration=1.821468912 podStartE2EDuration="1.821468912s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:15:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:15:20.820117605 +0000 UTC m=+2211.248231536" watchObservedRunningTime="2026-10-30 21:15:20.821468912 +0000 UTC m=+2211.249582831" Oct 30 21:15:20.874439 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:15:20.874414 5378 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ptg2r\" (UniqueName: \"kubernetes.io/projected/503a9e15-f5df-4fde-8ca1-ea1946fe581b-kube-api-access-ptg2r\") on node \"ip-10-0-138-149.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:15:20.874439 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:15:20.874439 5378 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zbmz8\" (UniqueName: \"kubernetes.io/projected/789b0c2c-6150-4867-9c6b-8e7fffa5e17f-kube-api-access-zbmz8\") on node \"ip-10-0-138-149.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:15:20.944292 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:15:20.944259 5378 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-7eaf74-29889915-g4gn9"] Oct 30 21:15:20.947167 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:15:20.947145 5378 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-7eaf74-29889915-g4gn9"] Oct 30 21:15:21.057477 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:15:21.057394 5378 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-d061d0-29889915-d4fwx"] Oct 30 21:15:21.068658 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:15:21.068631 5378 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-d061d0-29889915-d4fwx"] Oct 30 21:15:21.408333 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:15:21.408299 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:15:21.489802 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:15:21.489771 5378 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-2b425d-29889915-hpg7c" Oct 30 21:15:21.490279 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:15:21.489809 5378 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-2b425d-29889915-hpg7c" event={"ID":"503a9e15-f5df-4fde-8ca1-ea1946fe581b","Type":"ContainerDied","Data":"b85902fa94eebac2154132a1d6d5e4dc03992188a2ced57601b93618b538c5bc"} Oct 30 21:15:21.490279 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:15:21.489877 5378 scope.go:122] "RemoveContainer" containerID="c1c189cf728893a603903eda337835d42b5f3545a5649f9d03121e510f73dc2f" Oct 30 21:15:21.491678 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:15:21.491648 5378 generic.go:356] "Generic (PLEG): container finished" podID="b24b6671-0dce-4274-8cd1-af908fc9e591" containerID="65d9e00da5785d63948239191de1cb71acf1464768dbdf42ee5d168029863d49" exitCode=0 Oct 30 21:15:21.491797 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:15:21.491690 5378 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-390b71-484bv" event={"ID":"b24b6671-0dce-4274-8cd1-af908fc9e591","Type":"ContainerDied","Data":"65d9e00da5785d63948239191de1cb71acf1464768dbdf42ee5d168029863d49"} Oct 30 21:15:21.493128 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:15:21.493105 5378 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-781058-29889915-k2s9b" event={"ID":"789b0c2c-6150-4867-9c6b-8e7fffa5e17f","Type":"ContainerDied","Data":"ef7f232644193b170aec78d700ce912937f89b7e5c296b68d4c294a17d874183"} Oct 30 21:15:21.493238 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:15:21.493130 5378 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-781058-29889915-k2s9b" Oct 30 21:15:21.503701 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:15:21.503681 5378 scope.go:122] "RemoveContainer" containerID="3e0dd538ecd637daf51ed40154f92c639a960b63ae6af310bc43fc5794f79359" Oct 30 21:15:21.652428 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:15:21.652392 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:15:21.745016 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:15:21.744944 5378 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="03043216-5312-45de-9d5c-63b2cc34b320" path="/var/lib/kubelet/pods/03043216-5312-45de-9d5c-63b2cc34b320/volumes" Oct 30 21:15:21.745432 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:15:21.745415 5378 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cbd70c63-4eab-49cc-9b10-f039320c1fb0" path="/var/lib/kubelet/pods/cbd70c63-4eab-49cc-9b10-f039320c1fb0/volumes" Oct 30 21:15:21.745930 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:15:21.745909 5378 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="db796f69-9d96-4b13-930d-79276df5078d" path="/var/lib/kubelet/pods/db796f69-9d96-4b13-930d-79276df5078d/volumes" Oct 30 21:15:21.800496 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:15:21.800469 5378 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-2b425d-29889915-hpg7c"] Oct 30 21:15:21.811721 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:15:21.811696 5378 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-2b425d-29889915-hpg7c"] Oct 30 21:15:21.925067 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:15:21.925036 5378 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-781058-29889915-k2s9b"] Oct 30 21:15:21.929541 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:15:21.929518 5378 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-781058-29889915-k2s9b"] Oct 30 21:15:22.680741 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:15:22.680717 5378 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-390b71-484bv" Oct 30 21:15:22.799425 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:15:22.799306 5378 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b24b6671-0dce-4274-8cd1-af908fc9e591-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b24b6671-0dce-4274-8cd1-af908fc9e591-maas-api-openshift-ca\") pod \"b24b6671-0dce-4274-8cd1-af908fc9e591\" (UID: \"b24b6671-0dce-4274-8cd1-af908fc9e591\") " Oct 30 21:15:22.799425 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:15:22.799396 5378 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b24b6671-0dce-4274-8cd1-af908fc9e591-kube-api-access-h2927\" (UniqueName: \"kubernetes.io/projected/b24b6671-0dce-4274-8cd1-af908fc9e591-kube-api-access-h2927\") pod \"b24b6671-0dce-4274-8cd1-af908fc9e591\" (UID: \"b24b6671-0dce-4274-8cd1-af908fc9e591\") " Oct 30 21:15:22.799745 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:15:22.799541 5378 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b24b6671-0dce-4274-8cd1-af908fc9e591-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b24b6671-0dce-4274-8cd1-af908fc9e591-maas-api-xks-ca\") pod \"b24b6671-0dce-4274-8cd1-af908fc9e591\" (UID: \"b24b6671-0dce-4274-8cd1-af908fc9e591\") " Oct 30 21:15:22.799745 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:15:22.799700 5378 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b24b6671-0dce-4274-8cd1-af908fc9e591-maas-api-openshift-ca" pod "b24b6671-0dce-4274-8cd1-af908fc9e591" (UID: "b24b6671-0dce-4274-8cd1-af908fc9e591"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:15:22.800314 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:15:22.800267 5378 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b24b6671-0dce-4274-8cd1-af908fc9e591-maas-api-openshift-ca\") on node \"ip-10-0-138-149.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:15:22.802326 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:15:22.802292 5378 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b24b6671-0dce-4274-8cd1-af908fc9e591-kube-api-access-h2927" pod "b24b6671-0dce-4274-8cd1-af908fc9e591" (UID: "b24b6671-0dce-4274-8cd1-af908fc9e591"). InnerVolumeSpecName "kube-api-access-h2927". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:15:22.802815 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:15:22.802778 5378 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b24b6671-0dce-4274-8cd1-af908fc9e591-maas-api-xks-ca" pod "b24b6671-0dce-4274-8cd1-af908fc9e591" (UID: "b24b6671-0dce-4274-8cd1-af908fc9e591"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:15:22.902005 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:15:22.901975 5378 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h2927\" (UniqueName: \"kubernetes.io/projected/b24b6671-0dce-4274-8cd1-af908fc9e591-kube-api-access-h2927\") on node \"ip-10-0-138-149.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:15:22.902005 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:15:22.902003 5378 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b24b6671-0dce-4274-8cd1-af908fc9e591-maas-api-xks-ca\") on node \"ip-10-0-138-149.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:15:23.366898 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:15:23.366861 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:15:23.528589 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:15:23.528341 5378 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-390b71-484bv" event={"ID":"b24b6671-0dce-4274-8cd1-af908fc9e591","Type":"ContainerDied","Data":"e8be4e894fbaa3c9a0e0c1726787bcb70a9955d58f5f8f83c8a0278d8504d646"} Oct 30 21:15:23.528589 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:15:23.528405 5378 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e8be4e894fbaa3c9a0e0c1726787bcb70a9955d58f5f8f83c8a0278d8504d646" Oct 30 21:15:23.528589 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:15:23.528481 5378 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-390b71-484bv" Oct 30 21:15:23.752863 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:15:23.752754 5378 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="503a9e15-f5df-4fde-8ca1-ea1946fe581b" path="/var/lib/kubelet/pods/503a9e15-f5df-4fde-8ca1-ea1946fe581b/volumes" Oct 30 21:15:23.753390 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:15:23.753368 5378 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="789b0c2c-6150-4867-9c6b-8e7fffa5e17f" path="/var/lib/kubelet/pods/789b0c2c-6150-4867-9c6b-8e7fffa5e17f/volumes" Oct 30 21:15:24.299588 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:15:24.298549 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:15:24.451291 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:15:24.451261 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:15:24.858342 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:15:24.858300 5378 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-f0fe13-588d6cb567-29spw"] Oct 30 21:15:24.858978 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:15:24.858960 5378 cpu_manager.go:414] "containerMap: removing container" podUID="03043216-5312-45de-9d5c-63b2cc34b320" containerName="cleanup" Oct 30 21:15:24.858978 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:15:24.858979 5378 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="03043216-5312-45de-9d5c-63b2cc34b320" containerName="cleanup" Oct 30 21:15:24.859083 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:15:24.858998 5378 cpu_manager.go:414] "containerMap: removing container" podUID="789b0c2c-6150-4867-9c6b-8e7fffa5e17f" containerName="cleanup" Oct 30 21:15:24.859083 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:15:24.859004 5378 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="789b0c2c-6150-4867-9c6b-8e7fffa5e17f" containerName="cleanup" Oct 30 21:15:24.859083 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:15:24.859012 5378 cpu_manager.go:414] "containerMap: removing container" podUID="db796f69-9d96-4b13-930d-79276df5078d" containerName="cleanup" Oct 30 21:15:24.859083 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:15:24.859018 5378 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="db796f69-9d96-4b13-930d-79276df5078d" containerName="cleanup" Oct 30 21:15:24.859083 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:15:24.859025 5378 cpu_manager.go:414] "containerMap: removing container" podUID="503a9e15-f5df-4fde-8ca1-ea1946fe581b" containerName="cleanup" Oct 30 21:15:24.859083 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:15:24.859030 5378 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="503a9e15-f5df-4fde-8ca1-ea1946fe581b" containerName="cleanup" Oct 30 21:15:24.859083 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:15:24.859039 5378 cpu_manager.go:414] "containerMap: removing container" podUID="cbd70c63-4eab-49cc-9b10-f039320c1fb0" containerName="cleanup" Oct 30 21:15:24.859083 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:15:24.859044 5378 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cbd70c63-4eab-49cc-9b10-f039320c1fb0" containerName="cleanup" Oct 30 21:15:24.859083 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:15:24.859054 5378 cpu_manager.go:414] "containerMap: removing container" podUID="b24b6671-0dce-4274-8cd1-af908fc9e591" containerName="revoke-keys" Oct 30 21:15:24.859083 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:15:24.859058 5378 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b24b6671-0dce-4274-8cd1-af908fc9e591" containerName="revoke-keys" Oct 30 21:15:24.859083 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:15:24.859068 5378 cpu_manager.go:414] "containerMap: removing container" podUID="789b0c2c-6150-4867-9c6b-8e7fffa5e17f" containerName="cleanup" Oct 30 21:15:24.859083 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:15:24.859073 5378 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="789b0c2c-6150-4867-9c6b-8e7fffa5e17f" containerName="cleanup" Oct 30 21:15:24.859083 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:15:24.859086 5378 cpu_manager.go:414] "containerMap: removing container" podUID="503a9e15-f5df-4fde-8ca1-ea1946fe581b" containerName="cleanup" Oct 30 21:15:24.859083 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:15:24.859091 5378 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="503a9e15-f5df-4fde-8ca1-ea1946fe581b" containerName="cleanup" Oct 30 21:15:24.859788 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:15:24.859109 5378 cpu_manager.go:414] "containerMap: removing container" podUID="cbd70c63-4eab-49cc-9b10-f039320c1fb0" containerName="cleanup" Oct 30 21:15:24.859788 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:15:24.859115 5378 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cbd70c63-4eab-49cc-9b10-f039320c1fb0" containerName="cleanup" Oct 30 21:15:24.859788 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:15:24.859122 5378 cpu_manager.go:414] "containerMap: removing container" podUID="03043216-5312-45de-9d5c-63b2cc34b320" containerName="cleanup" Oct 30 21:15:24.859788 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:15:24.859127 5378 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="03043216-5312-45de-9d5c-63b2cc34b320" containerName="cleanup" Oct 30 21:15:24.859788 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:15:24.859215 5378 memory_manager.go:365] "RemoveStaleState removing state" podUID="789b0c2c-6150-4867-9c6b-8e7fffa5e17f" containerName="cleanup" Oct 30 21:15:24.859788 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:15:24.859223 5378 memory_manager.go:365] "RemoveStaleState removing state" podUID="503a9e15-f5df-4fde-8ca1-ea1946fe581b" containerName="cleanup" Oct 30 21:15:24.859788 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:15:24.859229 5378 memory_manager.go:365] "RemoveStaleState removing state" podUID="cbd70c63-4eab-49cc-9b10-f039320c1fb0" containerName="cleanup" Oct 30 21:15:24.859788 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:15:24.859237 5378 memory_manager.go:365] "RemoveStaleState removing state" podUID="b24b6671-0dce-4274-8cd1-af908fc9e591" containerName="revoke-keys" Oct 30 21:15:24.859788 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:15:24.859246 5378 memory_manager.go:365] "RemoveStaleState removing state" podUID="cbd70c63-4eab-49cc-9b10-f039320c1fb0" containerName="cleanup" Oct 30 21:15:24.859788 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:15:24.859252 5378 memory_manager.go:365] "RemoveStaleState removing state" podUID="503a9e15-f5df-4fde-8ca1-ea1946fe581b" containerName="cleanup" Oct 30 21:15:24.859788 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:15:24.859259 5378 memory_manager.go:365] "RemoveStaleState removing state" podUID="03043216-5312-45de-9d5c-63b2cc34b320" containerName="cleanup" Oct 30 21:15:24.859788 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:15:24.859264 5378 memory_manager.go:365] "RemoveStaleState removing state" podUID="03043216-5312-45de-9d5c-63b2cc34b320" containerName="cleanup" Oct 30 21:15:24.859788 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:15:24.859271 5378 memory_manager.go:365] "RemoveStaleState removing state" podUID="789b0c2c-6150-4867-9c6b-8e7fffa5e17f" containerName="cleanup" Oct 30 21:15:24.862624 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:15:24.862606 5378 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-f0fe13-588d6cb567-29spw" Oct 30 21:15:24.921864 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:15:24.921829 5378 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-a-f0fe13-serving-cert\"" Oct 30 21:15:24.945894 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:15:24.945854 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:15:24.976429 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:15:24.976398 5378 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-f0fe13-588d6cb567-29spw"] Oct 30 21:15:25.055276 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:15:25.055233 5378 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2zh4d\" (UniqueName: \"kubernetes.io/projected/d251ff08-e87c-4819-a8f9-4cc702bc75ac-kube-api-access-2zh4d\") pod \"maas-api-e2e-inf-a-f0fe13-588d6cb567-29spw\" (UID: \"d251ff08-e87c-4819-a8f9-4cc702bc75ac\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-f0fe13-588d6cb567-29spw" Oct 30 21:15:25.055475 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:15:25.055348 5378 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d251ff08-e87c-4819-a8f9-4cc702bc75ac-maas-api-tls\") pod \"maas-api-e2e-inf-a-f0fe13-588d6cb567-29spw\" (UID: \"d251ff08-e87c-4819-a8f9-4cc702bc75ac\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-f0fe13-588d6cb567-29spw" Oct 30 21:15:25.055475 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:15:25.055450 5378 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d251ff08-e87c-4819-a8f9-4cc702bc75ac-openshift-service-ca\") pod \"maas-api-e2e-inf-a-f0fe13-588d6cb567-29spw\" (UID: \"d251ff08-e87c-4819-a8f9-4cc702bc75ac\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-f0fe13-588d6cb567-29spw" Oct 30 21:15:25.055604 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:15:25.055511 5378 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d251ff08-e87c-4819-a8f9-4cc702bc75ac-metrics-certs\") pod \"maas-api-e2e-inf-a-f0fe13-588d6cb567-29spw\" (UID: \"d251ff08-e87c-4819-a8f9-4cc702bc75ac\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-f0fe13-588d6cb567-29spw" Oct 30 21:15:25.160817 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:15:25.157235 5378 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2zh4d\" (UniqueName: \"kubernetes.io/projected/d251ff08-e87c-4819-a8f9-4cc702bc75ac-kube-api-access-2zh4d\") pod \"maas-api-e2e-inf-a-f0fe13-588d6cb567-29spw\" (UID: \"d251ff08-e87c-4819-a8f9-4cc702bc75ac\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-f0fe13-588d6cb567-29spw" Oct 30 21:15:25.160817 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:15:25.157341 5378 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d251ff08-e87c-4819-a8f9-4cc702bc75ac-maas-api-tls\") pod \"maas-api-e2e-inf-a-f0fe13-588d6cb567-29spw\" (UID: \"d251ff08-e87c-4819-a8f9-4cc702bc75ac\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-f0fe13-588d6cb567-29spw" Oct 30 21:15:25.160817 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:15:25.157489 5378 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d251ff08-e87c-4819-a8f9-4cc702bc75ac-openshift-service-ca\") pod \"maas-api-e2e-inf-a-f0fe13-588d6cb567-29spw\" (UID: \"d251ff08-e87c-4819-a8f9-4cc702bc75ac\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-f0fe13-588d6cb567-29spw" Oct 30 21:15:25.160817 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:15:25.157542 5378 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d251ff08-e87c-4819-a8f9-4cc702bc75ac-metrics-certs\") pod \"maas-api-e2e-inf-a-f0fe13-588d6cb567-29spw\" (UID: \"d251ff08-e87c-4819-a8f9-4cc702bc75ac\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-f0fe13-588d6cb567-29spw" Oct 30 21:15:25.162942 ip-10-0-138-149 kubenswrapper[5378]: E1030 21:15:25.162901 5378 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-inf-a-f0fe13-serving-cert: secret "maas-api-e2e-inf-a-f0fe13-serving-cert" not found Oct 30 21:15:25.163081 ip-10-0-138-149 kubenswrapper[5378]: E1030 21:15:25.162989 5378 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d251ff08-e87c-4819-a8f9-4cc702bc75ac-maas-api-tls podName:d251ff08-e87c-4819-a8f9-4cc702bc75ac nodeName:}" failed. No retries permitted until 2026-10-30 21:15:25.662969166 +0000 UTC m=+2216.091083065 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/d251ff08-e87c-4819-a8f9-4cc702bc75ac-maas-api-tls") pod "maas-api-e2e-inf-a-f0fe13-588d6cb567-29spw" (UID: "d251ff08-e87c-4819-a8f9-4cc702bc75ac") : secret "maas-api-e2e-inf-a-f0fe13-serving-cert" not found Oct 30 21:15:25.164350 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:15:25.164307 5378 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d251ff08-e87c-4819-a8f9-4cc702bc75ac-openshift-service-ca\") pod \"maas-api-e2e-inf-a-f0fe13-588d6cb567-29spw\" (UID: \"d251ff08-e87c-4819-a8f9-4cc702bc75ac\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-f0fe13-588d6cb567-29spw" Oct 30 21:15:25.172610 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:15:25.166211 5378 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d251ff08-e87c-4819-a8f9-4cc702bc75ac-metrics-certs\") pod \"maas-api-e2e-inf-a-f0fe13-588d6cb567-29spw\" (UID: \"d251ff08-e87c-4819-a8f9-4cc702bc75ac\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-f0fe13-588d6cb567-29spw" Oct 30 21:15:25.206390 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:15:25.206356 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:15:25.226391 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:15:25.226359 5378 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2zh4d\" (UniqueName: \"kubernetes.io/projected/d251ff08-e87c-4819-a8f9-4cc702bc75ac-kube-api-access-2zh4d\") pod \"maas-api-e2e-inf-a-f0fe13-588d6cb567-29spw\" (UID: \"d251ff08-e87c-4819-a8f9-4cc702bc75ac\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-f0fe13-588d6cb567-29spw" Oct 30 21:15:25.353690 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:15:25.353656 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:15:25.664758 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:15:25.664714 5378 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d251ff08-e87c-4819-a8f9-4cc702bc75ac-maas-api-tls\") pod \"maas-api-e2e-inf-a-f0fe13-588d6cb567-29spw\" (UID: \"d251ff08-e87c-4819-a8f9-4cc702bc75ac\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-f0fe13-588d6cb567-29spw" Oct 30 21:15:25.667163 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:15:25.667134 5378 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d251ff08-e87c-4819-a8f9-4cc702bc75ac-maas-api-tls\") pod \"maas-api-e2e-inf-a-f0fe13-588d6cb567-29spw\" (UID: \"d251ff08-e87c-4819-a8f9-4cc702bc75ac\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-f0fe13-588d6cb567-29spw" Oct 30 21:15:25.776025 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:15:25.775989 5378 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-f0fe13-588d6cb567-29spw" Oct 30 21:15:25.975713 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:15:25.975619 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:15:26.162783 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:15:26.162745 5378 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-f0fe13-588d6cb567-29spw"] Oct 30 21:15:26.166908 ip-10-0-138-149 kubenswrapper[5378]: W1030 21:15:26.166875 5378 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd251ff08_e87c_4819_a8f9_4cc702bc75ac.slice/crio-3172122db6fc756718aff5d5d5d5b43b516cbf92b0c5c5f3bc14854c3e7201d2 WatchSource:0}: Error finding container 3172122db6fc756718aff5d5d5d5b43b516cbf92b0c5c5f3bc14854c3e7201d2: Status 404 returned error can't find the container with id 3172122db6fc756718aff5d5d5d5b43b516cbf92b0c5c5f3bc14854c3e7201d2 Oct 30 21:15:26.550445 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:15:26.550409 5378 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-f0fe13-588d6cb567-29spw" event={"ID":"d251ff08-e87c-4819-a8f9-4cc702bc75ac","Type":"ContainerStarted","Data":"3172122db6fc756718aff5d5d5d5b43b516cbf92b0c5c5f3bc14854c3e7201d2"} Oct 30 21:15:27.355692 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:15:27.355653 5378 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-2f9eff-openshift-default-76f4d54d8c-mlhvh"] Oct 30 21:15:27.356381 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:15:27.356361 5378 cpu_manager.go:414] "containerMap: removing container" podUID="789b0c2c-6150-4867-9c6b-8e7fffa5e17f" containerName="cleanup" Oct 30 21:15:27.356460 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:15:27.356384 5378 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="789b0c2c-6150-4867-9c6b-8e7fffa5e17f" containerName="cleanup" Oct 30 21:15:27.356460 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:15:27.356418 5378 cpu_manager.go:414] "containerMap: removing container" podUID="cbd70c63-4eab-49cc-9b10-f039320c1fb0" containerName="cleanup" Oct 30 21:15:27.356460 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:15:27.356424 5378 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cbd70c63-4eab-49cc-9b10-f039320c1fb0" containerName="cleanup" Oct 30 21:15:27.356460 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:15:27.356444 5378 cpu_manager.go:414] "containerMap: removing container" podUID="03043216-5312-45de-9d5c-63b2cc34b320" containerName="cleanup" Oct 30 21:15:27.356460 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:15:27.356449 5378 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="03043216-5312-45de-9d5c-63b2cc34b320" containerName="cleanup" Oct 30 21:15:27.356662 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:15:27.356464 5378 cpu_manager.go:414] "containerMap: removing container" podUID="503a9e15-f5df-4fde-8ca1-ea1946fe581b" containerName="cleanup" Oct 30 21:15:27.356662 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:15:27.356469 5378 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="503a9e15-f5df-4fde-8ca1-ea1946fe581b" containerName="cleanup" Oct 30 21:15:27.356662 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:15:27.356573 5378 memory_manager.go:365] "RemoveStaleState removing state" podUID="cbd70c63-4eab-49cc-9b10-f039320c1fb0" containerName="cleanup" Oct 30 21:15:27.356662 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:15:27.356590 5378 memory_manager.go:365] "RemoveStaleState removing state" podUID="03043216-5312-45de-9d5c-63b2cc34b320" containerName="cleanup" Oct 30 21:15:27.356662 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:15:27.356600 5378 memory_manager.go:365] "RemoveStaleState removing state" podUID="789b0c2c-6150-4867-9c6b-8e7fffa5e17f" containerName="cleanup" Oct 30 21:15:27.356662 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:15:27.356606 5378 memory_manager.go:365] "RemoveStaleState removing state" podUID="503a9e15-f5df-4fde-8ca1-ea1946fe581b" containerName="cleanup" Oct 30 21:15:27.359228 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:15:27.359213 5378 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-2f9eff-openshift-default-76f4d54d8c-mlhvh" Oct 30 21:15:27.391529 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:15:27.391487 5378 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bebe8fda-f6a0-4527-9798-2cb425757af6-workload-certs\") pod \"e2e-inf-a-2f9eff-openshift-default-76f4d54d8c-mlhvh\" (UID: \"bebe8fda-f6a0-4527-9798-2cb425757af6\") " pod="openshift-ingress/e2e-inf-a-2f9eff-openshift-default-76f4d54d8c-mlhvh" Oct 30 21:15:27.391744 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:15:27.391543 5378 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bebe8fda-f6a0-4527-9798-2cb425757af6-workload-socket\") pod \"e2e-inf-a-2f9eff-openshift-default-76f4d54d8c-mlhvh\" (UID: \"bebe8fda-f6a0-4527-9798-2cb425757af6\") " pod="openshift-ingress/e2e-inf-a-2f9eff-openshift-default-76f4d54d8c-mlhvh" Oct 30 21:15:27.391744 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:15:27.391668 5378 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bebe8fda-f6a0-4527-9798-2cb425757af6-istiod-ca-cert\") pod \"e2e-inf-a-2f9eff-openshift-default-76f4d54d8c-mlhvh\" (UID: \"bebe8fda-f6a0-4527-9798-2cb425757af6\") " pod="openshift-ingress/e2e-inf-a-2f9eff-openshift-default-76f4d54d8c-mlhvh" Oct 30 21:15:27.391744 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:15:27.391706 5378 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bebe8fda-f6a0-4527-9798-2cb425757af6-istio-data\") pod \"e2e-inf-a-2f9eff-openshift-default-76f4d54d8c-mlhvh\" (UID: \"bebe8fda-f6a0-4527-9798-2cb425757af6\") " pod="openshift-ingress/e2e-inf-a-2f9eff-openshift-default-76f4d54d8c-mlhvh" Oct 30 21:15:27.391744 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:15:27.391737 5378 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bebe8fda-f6a0-4527-9798-2cb425757af6-credential-socket\") pod \"e2e-inf-a-2f9eff-openshift-default-76f4d54d8c-mlhvh\" (UID: \"bebe8fda-f6a0-4527-9798-2cb425757af6\") " pod="openshift-ingress/e2e-inf-a-2f9eff-openshift-default-76f4d54d8c-mlhvh" Oct 30 21:15:27.391995 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:15:27.391820 5378 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qllvm\" (UniqueName: \"kubernetes.io/projected/bebe8fda-f6a0-4527-9798-2cb425757af6-kube-api-access-qllvm\") pod \"e2e-inf-a-2f9eff-openshift-default-76f4d54d8c-mlhvh\" (UID: \"bebe8fda-f6a0-4527-9798-2cb425757af6\") " pod="openshift-ingress/e2e-inf-a-2f9eff-openshift-default-76f4d54d8c-mlhvh" Oct 30 21:15:27.391995 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:15:27.391884 5378 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bebe8fda-f6a0-4527-9798-2cb425757af6-istio-envoy\") pod \"e2e-inf-a-2f9eff-openshift-default-76f4d54d8c-mlhvh\" (UID: \"bebe8fda-f6a0-4527-9798-2cb425757af6\") " pod="openshift-ingress/e2e-inf-a-2f9eff-openshift-default-76f4d54d8c-mlhvh" Oct 30 21:15:27.391995 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:15:27.391907 5378 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bebe8fda-f6a0-4527-9798-2cb425757af6-istio-token\") pod \"e2e-inf-a-2f9eff-openshift-default-76f4d54d8c-mlhvh\" (UID: \"bebe8fda-f6a0-4527-9798-2cb425757af6\") " pod="openshift-ingress/e2e-inf-a-2f9eff-openshift-default-76f4d54d8c-mlhvh" Oct 30 21:15:27.391995 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:15:27.391936 5378 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bebe8fda-f6a0-4527-9798-2cb425757af6-istio-podinfo\") pod \"e2e-inf-a-2f9eff-openshift-default-76f4d54d8c-mlhvh\" (UID: \"bebe8fda-f6a0-4527-9798-2cb425757af6\") " pod="openshift-ingress/e2e-inf-a-2f9eff-openshift-default-76f4d54d8c-mlhvh" Oct 30 21:15:27.441705 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:15:27.441653 5378 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-2f9eff-openshift-default-dockercfg-zcthz\"" Oct 30 21:15:27.467244 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:15:27.467208 5378 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-2f9eff-openshift-default-76f4d54d8c-mlhvh"] Oct 30 21:15:27.493177 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:15:27.493126 5378 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bebe8fda-f6a0-4527-9798-2cb425757af6-istiod-ca-cert\") pod \"e2e-inf-a-2f9eff-openshift-default-76f4d54d8c-mlhvh\" (UID: \"bebe8fda-f6a0-4527-9798-2cb425757af6\") " pod="openshift-ingress/e2e-inf-a-2f9eff-openshift-default-76f4d54d8c-mlhvh" Oct 30 21:15:27.493399 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:15:27.493221 5378 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bebe8fda-f6a0-4527-9798-2cb425757af6-istio-data\") pod \"e2e-inf-a-2f9eff-openshift-default-76f4d54d8c-mlhvh\" (UID: \"bebe8fda-f6a0-4527-9798-2cb425757af6\") " pod="openshift-ingress/e2e-inf-a-2f9eff-openshift-default-76f4d54d8c-mlhvh" Oct 30 21:15:27.493399 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:15:27.493272 5378 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bebe8fda-f6a0-4527-9798-2cb425757af6-credential-socket\") pod \"e2e-inf-a-2f9eff-openshift-default-76f4d54d8c-mlhvh\" (UID: \"bebe8fda-f6a0-4527-9798-2cb425757af6\") " pod="openshift-ingress/e2e-inf-a-2f9eff-openshift-default-76f4d54d8c-mlhvh" Oct 30 21:15:27.493399 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:15:27.493306 5378 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qllvm\" (UniqueName: \"kubernetes.io/projected/bebe8fda-f6a0-4527-9798-2cb425757af6-kube-api-access-qllvm\") pod \"e2e-inf-a-2f9eff-openshift-default-76f4d54d8c-mlhvh\" (UID: \"bebe8fda-f6a0-4527-9798-2cb425757af6\") " pod="openshift-ingress/e2e-inf-a-2f9eff-openshift-default-76f4d54d8c-mlhvh" Oct 30 21:15:27.493399 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:15:27.493394 5378 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bebe8fda-f6a0-4527-9798-2cb425757af6-istio-envoy\") pod \"e2e-inf-a-2f9eff-openshift-default-76f4d54d8c-mlhvh\" (UID: \"bebe8fda-f6a0-4527-9798-2cb425757af6\") " pod="openshift-ingress/e2e-inf-a-2f9eff-openshift-default-76f4d54d8c-mlhvh" Oct 30 21:15:27.493783 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:15:27.493757 5378 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bebe8fda-f6a0-4527-9798-2cb425757af6-credential-socket\") pod \"e2e-inf-a-2f9eff-openshift-default-76f4d54d8c-mlhvh\" (UID: \"bebe8fda-f6a0-4527-9798-2cb425757af6\") " pod="openshift-ingress/e2e-inf-a-2f9eff-openshift-default-76f4d54d8c-mlhvh" Oct 30 21:15:27.493884 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:15:27.493860 5378 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bebe8fda-f6a0-4527-9798-2cb425757af6-istio-token\") pod \"e2e-inf-a-2f9eff-openshift-default-76f4d54d8c-mlhvh\" (UID: \"bebe8fda-f6a0-4527-9798-2cb425757af6\") " pod="openshift-ingress/e2e-inf-a-2f9eff-openshift-default-76f4d54d8c-mlhvh" Oct 30 21:15:27.493941 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:15:27.493923 5378 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bebe8fda-f6a0-4527-9798-2cb425757af6-istio-podinfo\") pod \"e2e-inf-a-2f9eff-openshift-default-76f4d54d8c-mlhvh\" (UID: \"bebe8fda-f6a0-4527-9798-2cb425757af6\") " pod="openshift-ingress/e2e-inf-a-2f9eff-openshift-default-76f4d54d8c-mlhvh" Oct 30 21:15:27.494149 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:15:27.494127 5378 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bebe8fda-f6a0-4527-9798-2cb425757af6-workload-certs\") pod \"e2e-inf-a-2f9eff-openshift-default-76f4d54d8c-mlhvh\" (UID: \"bebe8fda-f6a0-4527-9798-2cb425757af6\") " pod="openshift-ingress/e2e-inf-a-2f9eff-openshift-default-76f4d54d8c-mlhvh" Oct 30 21:15:27.494420 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:15:27.494393 5378 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bebe8fda-f6a0-4527-9798-2cb425757af6-workload-socket\") pod \"e2e-inf-a-2f9eff-openshift-default-76f4d54d8c-mlhvh\" (UID: \"bebe8fda-f6a0-4527-9798-2cb425757af6\") " pod="openshift-ingress/e2e-inf-a-2f9eff-openshift-default-76f4d54d8c-mlhvh" Oct 30 21:15:27.494630 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:15:27.494604 5378 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bebe8fda-f6a0-4527-9798-2cb425757af6-workload-certs\") pod \"e2e-inf-a-2f9eff-openshift-default-76f4d54d8c-mlhvh\" (UID: \"bebe8fda-f6a0-4527-9798-2cb425757af6\") " pod="openshift-ingress/e2e-inf-a-2f9eff-openshift-default-76f4d54d8c-mlhvh" Oct 30 21:15:27.494842 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:15:27.494816 5378 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bebe8fda-f6a0-4527-9798-2cb425757af6-workload-socket\") pod \"e2e-inf-a-2f9eff-openshift-default-76f4d54d8c-mlhvh\" (UID: \"bebe8fda-f6a0-4527-9798-2cb425757af6\") " pod="openshift-ingress/e2e-inf-a-2f9eff-openshift-default-76f4d54d8c-mlhvh" Oct 30 21:15:27.494963 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:15:27.494896 5378 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bebe8fda-f6a0-4527-9798-2cb425757af6-istio-data\") pod \"e2e-inf-a-2f9eff-openshift-default-76f4d54d8c-mlhvh\" (UID: \"bebe8fda-f6a0-4527-9798-2cb425757af6\") " pod="openshift-ingress/e2e-inf-a-2f9eff-openshift-default-76f4d54d8c-mlhvh" Oct 30 21:15:27.495445 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:15:27.495422 5378 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bebe8fda-f6a0-4527-9798-2cb425757af6-istiod-ca-cert\") pod \"e2e-inf-a-2f9eff-openshift-default-76f4d54d8c-mlhvh\" (UID: \"bebe8fda-f6a0-4527-9798-2cb425757af6\") " pod="openshift-ingress/e2e-inf-a-2f9eff-openshift-default-76f4d54d8c-mlhvh" Oct 30 21:15:27.496714 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:15:27.496682 5378 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bebe8fda-f6a0-4527-9798-2cb425757af6-istio-envoy\") pod \"e2e-inf-a-2f9eff-openshift-default-76f4d54d8c-mlhvh\" (UID: \"bebe8fda-f6a0-4527-9798-2cb425757af6\") " pod="openshift-ingress/e2e-inf-a-2f9eff-openshift-default-76f4d54d8c-mlhvh" Oct 30 21:15:27.497698 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:15:27.497675 5378 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bebe8fda-f6a0-4527-9798-2cb425757af6-istio-podinfo\") pod \"e2e-inf-a-2f9eff-openshift-default-76f4d54d8c-mlhvh\" (UID: \"bebe8fda-f6a0-4527-9798-2cb425757af6\") " pod="openshift-ingress/e2e-inf-a-2f9eff-openshift-default-76f4d54d8c-mlhvh" Oct 30 21:15:27.553687 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:15:27.553651 5378 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qllvm\" (UniqueName: \"kubernetes.io/projected/bebe8fda-f6a0-4527-9798-2cb425757af6-kube-api-access-qllvm\") pod \"e2e-inf-a-2f9eff-openshift-default-76f4d54d8c-mlhvh\" (UID: \"bebe8fda-f6a0-4527-9798-2cb425757af6\") " pod="openshift-ingress/e2e-inf-a-2f9eff-openshift-default-76f4d54d8c-mlhvh" Oct 30 21:15:27.553838 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:15:27.553707 5378 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bebe8fda-f6a0-4527-9798-2cb425757af6-istio-token\") pod \"e2e-inf-a-2f9eff-openshift-default-76f4d54d8c-mlhvh\" (UID: \"bebe8fda-f6a0-4527-9798-2cb425757af6\") " pod="openshift-ingress/e2e-inf-a-2f9eff-openshift-default-76f4d54d8c-mlhvh" Oct 30 21:15:27.559507 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:15:27.559476 5378 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-f0fe13-588d6cb567-29spw" event={"ID":"d251ff08-e87c-4819-a8f9-4cc702bc75ac","Type":"ContainerStarted","Data":"824f1f814cf968c0d7a520d0c995fc6c051f513e9e35cf17103d6ff8f7fadf20"} Oct 30 21:15:27.559692 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:15:27.559547 5378 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-f0fe13-588d6cb567-29spw" Oct 30 21:15:27.672749 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:15:27.672663 5378 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-2f9eff-openshift-default-76f4d54d8c-mlhvh" Oct 30 21:15:27.699692 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:15:27.699643 5378 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-f0fe13-588d6cb567-29spw" podStartSLOduration=2.919726271 podStartE2EDuration="3.69962866s" totalImagesPullingTime="779.902389ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:15:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:15:27.698216932 +0000 UTC m=+2218.126330850" watchObservedRunningTime="2026-10-30 21:15:27.69962866 +0000 UTC m=+2218.127742576" Oct 30 21:15:28.114154 ip-10-0-138-149 kubenswrapper[5378]: W1030 21:15:28.114121 5378 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbebe8fda_f6a0_4527_9798_2cb425757af6.slice/crio-90ef0fda3cc250bcc1c4712b68d2e1d719bbedc548f1a6af4f8e9a48b18835b9 WatchSource:0}: Error finding container 90ef0fda3cc250bcc1c4712b68d2e1d719bbedc548f1a6af4f8e9a48b18835b9: Status 404 returned error can't find the container with id 90ef0fda3cc250bcc1c4712b68d2e1d719bbedc548f1a6af4f8e9a48b18835b9 Oct 30 21:15:28.114267 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:15:28.114208 5378 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-2f9eff-openshift-default-76f4d54d8c-mlhvh"] Oct 30 21:15:28.117145 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:15:28.117111 5378 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 21:15:28.117237 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:15:28.117182 5378 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 21:15:28.117237 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:15:28.117215 5378 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 21:15:28.574035 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:15:28.573948 5378 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-2f9eff-openshift-default-76f4d54d8c-mlhvh" event={"ID":"bebe8fda-f6a0-4527-9798-2cb425757af6","Type":"ContainerStarted","Data":"d909f2351ef68294519a050233e5adfed15786ac68ccf9d69a2d16d465dac571"} Oct 30 21:15:28.574035 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:15:28.574011 5378 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-2f9eff-openshift-default-76f4d54d8c-mlhvh" event={"ID":"bebe8fda-f6a0-4527-9798-2cb425757af6","Type":"ContainerStarted","Data":"90ef0fda3cc250bcc1c4712b68d2e1d719bbedc548f1a6af4f8e9a48b18835b9"} Oct 30 21:15:28.673280 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:15:28.673239 5378 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-2f9eff-openshift-default-76f4d54d8c-mlhvh" Oct 30 21:15:28.698017 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:15:28.697948 5378 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-2f9eff-openshift-default-76f4d54d8c-mlhvh" podStartSLOduration=1.697925954 podStartE2EDuration="1.697925954s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:15:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:15:28.693755705 +0000 UTC m=+2219.121869626" watchObservedRunningTime="2026-10-30 21:15:28.697925954 +0000 UTC m=+2219.126039877" Oct 30 21:15:29.679764 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:15:29.679725 5378 patch_prober.go:29] interesting pod/e2e-inf-a-2f9eff-openshift-default-76f4d54d8c-mlhvh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.159:15021/healthz/ready\": dial tcp 10.133.0.159:15021: connect: connection refused" start-of-body= Oct 30 21:15:29.680411 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:15:29.679781 5378 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-2f9eff-openshift-default-76f4d54d8c-mlhvh" podUID="bebe8fda-f6a0-4527-9798-2cb425757af6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.159:15021/healthz/ready\": dial tcp 10.133.0.159:15021: connect: connection refused" Oct 30 21:15:30.678297 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:15:30.678101 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:15:30.678591 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:15:30.678528 5378 patch_prober.go:29] interesting pod/e2e-inf-a-2f9eff-openshift-default-76f4d54d8c-mlhvh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.159:15021/healthz/ready\": dial tcp 10.133.0.159:15021: connect: connection refused" start-of-body= Oct 30 21:15:30.678714 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:15:30.678641 5378 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-2f9eff-openshift-default-76f4d54d8c-mlhvh" podUID="bebe8fda-f6a0-4527-9798-2cb425757af6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.159:15021/healthz/ready\": dial tcp 10.133.0.159:15021: connect: connection refused" Oct 30 21:15:31.676168 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:15:31.673914 5378 patch_prober.go:29] interesting pod/e2e-inf-a-2f9eff-openshift-default-76f4d54d8c-mlhvh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.159:15021/healthz/ready\": dial tcp 10.133.0.159:15021: connect: connection refused" start-of-body= Oct 30 21:15:31.676168 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:15:31.673972 5378 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-2f9eff-openshift-default-76f4d54d8c-mlhvh" podUID="bebe8fda-f6a0-4527-9798-2cb425757af6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.159:15021/healthz/ready\": dial tcp 10.133.0.159:15021: connect: connection refused" Oct 30 21:15:32.675588 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:15:32.675131 5378 patch_prober.go:29] interesting pod/e2e-inf-a-2f9eff-openshift-default-76f4d54d8c-mlhvh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.159:15021/healthz/ready\": dial tcp 10.133.0.159:15021: connect: connection refused" start-of-body= Oct 30 21:15:32.675588 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:15:32.675200 5378 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-2f9eff-openshift-default-76f4d54d8c-mlhvh" podUID="bebe8fda-f6a0-4527-9798-2cb425757af6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.159:15021/healthz/ready\": dial tcp 10.133.0.159:15021: connect: connection refused" Oct 30 21:15:32.723989 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:15:32.723922 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:15:32.887363 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:15:32.887327 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:15:33.093463 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:15:33.093377 5378 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w6-7eaf74-openshift-default-69c8f48765-r2cmt"] Oct 30 21:15:33.106303 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:15:33.106269 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:15:33.142063 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:15:33.142025 5378 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w6-7eaf74-openshift-default-69c8f48765-tmsjl"] Oct 30 21:15:33.144599 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:15:33.144526 5378 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w6-7eaf74-openshift-default-69c8f48765-tmsjl" podUID="3559349f-6be4-4ef6-8061-59c7d1a8b678" containerName="istio-proxy" containerID="cri-o://549031df7c36d7fbdf19c5af343bfe734c786b67dcbc348285d854daaf3e1b52" gracePeriod=30 Oct 30 21:15:33.145620 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:15:33.145583 5378 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w6-7eaf74-openshift-default-69c8f48765-phhsn"] Oct 30 21:15:33.145831 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:15:33.145799 5378 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w6-7eaf74-openshift-default-69c8f48765-phhsn" podUID="4521c8f6-928b-47a2-8457-de63ee943c09" containerName="istio-proxy" containerID="cri-o://0d9e73bb769d3c124c9bbae906f5dad2f2c95bce22b8e82a51fb49142ce3aede" gracePeriod=30 Oct 30 21:15:33.585864 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:15:33.585654 5378 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-f0fe13-588d6cb567-29spw" Oct 30 21:15:33.612533 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:15:33.612494 5378 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w6-7eaf74-openshift-default-69c8f48765-r2cmt" podUID="1eb6161d-5e96-4dd5-bbe1-68300df5c425" containerName="istio-proxy" containerID="cri-o://353294d3f04e0c51f2a7fc80a1531f6733ae2e83281940b801e73fa00431c4df" gracePeriod=30 Oct 30 21:15:33.620819 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:15:33.620793 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:15:33.674104 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:15:33.674059 5378 patch_prober.go:29] interesting pod/e2e-inf-a-2f9eff-openshift-default-76f4d54d8c-mlhvh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.159:15021/healthz/ready\": dial tcp 10.133.0.159:15021: connect: connection refused" start-of-body= Oct 30 21:15:33.674473 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:15:33.674428 5378 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-2f9eff-openshift-default-76f4d54d8c-mlhvh" podUID="bebe8fda-f6a0-4527-9798-2cb425757af6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.159:15021/healthz/ready\": dial tcp 10.133.0.159:15021: connect: connection refused" Oct 30 21:15:33.798166 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:15:33.797877 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:15:34.240992 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:15:34.240957 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:15:34.678526 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:15:34.678495 5378 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-2f9eff-openshift-default-76f4d54d8c-mlhvh" Oct 30 21:15:34.679011 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:15:34.678993 5378 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-2f9eff-openshift-default-76f4d54d8c-mlhvh" Oct 30 21:15:34.679915 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:15:34.679896 5378 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-2f9eff-openshift-default-76f4d54d8c-mlhvh" Oct 30 21:15:35.562286 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:15:35.562250 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:15:35.765798 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:15:35.765764 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:15:37.219014 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:15:37.218980 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:15:37.611671 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:15:37.611626 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:15:37.749641 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:15:37.749343 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:15:38.274891 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:15:38.274854 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:15:38.371982 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:15:38.371956 5378 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w6-7eaf74-openshift-default-69c8f48765-tmsjl" Oct 30 21:15:38.419055 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:15:38.419003 5378 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w6-7eaf74-openshift-default-69c8f48765-phhsn" Oct 30 21:15:38.434760 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:15:38.434735 5378 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3559349f-6be4-4ef6-8061-59c7d1a8b678-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3559349f-6be4-4ef6-8061-59c7d1a8b678-istio-envoy\") pod \"3559349f-6be4-4ef6-8061-59c7d1a8b678\" (UID: \"3559349f-6be4-4ef6-8061-59c7d1a8b678\") " Oct 30 21:15:38.434875 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:15:38.434785 5378 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3559349f-6be4-4ef6-8061-59c7d1a8b678-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3559349f-6be4-4ef6-8061-59c7d1a8b678-credential-socket\") pod \"3559349f-6be4-4ef6-8061-59c7d1a8b678\" (UID: \"3559349f-6be4-4ef6-8061-59c7d1a8b678\") " Oct 30 21:15:38.435032 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:15:38.434897 5378 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3559349f-6be4-4ef6-8061-59c7d1a8b678-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3559349f-6be4-4ef6-8061-59c7d1a8b678-workload-certs\") pod \"3559349f-6be4-4ef6-8061-59c7d1a8b678\" (UID: \"3559349f-6be4-4ef6-8061-59c7d1a8b678\") " Oct 30 21:15:38.435157 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:15:38.435075 5378 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3559349f-6be4-4ef6-8061-59c7d1a8b678-credential-socket" pod "3559349f-6be4-4ef6-8061-59c7d1a8b678" (UID: "3559349f-6be4-4ef6-8061-59c7d1a8b678"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:15:38.435157 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:15:38.435097 5378 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3559349f-6be4-4ef6-8061-59c7d1a8b678-workload-certs" pod "3559349f-6be4-4ef6-8061-59c7d1a8b678" (UID: "3559349f-6be4-4ef6-8061-59c7d1a8b678"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:15:38.435282 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:15:38.434998 5378 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/3559349f-6be4-4ef6-8061-59c7d1a8b678-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3559349f-6be4-4ef6-8061-59c7d1a8b678-istio-podinfo\") pod \"3559349f-6be4-4ef6-8061-59c7d1a8b678\" (UID: \"3559349f-6be4-4ef6-8061-59c7d1a8b678\") " Oct 30 21:15:38.435282 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:15:38.435261 5378 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3559349f-6be4-4ef6-8061-59c7d1a8b678-kube-api-access-c7mmd\" (UniqueName: \"kubernetes.io/projected/3559349f-6be4-4ef6-8061-59c7d1a8b678-kube-api-access-c7mmd\") pod \"3559349f-6be4-4ef6-8061-59c7d1a8b678\" (UID: \"3559349f-6be4-4ef6-8061-59c7d1a8b678\") " Oct 30 21:15:38.435391 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:15:38.435332 5378 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3559349f-6be4-4ef6-8061-59c7d1a8b678-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3559349f-6be4-4ef6-8061-59c7d1a8b678-workload-socket\") pod \"3559349f-6be4-4ef6-8061-59c7d1a8b678\" (UID: \"3559349f-6be4-4ef6-8061-59c7d1a8b678\") " Oct 30 21:15:38.435445 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:15:38.435414 5378 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3559349f-6be4-4ef6-8061-59c7d1a8b678-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3559349f-6be4-4ef6-8061-59c7d1a8b678-istio-data\") pod \"3559349f-6be4-4ef6-8061-59c7d1a8b678\" (UID: \"3559349f-6be4-4ef6-8061-59c7d1a8b678\") " Oct 30 21:15:38.435514 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:15:38.435472 5378 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3559349f-6be4-4ef6-8061-59c7d1a8b678-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3559349f-6be4-4ef6-8061-59c7d1a8b678-istiod-ca-cert\") pod \"3559349f-6be4-4ef6-8061-59c7d1a8b678\" (UID: \"3559349f-6be4-4ef6-8061-59c7d1a8b678\") " Oct 30 21:15:38.435599 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:15:38.435509 5378 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3559349f-6be4-4ef6-8061-59c7d1a8b678-istio-token\" (UniqueName: \"kubernetes.io/projected/3559349f-6be4-4ef6-8061-59c7d1a8b678-istio-token\") pod \"3559349f-6be4-4ef6-8061-59c7d1a8b678\" (UID: \"3559349f-6be4-4ef6-8061-59c7d1a8b678\") " Oct 30 21:15:38.436055 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:15:38.436028 5378 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3559349f-6be4-4ef6-8061-59c7d1a8b678-istio-data" pod "3559349f-6be4-4ef6-8061-59c7d1a8b678" (UID: "3559349f-6be4-4ef6-8061-59c7d1a8b678"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:15:38.436201 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:15:38.436106 5378 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3559349f-6be4-4ef6-8061-59c7d1a8b678-credential-socket\") on node \"ip-10-0-138-149.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:15:38.436201 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:15:38.436129 5378 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3559349f-6be4-4ef6-8061-59c7d1a8b678-workload-certs\") on node \"ip-10-0-138-149.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:15:38.436476 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:15:38.436457 5378 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3559349f-6be4-4ef6-8061-59c7d1a8b678-istiod-ca-cert" pod "3559349f-6be4-4ef6-8061-59c7d1a8b678" (UID: "3559349f-6be4-4ef6-8061-59c7d1a8b678"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:15:38.436556 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:15:38.436518 5378 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3559349f-6be4-4ef6-8061-59c7d1a8b678-workload-socket" pod "3559349f-6be4-4ef6-8061-59c7d1a8b678" (UID: "3559349f-6be4-4ef6-8061-59c7d1a8b678"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:15:38.437429 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:15:38.437398 5378 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3559349f-6be4-4ef6-8061-59c7d1a8b678-istio-envoy" pod "3559349f-6be4-4ef6-8061-59c7d1a8b678" (UID: "3559349f-6be4-4ef6-8061-59c7d1a8b678"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:15:38.437535 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:15:38.437445 5378 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/3559349f-6be4-4ef6-8061-59c7d1a8b678-istio-podinfo" pod "3559349f-6be4-4ef6-8061-59c7d1a8b678" (UID: "3559349f-6be4-4ef6-8061-59c7d1a8b678"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 21:15:38.437774 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:15:38.437758 5378 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3559349f-6be4-4ef6-8061-59c7d1a8b678-kube-api-access-c7mmd" pod "3559349f-6be4-4ef6-8061-59c7d1a8b678" (UID: "3559349f-6be4-4ef6-8061-59c7d1a8b678"). InnerVolumeSpecName "kube-api-access-c7mmd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:15:38.438195 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:15:38.438176 5378 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3559349f-6be4-4ef6-8061-59c7d1a8b678-istio-token" pod "3559349f-6be4-4ef6-8061-59c7d1a8b678" (UID: "3559349f-6be4-4ef6-8061-59c7d1a8b678"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:15:38.536691 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:15:38.536663 5378 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4521c8f6-928b-47a2-8457-de63ee943c09-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4521c8f6-928b-47a2-8457-de63ee943c09-istio-envoy\") pod \"4521c8f6-928b-47a2-8457-de63ee943c09\" (UID: \"4521c8f6-928b-47a2-8457-de63ee943c09\") " Oct 30 21:15:38.536878 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:15:38.536699 5378 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4521c8f6-928b-47a2-8457-de63ee943c09-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4521c8f6-928b-47a2-8457-de63ee943c09-workload-certs\") pod \"4521c8f6-928b-47a2-8457-de63ee943c09\" (UID: \"4521c8f6-928b-47a2-8457-de63ee943c09\") " Oct 30 21:15:38.536878 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:15:38.536723 5378 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4521c8f6-928b-47a2-8457-de63ee943c09-kube-api-access-vwffv\" (UniqueName: \"kubernetes.io/projected/4521c8f6-928b-47a2-8457-de63ee943c09-kube-api-access-vwffv\") pod \"4521c8f6-928b-47a2-8457-de63ee943c09\" (UID: \"4521c8f6-928b-47a2-8457-de63ee943c09\") " Oct 30 21:15:38.536878 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:15:38.536784 5378 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4521c8f6-928b-47a2-8457-de63ee943c09-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4521c8f6-928b-47a2-8457-de63ee943c09-istiod-ca-cert\") pod \"4521c8f6-928b-47a2-8457-de63ee943c09\" (UID: \"4521c8f6-928b-47a2-8457-de63ee943c09\") " Oct 30 21:15:38.536878 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:15:38.536839 5378 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4521c8f6-928b-47a2-8457-de63ee943c09-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4521c8f6-928b-47a2-8457-de63ee943c09-credential-socket\") pod \"4521c8f6-928b-47a2-8457-de63ee943c09\" (UID: \"4521c8f6-928b-47a2-8457-de63ee943c09\") " Oct 30 21:15:38.537091 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:15:38.536876 5378 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4521c8f6-928b-47a2-8457-de63ee943c09-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4521c8f6-928b-47a2-8457-de63ee943c09-workload-socket\") pod \"4521c8f6-928b-47a2-8457-de63ee943c09\" (UID: \"4521c8f6-928b-47a2-8457-de63ee943c09\") " Oct 30 21:15:38.537091 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:15:38.536925 5378 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4521c8f6-928b-47a2-8457-de63ee943c09-istio-token\" (UniqueName: \"kubernetes.io/projected/4521c8f6-928b-47a2-8457-de63ee943c09-istio-token\") pod \"4521c8f6-928b-47a2-8457-de63ee943c09\" (UID: \"4521c8f6-928b-47a2-8457-de63ee943c09\") " Oct 30 21:15:38.537091 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:15:38.536993 5378 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/4521c8f6-928b-47a2-8457-de63ee943c09-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4521c8f6-928b-47a2-8457-de63ee943c09-istio-podinfo\") pod \"4521c8f6-928b-47a2-8457-de63ee943c09\" (UID: \"4521c8f6-928b-47a2-8457-de63ee943c09\") " Oct 30 21:15:38.537091 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:15:38.537009 5378 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4521c8f6-928b-47a2-8457-de63ee943c09-workload-certs" pod "4521c8f6-928b-47a2-8457-de63ee943c09" (UID: "4521c8f6-928b-47a2-8457-de63ee943c09"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:15:38.537091 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:15:38.537034 5378 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4521c8f6-928b-47a2-8457-de63ee943c09-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4521c8f6-928b-47a2-8457-de63ee943c09-istio-data\") pod \"4521c8f6-928b-47a2-8457-de63ee943c09\" (UID: \"4521c8f6-928b-47a2-8457-de63ee943c09\") " Oct 30 21:15:38.537348 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:15:38.537197 5378 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4521c8f6-928b-47a2-8457-de63ee943c09-credential-socket" pod "4521c8f6-928b-47a2-8457-de63ee943c09" (UID: "4521c8f6-928b-47a2-8457-de63ee943c09"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:15:38.537348 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:15:38.537214 5378 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4521c8f6-928b-47a2-8457-de63ee943c09-istiod-ca-cert" pod "4521c8f6-928b-47a2-8457-de63ee943c09" (UID: "4521c8f6-928b-47a2-8457-de63ee943c09"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:15:38.537594 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:15:38.537528 5378 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4521c8f6-928b-47a2-8457-de63ee943c09-istio-data" pod "4521c8f6-928b-47a2-8457-de63ee943c09" (UID: "4521c8f6-928b-47a2-8457-de63ee943c09"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:15:38.537594 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:15:38.537547 5378 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4521c8f6-928b-47a2-8457-de63ee943c09-workload-socket" pod "4521c8f6-928b-47a2-8457-de63ee943c09" (UID: "4521c8f6-928b-47a2-8457-de63ee943c09"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:15:38.537829 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:15:38.537806 5378 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4521c8f6-928b-47a2-8457-de63ee943c09-workload-certs\") on node \"ip-10-0-138-149.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:15:38.537875 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:15:38.537836 5378 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4521c8f6-928b-47a2-8457-de63ee943c09-istiod-ca-cert\") on node \"ip-10-0-138-149.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:15:38.537875 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:15:38.537854 5378 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3559349f-6be4-4ef6-8061-59c7d1a8b678-istio-podinfo\") on node \"ip-10-0-138-149.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:15:38.537945 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:15:38.537870 5378 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c7mmd\" (UniqueName: \"kubernetes.io/projected/3559349f-6be4-4ef6-8061-59c7d1a8b678-kube-api-access-c7mmd\") on node \"ip-10-0-138-149.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:15:38.537945 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:15:38.537887 5378 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4521c8f6-928b-47a2-8457-de63ee943c09-credential-socket\") on node \"ip-10-0-138-149.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:15:38.537945 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:15:38.537901 5378 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4521c8f6-928b-47a2-8457-de63ee943c09-workload-socket\") on node \"ip-10-0-138-149.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:15:38.537945 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:15:38.537916 5378 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3559349f-6be4-4ef6-8061-59c7d1a8b678-workload-socket\") on node \"ip-10-0-138-149.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:15:38.537945 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:15:38.537930 5378 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3559349f-6be4-4ef6-8061-59c7d1a8b678-istio-data\") on node \"ip-10-0-138-149.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:15:38.538096 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:15:38.537944 5378 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4521c8f6-928b-47a2-8457-de63ee943c09-istio-data\") on node \"ip-10-0-138-149.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:15:38.538096 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:15:38.537960 5378 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3559349f-6be4-4ef6-8061-59c7d1a8b678-istiod-ca-cert\") on node \"ip-10-0-138-149.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:15:38.538096 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:15:38.537974 5378 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3559349f-6be4-4ef6-8061-59c7d1a8b678-istio-token\") on node \"ip-10-0-138-149.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:15:38.538096 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:15:38.537990 5378 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3559349f-6be4-4ef6-8061-59c7d1a8b678-istio-envoy\") on node \"ip-10-0-138-149.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:15:38.539150 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:15:38.539120 5378 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4521c8f6-928b-47a2-8457-de63ee943c09-istio-token" pod "4521c8f6-928b-47a2-8457-de63ee943c09" (UID: "4521c8f6-928b-47a2-8457-de63ee943c09"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:15:38.539233 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:15:38.539156 5378 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4521c8f6-928b-47a2-8457-de63ee943c09-kube-api-access-vwffv" pod "4521c8f6-928b-47a2-8457-de63ee943c09" (UID: "4521c8f6-928b-47a2-8457-de63ee943c09"). InnerVolumeSpecName "kube-api-access-vwffv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:15:38.539326 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:15:38.539305 5378 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4521c8f6-928b-47a2-8457-de63ee943c09-istio-envoy" pod "4521c8f6-928b-47a2-8457-de63ee943c09" (UID: "4521c8f6-928b-47a2-8457-de63ee943c09"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:15:38.539550 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:15:38.539530 5378 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/4521c8f6-928b-47a2-8457-de63ee943c09-istio-podinfo" pod "4521c8f6-928b-47a2-8457-de63ee943c09" (UID: "4521c8f6-928b-47a2-8457-de63ee943c09"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 21:15:38.638859 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:15:38.638832 5378 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4521c8f6-928b-47a2-8457-de63ee943c09-istio-envoy\") on node \"ip-10-0-138-149.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:15:38.638859 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:15:38.638860 5378 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vwffv\" (UniqueName: \"kubernetes.io/projected/4521c8f6-928b-47a2-8457-de63ee943c09-kube-api-access-vwffv\") on node \"ip-10-0-138-149.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:15:38.639011 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:15:38.638870 5378 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4521c8f6-928b-47a2-8457-de63ee943c09-istio-token\") on node \"ip-10-0-138-149.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:15:38.639011 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:15:38.638878 5378 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4521c8f6-928b-47a2-8457-de63ee943c09-istio-podinfo\") on node \"ip-10-0-138-149.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:15:38.652127 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:15:38.652098 5378 generic.go:356] "Generic (PLEG): container finished" podID="3559349f-6be4-4ef6-8061-59c7d1a8b678" containerID="549031df7c36d7fbdf19c5af343bfe734c786b67dcbc348285d854daaf3e1b52" exitCode=0 Oct 30 21:15:38.652240 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:15:38.652158 5378 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w6-7eaf74-openshift-default-69c8f48765-tmsjl" Oct 30 21:15:38.652240 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:15:38.652180 5378 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w6-7eaf74-openshift-default-69c8f48765-tmsjl" event={"ID":"3559349f-6be4-4ef6-8061-59c7d1a8b678","Type":"ContainerDied","Data":"549031df7c36d7fbdf19c5af343bfe734c786b67dcbc348285d854daaf3e1b52"} Oct 30 21:15:38.652240 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:15:38.652233 5378 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w6-7eaf74-openshift-default-69c8f48765-tmsjl" event={"ID":"3559349f-6be4-4ef6-8061-59c7d1a8b678","Type":"ContainerDied","Data":"eef112527c199e727a3c6d033c3386cd15026f35ca26e9c137c754e20da072d7"} Oct 30 21:15:38.652389 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:15:38.652261 5378 scope.go:122] "RemoveContainer" containerID="549031df7c36d7fbdf19c5af343bfe734c786b67dcbc348285d854daaf3e1b52" Oct 30 21:15:38.653880 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:15:38.653862 5378 generic.go:356] "Generic (PLEG): container finished" podID="4521c8f6-928b-47a2-8457-de63ee943c09" containerID="0d9e73bb769d3c124c9bbae906f5dad2f2c95bce22b8e82a51fb49142ce3aede" exitCode=0 Oct 30 21:15:38.653988 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:15:38.653888 5378 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w6-7eaf74-openshift-default-69c8f48765-phhsn" event={"ID":"4521c8f6-928b-47a2-8457-de63ee943c09","Type":"ContainerDied","Data":"0d9e73bb769d3c124c9bbae906f5dad2f2c95bce22b8e82a51fb49142ce3aede"} Oct 30 21:15:38.653988 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:15:38.653902 5378 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w6-7eaf74-openshift-default-69c8f48765-phhsn" Oct 30 21:15:38.653988 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:15:38.653917 5378 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w6-7eaf74-openshift-default-69c8f48765-phhsn" event={"ID":"4521c8f6-928b-47a2-8457-de63ee943c09","Type":"ContainerDied","Data":"768609ea3f54dd446802b5a8bbed89cdb068d346679eac4e4d45487239b9599c"} Oct 30 21:15:38.727893 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:15:38.727865 5378 scope.go:122] "RemoveContainer" containerID="7f6ec1fe4760ca648091b49d5fd6b282dd26bfb546798a789e4dc64db5bb8a6b" Oct 30 21:15:38.737901 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:15:38.737884 5378 scope.go:122] "RemoveContainer" containerID="549031df7c36d7fbdf19c5af343bfe734c786b67dcbc348285d854daaf3e1b52" Oct 30 21:15:38.738254 ip-10-0-138-149 kubenswrapper[5378]: E1030 21:15:38.738227 5378 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"549031df7c36d7fbdf19c5af343bfe734c786b67dcbc348285d854daaf3e1b52\": container with ID starting with 549031df7c36d7fbdf19c5af343bfe734c786b67dcbc348285d854daaf3e1b52 not found: ID does not exist" containerID="549031df7c36d7fbdf19c5af343bfe734c786b67dcbc348285d854daaf3e1b52" Oct 30 21:15:38.738363 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:15:38.738261 5378 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"549031df7c36d7fbdf19c5af343bfe734c786b67dcbc348285d854daaf3e1b52"} err="failed to get container status \"549031df7c36d7fbdf19c5af343bfe734c786b67dcbc348285d854daaf3e1b52\": rpc error: code = NotFound desc = could not find container \"549031df7c36d7fbdf19c5af343bfe734c786b67dcbc348285d854daaf3e1b52\": container with ID starting with 549031df7c36d7fbdf19c5af343bfe734c786b67dcbc348285d854daaf3e1b52 not found: ID does not exist" Oct 30 21:15:38.738363 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:15:38.738281 5378 scope.go:122] "RemoveContainer" containerID="7f6ec1fe4760ca648091b49d5fd6b282dd26bfb546798a789e4dc64db5bb8a6b" Oct 30 21:15:38.738649 ip-10-0-138-149 kubenswrapper[5378]: E1030 21:15:38.738619 5378 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7f6ec1fe4760ca648091b49d5fd6b282dd26bfb546798a789e4dc64db5bb8a6b\": container with ID starting with 7f6ec1fe4760ca648091b49d5fd6b282dd26bfb546798a789e4dc64db5bb8a6b not found: ID does not exist" containerID="7f6ec1fe4760ca648091b49d5fd6b282dd26bfb546798a789e4dc64db5bb8a6b" Oct 30 21:15:38.738727 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:15:38.738661 5378 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7f6ec1fe4760ca648091b49d5fd6b282dd26bfb546798a789e4dc64db5bb8a6b"} err="failed to get container status \"7f6ec1fe4760ca648091b49d5fd6b282dd26bfb546798a789e4dc64db5bb8a6b\": rpc error: code = NotFound desc = could not find container \"7f6ec1fe4760ca648091b49d5fd6b282dd26bfb546798a789e4dc64db5bb8a6b\": container with ID starting with 7f6ec1fe4760ca648091b49d5fd6b282dd26bfb546798a789e4dc64db5bb8a6b not found: ID does not exist" Oct 30 21:15:38.738727 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:15:38.738686 5378 scope.go:122] "RemoveContainer" containerID="0d9e73bb769d3c124c9bbae906f5dad2f2c95bce22b8e82a51fb49142ce3aede" Oct 30 21:15:38.748791 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:15:38.748769 5378 scope.go:122] "RemoveContainer" containerID="0d9e73bb769d3c124c9bbae906f5dad2f2c95bce22b8e82a51fb49142ce3aede" Oct 30 21:15:38.749110 ip-10-0-138-149 kubenswrapper[5378]: E1030 21:15:38.749076 5378 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0d9e73bb769d3c124c9bbae906f5dad2f2c95bce22b8e82a51fb49142ce3aede\": container with ID starting with 0d9e73bb769d3c124c9bbae906f5dad2f2c95bce22b8e82a51fb49142ce3aede not found: ID does not exist" containerID="0d9e73bb769d3c124c9bbae906f5dad2f2c95bce22b8e82a51fb49142ce3aede" Oct 30 21:15:38.749216 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:15:38.749124 5378 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0d9e73bb769d3c124c9bbae906f5dad2f2c95bce22b8e82a51fb49142ce3aede"} err="failed to get container status \"0d9e73bb769d3c124c9bbae906f5dad2f2c95bce22b8e82a51fb49142ce3aede\": rpc error: code = NotFound desc = could not find container \"0d9e73bb769d3c124c9bbae906f5dad2f2c95bce22b8e82a51fb49142ce3aede\": container with ID starting with 0d9e73bb769d3c124c9bbae906f5dad2f2c95bce22b8e82a51fb49142ce3aede not found: ID does not exist" Oct 30 21:15:38.809880 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:15:38.809850 5378 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w6-7eaf74-openshift-default-69c8f48765-tmsjl"] Oct 30 21:15:38.822300 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:15:38.822264 5378 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w6-7eaf74-openshift-default-69c8f48765-tmsjl"] Oct 30 21:15:38.906558 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:15:38.906531 5378 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w6-7eaf74-openshift-default-69c8f48765-r2cmt" Oct 30 21:15:38.936687 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:15:38.936625 5378 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w6-7eaf74-openshift-default-69c8f48765-phhsn"] Oct 30 21:15:38.941884 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:15:38.941855 5378 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1eb6161d-5e96-4dd5-bbe1-68300df5c425-istio-token\" (UniqueName: \"kubernetes.io/projected/1eb6161d-5e96-4dd5-bbe1-68300df5c425-istio-token\") pod \"1eb6161d-5e96-4dd5-bbe1-68300df5c425\" (UID: \"1eb6161d-5e96-4dd5-bbe1-68300df5c425\") " Oct 30 21:15:38.942042 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:15:38.941950 5378 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1eb6161d-5e96-4dd5-bbe1-68300df5c425-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1eb6161d-5e96-4dd5-bbe1-68300df5c425-istio-envoy\") pod \"1eb6161d-5e96-4dd5-bbe1-68300df5c425\" (UID: \"1eb6161d-5e96-4dd5-bbe1-68300df5c425\") " Oct 30 21:15:38.942042 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:15:38.942002 5378 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1eb6161d-5e96-4dd5-bbe1-68300df5c425-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1eb6161d-5e96-4dd5-bbe1-68300df5c425-credential-socket\") pod \"1eb6161d-5e96-4dd5-bbe1-68300df5c425\" (UID: \"1eb6161d-5e96-4dd5-bbe1-68300df5c425\") " Oct 30 21:15:38.942159 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:15:38.942048 5378 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/1eb6161d-5e96-4dd5-bbe1-68300df5c425-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1eb6161d-5e96-4dd5-bbe1-68300df5c425-istio-podinfo\") pod \"1eb6161d-5e96-4dd5-bbe1-68300df5c425\" (UID: \"1eb6161d-5e96-4dd5-bbe1-68300df5c425\") " Oct 30 21:15:38.942159 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:15:38.942075 5378 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1eb6161d-5e96-4dd5-bbe1-68300df5c425-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1eb6161d-5e96-4dd5-bbe1-68300df5c425-workload-certs\") pod \"1eb6161d-5e96-4dd5-bbe1-68300df5c425\" (UID: \"1eb6161d-5e96-4dd5-bbe1-68300df5c425\") " Oct 30 21:15:38.942159 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:15:38.942133 5378 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1eb6161d-5e96-4dd5-bbe1-68300df5c425-kube-api-access-wvvfh\" (UniqueName: \"kubernetes.io/projected/1eb6161d-5e96-4dd5-bbe1-68300df5c425-kube-api-access-wvvfh\") pod \"1eb6161d-5e96-4dd5-bbe1-68300df5c425\" (UID: \"1eb6161d-5e96-4dd5-bbe1-68300df5c425\") " Oct 30 21:15:38.942318 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:15:38.942192 5378 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1eb6161d-5e96-4dd5-bbe1-68300df5c425-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1eb6161d-5e96-4dd5-bbe1-68300df5c425-istio-data\") pod \"1eb6161d-5e96-4dd5-bbe1-68300df5c425\" (UID: \"1eb6161d-5e96-4dd5-bbe1-68300df5c425\") " Oct 30 21:15:38.942318 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:15:38.942224 5378 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1eb6161d-5e96-4dd5-bbe1-68300df5c425-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1eb6161d-5e96-4dd5-bbe1-68300df5c425-istiod-ca-cert\") pod \"1eb6161d-5e96-4dd5-bbe1-68300df5c425\" (UID: \"1eb6161d-5e96-4dd5-bbe1-68300df5c425\") " Oct 30 21:15:38.942318 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:15:38.942266 5378 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1eb6161d-5e96-4dd5-bbe1-68300df5c425-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1eb6161d-5e96-4dd5-bbe1-68300df5c425-workload-socket\") pod \"1eb6161d-5e96-4dd5-bbe1-68300df5c425\" (UID: \"1eb6161d-5e96-4dd5-bbe1-68300df5c425\") " Oct 30 21:15:38.942486 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:15:38.942411 5378 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1eb6161d-5e96-4dd5-bbe1-68300df5c425-workload-certs" pod "1eb6161d-5e96-4dd5-bbe1-68300df5c425" (UID: "1eb6161d-5e96-4dd5-bbe1-68300df5c425"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:15:38.942486 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:15:38.942447 5378 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1eb6161d-5e96-4dd5-bbe1-68300df5c425-istio-data" pod "1eb6161d-5e96-4dd5-bbe1-68300df5c425" (UID: "1eb6161d-5e96-4dd5-bbe1-68300df5c425"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:15:38.942706 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:15:38.942684 5378 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1eb6161d-5e96-4dd5-bbe1-68300df5c425-credential-socket" pod "1eb6161d-5e96-4dd5-bbe1-68300df5c425" (UID: "1eb6161d-5e96-4dd5-bbe1-68300df5c425"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:15:38.942845 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:15:38.942706 5378 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1eb6161d-5e96-4dd5-bbe1-68300df5c425-istiod-ca-cert" pod "1eb6161d-5e96-4dd5-bbe1-68300df5c425" (UID: "1eb6161d-5e96-4dd5-bbe1-68300df5c425"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:15:38.942987 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:15:38.942844 5378 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1eb6161d-5e96-4dd5-bbe1-68300df5c425-workload-socket" pod "1eb6161d-5e96-4dd5-bbe1-68300df5c425" (UID: "1eb6161d-5e96-4dd5-bbe1-68300df5c425"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:15:38.942987 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:15:38.942892 5378 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w6-7eaf74-openshift-default-69c8f48765-phhsn"] Oct 30 21:15:38.943838 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:15:38.943811 5378 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1eb6161d-5e96-4dd5-bbe1-68300df5c425-credential-socket\") on node \"ip-10-0-138-149.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:15:38.943965 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:15:38.943842 5378 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1eb6161d-5e96-4dd5-bbe1-68300df5c425-workload-certs\") on node \"ip-10-0-138-149.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:15:38.943965 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:15:38.943860 5378 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1eb6161d-5e96-4dd5-bbe1-68300df5c425-istio-data\") on node \"ip-10-0-138-149.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:15:38.943965 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:15:38.943875 5378 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1eb6161d-5e96-4dd5-bbe1-68300df5c425-istiod-ca-cert\") on node \"ip-10-0-138-149.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:15:38.943965 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:15:38.943889 5378 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1eb6161d-5e96-4dd5-bbe1-68300df5c425-workload-socket\") on node \"ip-10-0-138-149.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:15:38.944317 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:15:38.944293 5378 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1eb6161d-5e96-4dd5-bbe1-68300df5c425-istio-token" pod "1eb6161d-5e96-4dd5-bbe1-68300df5c425" (UID: "1eb6161d-5e96-4dd5-bbe1-68300df5c425"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:15:38.944437 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:15:38.944345 5378 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/1eb6161d-5e96-4dd5-bbe1-68300df5c425-istio-podinfo" pod "1eb6161d-5e96-4dd5-bbe1-68300df5c425" (UID: "1eb6161d-5e96-4dd5-bbe1-68300df5c425"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 21:15:38.944437 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:15:38.944393 5378 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1eb6161d-5e96-4dd5-bbe1-68300df5c425-istio-envoy" pod "1eb6161d-5e96-4dd5-bbe1-68300df5c425" (UID: "1eb6161d-5e96-4dd5-bbe1-68300df5c425"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:15:38.944600 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:15:38.944583 5378 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1eb6161d-5e96-4dd5-bbe1-68300df5c425-kube-api-access-wvvfh" pod "1eb6161d-5e96-4dd5-bbe1-68300df5c425" (UID: "1eb6161d-5e96-4dd5-bbe1-68300df5c425"). InnerVolumeSpecName "kube-api-access-wvvfh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:15:39.045494 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:15:39.045460 5378 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1eb6161d-5e96-4dd5-bbe1-68300df5c425-istio-token\") on node \"ip-10-0-138-149.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:15:39.045494 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:15:39.045488 5378 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1eb6161d-5e96-4dd5-bbe1-68300df5c425-istio-envoy\") on node \"ip-10-0-138-149.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:15:39.045704 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:15:39.045499 5378 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1eb6161d-5e96-4dd5-bbe1-68300df5c425-istio-podinfo\") on node \"ip-10-0-138-149.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:15:39.045704 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:15:39.045527 5378 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wvvfh\" (UniqueName: \"kubernetes.io/projected/1eb6161d-5e96-4dd5-bbe1-68300df5c425-kube-api-access-wvvfh\") on node \"ip-10-0-138-149.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:15:39.661656 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:15:39.661624 5378 generic.go:356] "Generic (PLEG): container finished" podID="1eb6161d-5e96-4dd5-bbe1-68300df5c425" containerID="353294d3f04e0c51f2a7fc80a1531f6733ae2e83281940b801e73fa00431c4df" exitCode=0 Oct 30 21:15:39.662058 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:15:39.661702 5378 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w6-7eaf74-openshift-default-69c8f48765-r2cmt" event={"ID":"1eb6161d-5e96-4dd5-bbe1-68300df5c425","Type":"ContainerDied","Data":"353294d3f04e0c51f2a7fc80a1531f6733ae2e83281940b801e73fa00431c4df"} Oct 30 21:15:39.662157 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:15:39.662135 5378 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w6-7eaf74-openshift-default-69c8f48765-r2cmt" Oct 30 21:15:39.662274 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:15:39.661747 5378 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w6-7eaf74-openshift-default-69c8f48765-r2cmt" event={"ID":"1eb6161d-5e96-4dd5-bbe1-68300df5c425","Type":"ContainerDied","Data":"36a78e7c35f2196cb956ce7ef0236bce7990c83a169cdc4d729130c95f149dba"} Oct 30 21:15:39.665629 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:15:39.661755 5378 scope.go:122] "RemoveContainer" containerID="353294d3f04e0c51f2a7fc80a1531f6733ae2e83281940b801e73fa00431c4df" Oct 30 21:15:39.686985 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:15:39.686964 5378 scope.go:122] "RemoveContainer" containerID="353294d3f04e0c51f2a7fc80a1531f6733ae2e83281940b801e73fa00431c4df" Oct 30 21:15:39.687282 ip-10-0-138-149 kubenswrapper[5378]: E1030 21:15:39.687261 5378 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"353294d3f04e0c51f2a7fc80a1531f6733ae2e83281940b801e73fa00431c4df\": container with ID starting with 353294d3f04e0c51f2a7fc80a1531f6733ae2e83281940b801e73fa00431c4df not found: ID does not exist" containerID="353294d3f04e0c51f2a7fc80a1531f6733ae2e83281940b801e73fa00431c4df" Oct 30 21:15:39.687339 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:15:39.687290 5378 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"353294d3f04e0c51f2a7fc80a1531f6733ae2e83281940b801e73fa00431c4df"} err="failed to get container status \"353294d3f04e0c51f2a7fc80a1531f6733ae2e83281940b801e73fa00431c4df\": rpc error: code = NotFound desc = could not find container \"353294d3f04e0c51f2a7fc80a1531f6733ae2e83281940b801e73fa00431c4df\": container with ID starting with 353294d3f04e0c51f2a7fc80a1531f6733ae2e83281940b801e73fa00431c4df not found: ID does not exist" Oct 30 21:15:39.744225 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:15:39.744199 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:15:39.744225 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:15:39.744213 5378 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3559349f-6be4-4ef6-8061-59c7d1a8b678" path="/var/lib/kubelet/pods/3559349f-6be4-4ef6-8061-59c7d1a8b678/volumes" Oct 30 21:15:39.744744 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:15:39.744729 5378 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4521c8f6-928b-47a2-8457-de63ee943c09" path="/var/lib/kubelet/pods/4521c8f6-928b-47a2-8457-de63ee943c09/volumes" Oct 30 21:15:39.835360 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:15:39.835331 5378 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w6-7eaf74-openshift-default-69c8f48765-r2cmt"] Oct 30 21:15:39.885650 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:15:39.885616 5378 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w6-7eaf74-openshift-default-69c8f48765-r2cmt"] Oct 30 21:15:39.996440 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:15:39.996363 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:15:40.184886 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:15:40.184852 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:15:41.743274 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:15:41.743242 5378 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1eb6161d-5e96-4dd5-bbe1-68300df5c425" path="/var/lib/kubelet/pods/1eb6161d-5e96-4dd5-bbe1-68300df5c425/volumes" Oct 30 21:15:43.482440 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:15:43.482405 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:15:43.748141 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:15:43.746479 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:15:43.905495 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:15:43.905434 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:15:45.144446 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:15:45.144411 5378 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-r9g59"] Oct 30 21:15:45.267597 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:15:45.267533 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:15:45.423633 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:15:45.423269 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:15:45.650226 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:15:45.650141 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:15:48.613297 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:15:48.613249 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:15:48.827866 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:15:48.827829 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:15:48.842521 ip-10-0-138-149 kubenswrapper[5378]: E1030 21:15:48.842201 5378 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q6mx7" containerName="extractor" Oct 30 21:15:53.124114 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:15:53.124047 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:15:54.543807 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:15:54.543762 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:15:54.792384 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:15:54.792343 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:15:54.934523 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:15:54.932543 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:15:55.477966 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:15:55.477926 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:15:59.990699 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:15:59.990664 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:16:02.062607 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:16:02.062550 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:16:02.278952 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:16:02.278916 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:16:02.492975 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:16:02.492939 5378 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-d061d0-openshift-default-86b87f495-k5sd4"] Oct 30 21:16:02.493174 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:16:02.493153 5378 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-d061d0-openshift-default-86b87f495-k5sd4" podUID="ddde4251-43ac-4def-b4d4-e1f9a2a7dc30" containerName="istio-proxy" containerID="cri-o://73fdb355e362e5e5332c460f6b2853ee26e3fcc20a164506e4e54a3aeff4d1c9" gracePeriod=30 Oct 30 21:16:02.546634 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:16:02.546595 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:16:04.979593 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:16:04.979536 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:16:05.423151 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:16:05.423108 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:16:05.576373 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:16:05.576335 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:16:06.288333 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:16:06.288294 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:16:06.594706 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:16:06.594612 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:16:07.796665 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:16:07.796641 5378 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-d061d0-openshift-default-86b87f495-k5sd4" Oct 30 21:16:07.926955 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:16:07.926916 5378 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/ddde4251-43ac-4def-b4d4-e1f9a2a7dc30-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ddde4251-43ac-4def-b4d4-e1f9a2a7dc30-istio-podinfo\") pod \"ddde4251-43ac-4def-b4d4-e1f9a2a7dc30\" (UID: \"ddde4251-43ac-4def-b4d4-e1f9a2a7dc30\") " Oct 30 21:16:07.927142 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:16:07.926969 5378 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ddde4251-43ac-4def-b4d4-e1f9a2a7dc30-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ddde4251-43ac-4def-b4d4-e1f9a2a7dc30-istio-data\") pod \"ddde4251-43ac-4def-b4d4-e1f9a2a7dc30\" (UID: \"ddde4251-43ac-4def-b4d4-e1f9a2a7dc30\") " Oct 30 21:16:07.927225 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:16:07.927207 5378 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ddde4251-43ac-4def-b4d4-e1f9a2a7dc30-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ddde4251-43ac-4def-b4d4-e1f9a2a7dc30-workload-certs\") pod \"ddde4251-43ac-4def-b4d4-e1f9a2a7dc30\" (UID: \"ddde4251-43ac-4def-b4d4-e1f9a2a7dc30\") " Oct 30 21:16:07.927289 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:16:07.927220 5378 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ddde4251-43ac-4def-b4d4-e1f9a2a7dc30-istio-data" pod "ddde4251-43ac-4def-b4d4-e1f9a2a7dc30" (UID: "ddde4251-43ac-4def-b4d4-e1f9a2a7dc30"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:16:07.927289 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:16:07.927249 5378 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ddde4251-43ac-4def-b4d4-e1f9a2a7dc30-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ddde4251-43ac-4def-b4d4-e1f9a2a7dc30-workload-socket\") pod \"ddde4251-43ac-4def-b4d4-e1f9a2a7dc30\" (UID: \"ddde4251-43ac-4def-b4d4-e1f9a2a7dc30\") " Oct 30 21:16:07.927410 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:16:07.927327 5378 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ddde4251-43ac-4def-b4d4-e1f9a2a7dc30-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ddde4251-43ac-4def-b4d4-e1f9a2a7dc30-istiod-ca-cert\") pod \"ddde4251-43ac-4def-b4d4-e1f9a2a7dc30\" (UID: \"ddde4251-43ac-4def-b4d4-e1f9a2a7dc30\") " Oct 30 21:16:07.927522 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:16:07.927502 5378 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ddde4251-43ac-4def-b4d4-e1f9a2a7dc30-istio-token\" (UniqueName: \"kubernetes.io/projected/ddde4251-43ac-4def-b4d4-e1f9a2a7dc30-istio-token\") pod \"ddde4251-43ac-4def-b4d4-e1f9a2a7dc30\" (UID: \"ddde4251-43ac-4def-b4d4-e1f9a2a7dc30\") " Oct 30 21:16:07.927619 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:16:07.927580 5378 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ddde4251-43ac-4def-b4d4-e1f9a2a7dc30-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ddde4251-43ac-4def-b4d4-e1f9a2a7dc30-istio-envoy\") pod \"ddde4251-43ac-4def-b4d4-e1f9a2a7dc30\" (UID: \"ddde4251-43ac-4def-b4d4-e1f9a2a7dc30\") " Oct 30 21:16:07.927619 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:16:07.927554 5378 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ddde4251-43ac-4def-b4d4-e1f9a2a7dc30-workload-certs" pod "ddde4251-43ac-4def-b4d4-e1f9a2a7dc30" (UID: "ddde4251-43ac-4def-b4d4-e1f9a2a7dc30"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:16:07.927737 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:16:07.927630 5378 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ddde4251-43ac-4def-b4d4-e1f9a2a7dc30-kube-api-access-s4f9m\" (UniqueName: \"kubernetes.io/projected/ddde4251-43ac-4def-b4d4-e1f9a2a7dc30-kube-api-access-s4f9m\") pod \"ddde4251-43ac-4def-b4d4-e1f9a2a7dc30\" (UID: \"ddde4251-43ac-4def-b4d4-e1f9a2a7dc30\") " Oct 30 21:16:07.927737 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:16:07.927671 5378 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ddde4251-43ac-4def-b4d4-e1f9a2a7dc30-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ddde4251-43ac-4def-b4d4-e1f9a2a7dc30-credential-socket\") pod \"ddde4251-43ac-4def-b4d4-e1f9a2a7dc30\" (UID: \"ddde4251-43ac-4def-b4d4-e1f9a2a7dc30\") " Oct 30 21:16:07.927914 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:16:07.927888 5378 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ddde4251-43ac-4def-b4d4-e1f9a2a7dc30-istiod-ca-cert" pod "ddde4251-43ac-4def-b4d4-e1f9a2a7dc30" (UID: "ddde4251-43ac-4def-b4d4-e1f9a2a7dc30"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:16:07.928150 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:16:07.928128 5378 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ddde4251-43ac-4def-b4d4-e1f9a2a7dc30-credential-socket" pod "ddde4251-43ac-4def-b4d4-e1f9a2a7dc30" (UID: "ddde4251-43ac-4def-b4d4-e1f9a2a7dc30"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:16:07.928529 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:16:07.928419 5378 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ddde4251-43ac-4def-b4d4-e1f9a2a7dc30-credential-socket\") on node \"ip-10-0-138-149.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:16:07.928529 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:16:07.928447 5378 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ddde4251-43ac-4def-b4d4-e1f9a2a7dc30-istio-data\") on node \"ip-10-0-138-149.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:16:07.928529 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:16:07.928492 5378 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ddde4251-43ac-4def-b4d4-e1f9a2a7dc30-workload-certs\") on node \"ip-10-0-138-149.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:16:07.928529 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:16:07.928507 5378 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ddde4251-43ac-4def-b4d4-e1f9a2a7dc30-istiod-ca-cert\") on node \"ip-10-0-138-149.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:16:07.928984 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:16:07.928953 5378 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ddde4251-43ac-4def-b4d4-e1f9a2a7dc30-workload-socket" pod "ddde4251-43ac-4def-b4d4-e1f9a2a7dc30" (UID: "ddde4251-43ac-4def-b4d4-e1f9a2a7dc30"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:16:07.930686 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:16:07.930661 5378 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ddde4251-43ac-4def-b4d4-e1f9a2a7dc30-istio-envoy" pod "ddde4251-43ac-4def-b4d4-e1f9a2a7dc30" (UID: "ddde4251-43ac-4def-b4d4-e1f9a2a7dc30"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:16:07.930686 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:16:07.930676 5378 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ddde4251-43ac-4def-b4d4-e1f9a2a7dc30-istio-podinfo" pod "ddde4251-43ac-4def-b4d4-e1f9a2a7dc30" (UID: "ddde4251-43ac-4def-b4d4-e1f9a2a7dc30"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 21:16:07.931211 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:16:07.931186 5378 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ddde4251-43ac-4def-b4d4-e1f9a2a7dc30-kube-api-access-s4f9m" pod "ddde4251-43ac-4def-b4d4-e1f9a2a7dc30" (UID: "ddde4251-43ac-4def-b4d4-e1f9a2a7dc30"). InnerVolumeSpecName "kube-api-access-s4f9m". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:16:07.933552 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:16:07.933525 5378 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ddde4251-43ac-4def-b4d4-e1f9a2a7dc30-istio-token" pod "ddde4251-43ac-4def-b4d4-e1f9a2a7dc30" (UID: "ddde4251-43ac-4def-b4d4-e1f9a2a7dc30"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:16:07.937890 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:16:07.937861 5378 generic.go:356] "Generic (PLEG): container finished" podID="ddde4251-43ac-4def-b4d4-e1f9a2a7dc30" containerID="73fdb355e362e5e5332c460f6b2853ee26e3fcc20a164506e4e54a3aeff4d1c9" exitCode=0 Oct 30 21:16:07.938068 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:16:07.938047 5378 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-d061d0-openshift-default-86b87f495-k5sd4" Oct 30 21:16:07.938145 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:16:07.938074 5378 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-d061d0-openshift-default-86b87f495-k5sd4" event={"ID":"ddde4251-43ac-4def-b4d4-e1f9a2a7dc30","Type":"ContainerDied","Data":"73fdb355e362e5e5332c460f6b2853ee26e3fcc20a164506e4e54a3aeff4d1c9"} Oct 30 21:16:07.938145 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:16:07.938128 5378 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-d061d0-openshift-default-86b87f495-k5sd4" event={"ID":"ddde4251-43ac-4def-b4d4-e1f9a2a7dc30","Type":"ContainerDied","Data":"d2389f0c98bf4ee678373dab6e5727803eb94c352fc117ba4cc09dccd021b17d"} Oct 30 21:16:07.938262 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:16:07.938160 5378 scope.go:122] "RemoveContainer" containerID="73fdb355e362e5e5332c460f6b2853ee26e3fcc20a164506e4e54a3aeff4d1c9" Oct 30 21:16:07.965336 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:16:07.965296 5378 scope.go:122] "RemoveContainer" containerID="73fdb355e362e5e5332c460f6b2853ee26e3fcc20a164506e4e54a3aeff4d1c9" Oct 30 21:16:07.965671 ip-10-0-138-149 kubenswrapper[5378]: E1030 21:16:07.965652 5378 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"73fdb355e362e5e5332c460f6b2853ee26e3fcc20a164506e4e54a3aeff4d1c9\": container with ID starting with 73fdb355e362e5e5332c460f6b2853ee26e3fcc20a164506e4e54a3aeff4d1c9 not found: ID does not exist" containerID="73fdb355e362e5e5332c460f6b2853ee26e3fcc20a164506e4e54a3aeff4d1c9" Oct 30 21:16:07.965761 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:16:07.965681 5378 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"73fdb355e362e5e5332c460f6b2853ee26e3fcc20a164506e4e54a3aeff4d1c9"} err="failed to get container status \"73fdb355e362e5e5332c460f6b2853ee26e3fcc20a164506e4e54a3aeff4d1c9\": rpc error: code = NotFound desc = could not find container \"73fdb355e362e5e5332c460f6b2853ee26e3fcc20a164506e4e54a3aeff4d1c9\": container with ID starting with 73fdb355e362e5e5332c460f6b2853ee26e3fcc20a164506e4e54a3aeff4d1c9 not found: ID does not exist" Oct 30 21:16:08.029439 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:16:08.029356 5378 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ddde4251-43ac-4def-b4d4-e1f9a2a7dc30-istio-envoy\") on node \"ip-10-0-138-149.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:16:08.029439 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:16:08.029386 5378 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s4f9m\" (UniqueName: \"kubernetes.io/projected/ddde4251-43ac-4def-b4d4-e1f9a2a7dc30-kube-api-access-s4f9m\") on node \"ip-10-0-138-149.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:16:08.029439 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:16:08.029396 5378 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ddde4251-43ac-4def-b4d4-e1f9a2a7dc30-istio-podinfo\") on node \"ip-10-0-138-149.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:16:08.029439 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:16:08.029406 5378 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ddde4251-43ac-4def-b4d4-e1f9a2a7dc30-workload-socket\") on node \"ip-10-0-138-149.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:16:08.029439 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:16:08.029415 5378 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ddde4251-43ac-4def-b4d4-e1f9a2a7dc30-istio-token\") on node \"ip-10-0-138-149.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:16:08.097971 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:16:08.097934 5378 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-d061d0-openshift-default-86b87f495-k5sd4"] Oct 30 21:16:08.103103 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:16:08.103064 5378 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-d061d0-openshift-default-86b87f495-k5sd4"] Oct 30 21:16:09.744050 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:16:09.743975 5378 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ddde4251-43ac-4def-b4d4-e1f9a2a7dc30" path="/var/lib/kubelet/pods/ddde4251-43ac-4def-b4d4-e1f9a2a7dc30/volumes" Oct 30 21:16:10.310017 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:16:10.309962 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:16:10.518209 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:16:10.518173 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:16:10.749466 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:16:10.749435 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:16:11.146707 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:16:11.146678 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:16:13.563774 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:16:13.563737 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:16:13.783594 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:16:13.781476 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:16:16.688250 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:16:16.688216 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:16:18.836684 ip-10-0-138-149 kubenswrapper[5378]: E1030 21:16:18.836648 5378 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q6mx7" containerName="extractor" Oct 30 21:16:21.768593 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:16:21.768533 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:16:24.519065 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:16:24.519031 5378 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-r9g59"] Oct 30 21:16:25.919654 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:16:25.919606 5378 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-390b71-64988c8666-xfnmq"] Oct 30 21:16:25.920191 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:16:25.919896 5378 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-390b71-64988c8666-xfnmq" podUID="1a34eaf4-4b44-4e51-b497-eada6dd5a4fd" containerName="maas-api" containerID="cri-o://02e66a089d3895d4ae17d332127b2a1c579141bb1f755ea4ef8c3728a36e2cf6" gracePeriod=30 Oct 30 21:16:26.046094 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:16:26.046057 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:16:26.071611 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:16:26.071581 5378 generic.go:356] "Generic (PLEG): container finished" podID="1a34eaf4-4b44-4e51-b497-eada6dd5a4fd" containerID="02e66a089d3895d4ae17d332127b2a1c579141bb1f755ea4ef8c3728a36e2cf6" exitCode=0 Oct 30 21:16:26.071984 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:16:26.071959 5378 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-390b71-64988c8666-xfnmq" event={"ID":"1a34eaf4-4b44-4e51-b497-eada6dd5a4fd","Type":"ContainerDied","Data":"02e66a089d3895d4ae17d332127b2a1c579141bb1f755ea4ef8c3728a36e2cf6"} Oct 30 21:16:26.460820 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:16:26.460744 5378 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-390b71-64988c8666-xfnmq" Oct 30 21:16:26.544969 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:16:26.543024 5378 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1a34eaf4-4b44-4e51-b497-eada6dd5a4fd-kube-api-access-skhkx\" (UniqueName: \"kubernetes.io/projected/1a34eaf4-4b44-4e51-b497-eada6dd5a4fd-kube-api-access-skhkx\") pod \"1a34eaf4-4b44-4e51-b497-eada6dd5a4fd\" (UID: \"1a34eaf4-4b44-4e51-b497-eada6dd5a4fd\") " Oct 30 21:16:26.544969 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:16:26.543167 5378 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1a34eaf4-4b44-4e51-b497-eada6dd5a4fd-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1a34eaf4-4b44-4e51-b497-eada6dd5a4fd-maas-api-tls\") pod \"1a34eaf4-4b44-4e51-b497-eada6dd5a4fd\" (UID: \"1a34eaf4-4b44-4e51-b497-eada6dd5a4fd\") " Oct 30 21:16:26.544969 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:16:26.543386 5378 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1a34eaf4-4b44-4e51-b497-eada6dd5a4fd-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1a34eaf4-4b44-4e51-b497-eada6dd5a4fd-openshift-service-ca\") pod \"1a34eaf4-4b44-4e51-b497-eada6dd5a4fd\" (UID: \"1a34eaf4-4b44-4e51-b497-eada6dd5a4fd\") " Oct 30 21:16:26.544969 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:16:26.543441 5378 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1a34eaf4-4b44-4e51-b497-eada6dd5a4fd-metrics-certs\" (UniqueName: \"kubernetes.io/secret/1a34eaf4-4b44-4e51-b497-eada6dd5a4fd-metrics-certs\") pod \"1a34eaf4-4b44-4e51-b497-eada6dd5a4fd\" (UID: \"1a34eaf4-4b44-4e51-b497-eada6dd5a4fd\") " Oct 30 21:16:26.546110 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:16:26.546077 5378 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1a34eaf4-4b44-4e51-b497-eada6dd5a4fd-openshift-service-ca" pod "1a34eaf4-4b44-4e51-b497-eada6dd5a4fd" (UID: "1a34eaf4-4b44-4e51-b497-eada6dd5a4fd"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:16:26.546821 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:16:26.546792 5378 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1a34eaf4-4b44-4e51-b497-eada6dd5a4fd-metrics-certs" pod "1a34eaf4-4b44-4e51-b497-eada6dd5a4fd" (UID: "1a34eaf4-4b44-4e51-b497-eada6dd5a4fd"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:16:26.548515 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:16:26.548477 5378 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1a34eaf4-4b44-4e51-b497-eada6dd5a4fd-maas-api-tls" pod "1a34eaf4-4b44-4e51-b497-eada6dd5a4fd" (UID: "1a34eaf4-4b44-4e51-b497-eada6dd5a4fd"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:16:26.549581 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:16:26.549540 5378 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1a34eaf4-4b44-4e51-b497-eada6dd5a4fd-kube-api-access-skhkx" pod "1a34eaf4-4b44-4e51-b497-eada6dd5a4fd" (UID: "1a34eaf4-4b44-4e51-b497-eada6dd5a4fd"). InnerVolumeSpecName "kube-api-access-skhkx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:16:26.646930 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:16:26.646853 5378 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1a34eaf4-4b44-4e51-b497-eada6dd5a4fd-openshift-service-ca\") on node \"ip-10-0-138-149.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:16:26.646930 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:16:26.646889 5378 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1a34eaf4-4b44-4e51-b497-eada6dd5a4fd-metrics-certs\") on node \"ip-10-0-138-149.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:16:26.646930 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:16:26.646903 5378 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-skhkx\" (UniqueName: \"kubernetes.io/projected/1a34eaf4-4b44-4e51-b497-eada6dd5a4fd-kube-api-access-skhkx\") on node \"ip-10-0-138-149.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:16:26.646930 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:16:26.646915 5378 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1a34eaf4-4b44-4e51-b497-eada6dd5a4fd-maas-api-tls\") on node \"ip-10-0-138-149.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:16:27.124327 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:16:27.124285 5378 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-390b71-64988c8666-xfnmq" event={"ID":"1a34eaf4-4b44-4e51-b497-eada6dd5a4fd","Type":"ContainerDied","Data":"c87b3a1240cd2d3a130b8cb71b568bf11633db536dd3a49f9c10a2ef82e9901f"} Oct 30 21:16:27.144693 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:16:27.144657 5378 scope.go:122] "RemoveContainer" containerID="02e66a089d3895d4ae17d332127b2a1c579141bb1f755ea4ef8c3728a36e2cf6" Oct 30 21:16:27.145148 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:16:27.145125 5378 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-390b71-64988c8666-xfnmq" Oct 30 21:16:27.240209 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:16:27.240107 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:16:27.479165 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:16:27.479122 5378 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-390b71-64988c8666-xfnmq"] Oct 30 21:16:27.479463 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:16:27.479443 5378 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-390b71-64988c8666-xfnmq"] Oct 30 21:16:27.629278 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:16:27.629230 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:16:27.880666 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:16:27.874259 5378 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1a34eaf4-4b44-4e51-b497-eada6dd5a4fd" path="/var/lib/kubelet/pods/1a34eaf4-4b44-4e51-b497-eada6dd5a4fd/volumes" Oct 30 21:16:28.021858 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:16:28.021818 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:16:28.273665 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:16:28.272948 5378 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-390b71-484bv"] Oct 30 21:16:28.298372 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:16:28.298329 5378 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-390b71-484bv"] Oct 30 21:16:28.550117 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:16:28.549983 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:16:28.675241 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:16:28.675197 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:16:29.037747 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:16:29.037708 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:16:29.337467 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:16:29.337380 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:16:29.460741 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:16:29.460698 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:16:29.746270 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:16:29.746230 5378 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b24b6671-0dce-4274-8cd1-af908fc9e591" path="/var/lib/kubelet/pods/b24b6671-0dce-4274-8cd1-af908fc9e591/volumes" Oct 30 21:16:30.025743 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:16:30.025658 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:16:34.840429 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:16:34.840378 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:16:35.501747 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:16:35.501714 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:16:35.639904 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:16:35.639868 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:16:35.836718 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:16:35.836635 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:16:35.866639 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:16:35.866607 5378 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-390b71-openshift-default-df8b44c8c-5k4nj"] Oct 30 21:16:35.866999 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:16:35.866819 5378 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-390b71-openshift-default-df8b44c8c-5k4nj" podUID="dd395160-55d3-4dfb-a2b1-6e68c62f32b5" containerName="istio-proxy" containerID="cri-o://52716f3a2df2478b92010b966b4f1c27c7bc90c5b7fe7b0b16c961c15fc20bb3" gracePeriod=30 Oct 30 21:16:36.563475 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:16:36.563442 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:16:36.701725 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:16:36.701687 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:16:36.926515 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:16:36.926476 5378 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-b888d0-openshift-default-65d785fbfc-7lvf5"] Oct 30 21:16:36.926959 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:16:36.926734 5378 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-a-b888d0-openshift-default-65d785fbfc-7lvf5" podUID="dce080ad-1859-4657-aea5-9a2749ac1866" containerName="istio-proxy" containerID="cri-o://87b4a6c4032af7c847dd052c0abd6e5eeb843593014c5658c6c85db0df22db28" gracePeriod=30 Oct 30 21:16:36.926959 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:16:36.926734 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:16:36.935301 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:16:36.935273 5378 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-b888d0-openshift-default-65d785fbfc-m9ld8"] Oct 30 21:16:36.935484 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:16:36.935463 5378 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-a-b888d0-openshift-default-65d785fbfc-m9ld8" podUID="ffcf4858-801d-4c04-be59-2c2492f56947" containerName="istio-proxy" containerID="cri-o://bedb5cdbeaa14b70dc1b81bc8a66c6dc91881456249eb23ac3104f92190d47ee" gracePeriod=30 Oct 30 21:16:37.759370 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:16:37.759332 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:16:40.132350 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:16:40.132320 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:16:40.297539 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:16:40.297504 5378 patch_prober.go:29] interesting pod/e2e-inf-a-390b71-openshift-default-df8b44c8c-5k4nj container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 21:16:40.297717 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:16:40.297602 5378 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-a-390b71-openshift-default-df8b44c8c-5k4nj" podUID="dd395160-55d3-4dfb-a2b1-6e68c62f32b5" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 21:16:41.154374 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:16:41.154341 5378 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-390b71-openshift-default-df8b44c8c-5k4nj" Oct 30 21:16:41.216672 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:16:41.216636 5378 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dd395160-55d3-4dfb-a2b1-6e68c62f32b5-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dd395160-55d3-4dfb-a2b1-6e68c62f32b5-istiod-ca-cert\") pod \"dd395160-55d3-4dfb-a2b1-6e68c62f32b5\" (UID: \"dd395160-55d3-4dfb-a2b1-6e68c62f32b5\") " Oct 30 21:16:41.216879 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:16:41.216731 5378 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dd395160-55d3-4dfb-a2b1-6e68c62f32b5-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dd395160-55d3-4dfb-a2b1-6e68c62f32b5-workload-certs\") pod \"dd395160-55d3-4dfb-a2b1-6e68c62f32b5\" (UID: \"dd395160-55d3-4dfb-a2b1-6e68c62f32b5\") " Oct 30 21:16:41.216879 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:16:41.216771 5378 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dd395160-55d3-4dfb-a2b1-6e68c62f32b5-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dd395160-55d3-4dfb-a2b1-6e68c62f32b5-istio-data\") pod \"dd395160-55d3-4dfb-a2b1-6e68c62f32b5\" (UID: \"dd395160-55d3-4dfb-a2b1-6e68c62f32b5\") " Oct 30 21:16:41.216879 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:16:41.216856 5378 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/dd395160-55d3-4dfb-a2b1-6e68c62f32b5-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dd395160-55d3-4dfb-a2b1-6e68c62f32b5-istio-podinfo\") pod \"dd395160-55d3-4dfb-a2b1-6e68c62f32b5\" (UID: \"dd395160-55d3-4dfb-a2b1-6e68c62f32b5\") " Oct 30 21:16:41.217063 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:16:41.216894 5378 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dd395160-55d3-4dfb-a2b1-6e68c62f32b5-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dd395160-55d3-4dfb-a2b1-6e68c62f32b5-workload-socket\") pod \"dd395160-55d3-4dfb-a2b1-6e68c62f32b5\" (UID: \"dd395160-55d3-4dfb-a2b1-6e68c62f32b5\") " Oct 30 21:16:41.217063 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:16:41.216926 5378 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dd395160-55d3-4dfb-a2b1-6e68c62f32b5-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dd395160-55d3-4dfb-a2b1-6e68c62f32b5-credential-socket\") pod \"dd395160-55d3-4dfb-a2b1-6e68c62f32b5\" (UID: \"dd395160-55d3-4dfb-a2b1-6e68c62f32b5\") " Oct 30 21:16:41.217063 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:16:41.216999 5378 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dd395160-55d3-4dfb-a2b1-6e68c62f32b5-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dd395160-55d3-4dfb-a2b1-6e68c62f32b5-istio-envoy\") pod \"dd395160-55d3-4dfb-a2b1-6e68c62f32b5\" (UID: \"dd395160-55d3-4dfb-a2b1-6e68c62f32b5\") " Oct 30 21:16:41.217231 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:16:41.217079 5378 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dd395160-55d3-4dfb-a2b1-6e68c62f32b5-istiod-ca-cert" pod "dd395160-55d3-4dfb-a2b1-6e68c62f32b5" (UID: "dd395160-55d3-4dfb-a2b1-6e68c62f32b5"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:16:41.217231 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:16:41.217087 5378 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dd395160-55d3-4dfb-a2b1-6e68c62f32b5-workload-certs" pod "dd395160-55d3-4dfb-a2b1-6e68c62f32b5" (UID: "dd395160-55d3-4dfb-a2b1-6e68c62f32b5"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:16:41.217231 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:16:41.217104 5378 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dd395160-55d3-4dfb-a2b1-6e68c62f32b5-istio-token\" (UniqueName: \"kubernetes.io/projected/dd395160-55d3-4dfb-a2b1-6e68c62f32b5-istio-token\") pod \"dd395160-55d3-4dfb-a2b1-6e68c62f32b5\" (UID: \"dd395160-55d3-4dfb-a2b1-6e68c62f32b5\") " Oct 30 21:16:41.217231 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:16:41.217128 5378 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dd395160-55d3-4dfb-a2b1-6e68c62f32b5-istio-data" pod "dd395160-55d3-4dfb-a2b1-6e68c62f32b5" (UID: "dd395160-55d3-4dfb-a2b1-6e68c62f32b5"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:16:41.217231 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:16:41.217170 5378 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dd395160-55d3-4dfb-a2b1-6e68c62f32b5-kube-api-access-9c7fs\" (UniqueName: \"kubernetes.io/projected/dd395160-55d3-4dfb-a2b1-6e68c62f32b5-kube-api-access-9c7fs\") pod \"dd395160-55d3-4dfb-a2b1-6e68c62f32b5\" (UID: \"dd395160-55d3-4dfb-a2b1-6e68c62f32b5\") " Oct 30 21:16:41.217443 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:16:41.217296 5378 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dd395160-55d3-4dfb-a2b1-6e68c62f32b5-credential-socket" pod "dd395160-55d3-4dfb-a2b1-6e68c62f32b5" (UID: "dd395160-55d3-4dfb-a2b1-6e68c62f32b5"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:16:41.217513 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:16:41.217477 5378 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dd395160-55d3-4dfb-a2b1-6e68c62f32b5-workload-socket" pod "dd395160-55d3-4dfb-a2b1-6e68c62f32b5" (UID: "dd395160-55d3-4dfb-a2b1-6e68c62f32b5"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:16:41.217816 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:16:41.217795 5378 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dd395160-55d3-4dfb-a2b1-6e68c62f32b5-workload-certs\") on node \"ip-10-0-138-149.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:16:41.217816 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:16:41.217815 5378 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dd395160-55d3-4dfb-a2b1-6e68c62f32b5-istio-data\") on node \"ip-10-0-138-149.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:16:41.218012 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:16:41.217824 5378 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dd395160-55d3-4dfb-a2b1-6e68c62f32b5-workload-socket\") on node \"ip-10-0-138-149.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:16:41.218012 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:16:41.217832 5378 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dd395160-55d3-4dfb-a2b1-6e68c62f32b5-credential-socket\") on node \"ip-10-0-138-149.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:16:41.218012 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:16:41.217841 5378 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dd395160-55d3-4dfb-a2b1-6e68c62f32b5-istiod-ca-cert\") on node \"ip-10-0-138-149.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:16:41.219343 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:16:41.219319 5378 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dd395160-55d3-4dfb-a2b1-6e68c62f32b5-istio-token" pod "dd395160-55d3-4dfb-a2b1-6e68c62f32b5" (UID: "dd395160-55d3-4dfb-a2b1-6e68c62f32b5"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:16:41.219510 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:16:41.219489 5378 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dd395160-55d3-4dfb-a2b1-6e68c62f32b5-istio-envoy" pod "dd395160-55d3-4dfb-a2b1-6e68c62f32b5" (UID: "dd395160-55d3-4dfb-a2b1-6e68c62f32b5"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:16:41.219733 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:16:41.219715 5378 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/dd395160-55d3-4dfb-a2b1-6e68c62f32b5-istio-podinfo" pod "dd395160-55d3-4dfb-a2b1-6e68c62f32b5" (UID: "dd395160-55d3-4dfb-a2b1-6e68c62f32b5"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 21:16:41.219733 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:16:41.219713 5378 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dd395160-55d3-4dfb-a2b1-6e68c62f32b5-kube-api-access-9c7fs" pod "dd395160-55d3-4dfb-a2b1-6e68c62f32b5" (UID: "dd395160-55d3-4dfb-a2b1-6e68c62f32b5"). InnerVolumeSpecName "kube-api-access-9c7fs". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:16:41.238812 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:16:41.238786 5378 generic.go:356] "Generic (PLEG): container finished" podID="dd395160-55d3-4dfb-a2b1-6e68c62f32b5" containerID="52716f3a2df2478b92010b966b4f1c27c7bc90c5b7fe7b0b16c961c15fc20bb3" exitCode=0 Oct 30 21:16:41.238910 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:16:41.238842 5378 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-390b71-openshift-default-df8b44c8c-5k4nj" event={"ID":"dd395160-55d3-4dfb-a2b1-6e68c62f32b5","Type":"ContainerDied","Data":"52716f3a2df2478b92010b966b4f1c27c7bc90c5b7fe7b0b16c961c15fc20bb3"} Oct 30 21:16:41.238910 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:16:41.238888 5378 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-390b71-openshift-default-df8b44c8c-5k4nj" event={"ID":"dd395160-55d3-4dfb-a2b1-6e68c62f32b5","Type":"ContainerDied","Data":"31b939607c7f5288ea403e221a9a3c6fed3ad68ce72306f58b77af1affacfbe1"} Oct 30 21:16:41.238987 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:16:41.238909 5378 scope.go:122] "RemoveContainer" containerID="52716f3a2df2478b92010b966b4f1c27c7bc90c5b7fe7b0b16c961c15fc20bb3" Oct 30 21:16:41.238987 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:16:41.238842 5378 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-390b71-openshift-default-df8b44c8c-5k4nj" Oct 30 21:16:41.249150 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:16:41.249132 5378 scope.go:122] "RemoveContainer" containerID="28e80945bafbdf729da98e7b0864087040d0ba4695c109e85aebe64751bebafd" Oct 30 21:16:41.258552 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:16:41.258530 5378 scope.go:122] "RemoveContainer" containerID="52716f3a2df2478b92010b966b4f1c27c7bc90c5b7fe7b0b16c961c15fc20bb3" Oct 30 21:16:41.258923 ip-10-0-138-149 kubenswrapper[5378]: E1030 21:16:41.258901 5378 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"52716f3a2df2478b92010b966b4f1c27c7bc90c5b7fe7b0b16c961c15fc20bb3\": container with ID starting with 52716f3a2df2478b92010b966b4f1c27c7bc90c5b7fe7b0b16c961c15fc20bb3 not found: ID does not exist" containerID="52716f3a2df2478b92010b966b4f1c27c7bc90c5b7fe7b0b16c961c15fc20bb3" Oct 30 21:16:41.258990 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:16:41.258933 5378 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"52716f3a2df2478b92010b966b4f1c27c7bc90c5b7fe7b0b16c961c15fc20bb3"} err="failed to get container status \"52716f3a2df2478b92010b966b4f1c27c7bc90c5b7fe7b0b16c961c15fc20bb3\": rpc error: code = NotFound desc = could not find container \"52716f3a2df2478b92010b966b4f1c27c7bc90c5b7fe7b0b16c961c15fc20bb3\": container with ID starting with 52716f3a2df2478b92010b966b4f1c27c7bc90c5b7fe7b0b16c961c15fc20bb3 not found: ID does not exist" Oct 30 21:16:41.258990 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:16:41.258952 5378 scope.go:122] "RemoveContainer" containerID="28e80945bafbdf729da98e7b0864087040d0ba4695c109e85aebe64751bebafd" Oct 30 21:16:41.259218 ip-10-0-138-149 kubenswrapper[5378]: E1030 21:16:41.259199 5378 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"28e80945bafbdf729da98e7b0864087040d0ba4695c109e85aebe64751bebafd\": container with ID starting with 28e80945bafbdf729da98e7b0864087040d0ba4695c109e85aebe64751bebafd not found: ID does not exist" containerID="28e80945bafbdf729da98e7b0864087040d0ba4695c109e85aebe64751bebafd" Oct 30 21:16:41.259263 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:16:41.259226 5378 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"28e80945bafbdf729da98e7b0864087040d0ba4695c109e85aebe64751bebafd"} err="failed to get container status \"28e80945bafbdf729da98e7b0864087040d0ba4695c109e85aebe64751bebafd\": rpc error: code = NotFound desc = could not find container \"28e80945bafbdf729da98e7b0864087040d0ba4695c109e85aebe64751bebafd\": container with ID starting with 28e80945bafbdf729da98e7b0864087040d0ba4695c109e85aebe64751bebafd not found: ID does not exist" Oct 30 21:16:41.318946 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:16:41.318908 5378 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dd395160-55d3-4dfb-a2b1-6e68c62f32b5-istio-envoy\") on node \"ip-10-0-138-149.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:16:41.318946 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:16:41.318941 5378 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dd395160-55d3-4dfb-a2b1-6e68c62f32b5-istio-token\") on node \"ip-10-0-138-149.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:16:41.318946 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:16:41.318952 5378 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9c7fs\" (UniqueName: \"kubernetes.io/projected/dd395160-55d3-4dfb-a2b1-6e68c62f32b5-kube-api-access-9c7fs\") on node \"ip-10-0-138-149.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:16:41.319223 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:16:41.318961 5378 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dd395160-55d3-4dfb-a2b1-6e68c62f32b5-istio-podinfo\") on node \"ip-10-0-138-149.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:16:41.486392 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:16:41.486304 5378 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-390b71-openshift-default-df8b44c8c-5k4nj"] Oct 30 21:16:41.491326 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:16:41.491289 5378 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-390b71-openshift-default-df8b44c8c-5k4nj"] Oct 30 21:16:41.742248 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:16:41.742163 5378 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dd395160-55d3-4dfb-a2b1-6e68c62f32b5" path="/var/lib/kubelet/pods/dd395160-55d3-4dfb-a2b1-6e68c62f32b5/volumes" Oct 30 21:16:42.207947 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:16:42.207919 5378 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-b888d0-openshift-default-65d785fbfc-7lvf5" Oct 30 21:16:42.246413 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:16:42.246381 5378 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-b888d0-openshift-default-65d785fbfc-m9ld8" Oct 30 21:16:42.247622 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:16:42.247592 5378 generic.go:356] "Generic (PLEG): container finished" podID="dce080ad-1859-4657-aea5-9a2749ac1866" containerID="87b4a6c4032af7c847dd052c0abd6e5eeb843593014c5658c6c85db0df22db28" exitCode=0 Oct 30 21:16:42.247764 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:16:42.247650 5378 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-b888d0-openshift-default-65d785fbfc-7lvf5" Oct 30 21:16:42.247764 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:16:42.247684 5378 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-b888d0-openshift-default-65d785fbfc-7lvf5" event={"ID":"dce080ad-1859-4657-aea5-9a2749ac1866","Type":"ContainerDied","Data":"87b4a6c4032af7c847dd052c0abd6e5eeb843593014c5658c6c85db0df22db28"} Oct 30 21:16:42.247764 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:16:42.247736 5378 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-b888d0-openshift-default-65d785fbfc-7lvf5" event={"ID":"dce080ad-1859-4657-aea5-9a2749ac1866","Type":"ContainerDied","Data":"bdb03cc247502914f9875ae64eec845a373a1d4f5d1864214d4e59dc646879eb"} Oct 30 21:16:42.247912 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:16:42.247770 5378 scope.go:122] "RemoveContainer" containerID="87b4a6c4032af7c847dd052c0abd6e5eeb843593014c5658c6c85db0df22db28" Oct 30 21:16:42.248976 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:16:42.248951 5378 generic.go:356] "Generic (PLEG): container finished" podID="ffcf4858-801d-4c04-be59-2c2492f56947" containerID="bedb5cdbeaa14b70dc1b81bc8a66c6dc91881456249eb23ac3104f92190d47ee" exitCode=0 Oct 30 21:16:42.249066 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:16:42.249033 5378 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-b888d0-openshift-default-65d785fbfc-m9ld8" Oct 30 21:16:42.249120 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:16:42.249074 5378 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-b888d0-openshift-default-65d785fbfc-m9ld8" event={"ID":"ffcf4858-801d-4c04-be59-2c2492f56947","Type":"ContainerDied","Data":"bedb5cdbeaa14b70dc1b81bc8a66c6dc91881456249eb23ac3104f92190d47ee"} Oct 30 21:16:42.249178 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:16:42.249134 5378 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-b888d0-openshift-default-65d785fbfc-m9ld8" event={"ID":"ffcf4858-801d-4c04-be59-2c2492f56947","Type":"ContainerDied","Data":"686ef4e51b8c165372eff5d0336caa439034e1be38ed795ecfbc00b3fa4cb974"} Oct 30 21:16:42.257874 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:16:42.257854 5378 scope.go:122] "RemoveContainer" containerID="87b4a6c4032af7c847dd052c0abd6e5eeb843593014c5658c6c85db0df22db28" Oct 30 21:16:42.258181 ip-10-0-138-149 kubenswrapper[5378]: E1030 21:16:42.258160 5378 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"87b4a6c4032af7c847dd052c0abd6e5eeb843593014c5658c6c85db0df22db28\": container with ID starting with 87b4a6c4032af7c847dd052c0abd6e5eeb843593014c5658c6c85db0df22db28 not found: ID does not exist" containerID="87b4a6c4032af7c847dd052c0abd6e5eeb843593014c5658c6c85db0df22db28" Oct 30 21:16:42.258230 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:16:42.258189 5378 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"87b4a6c4032af7c847dd052c0abd6e5eeb843593014c5658c6c85db0df22db28"} err="failed to get container status \"87b4a6c4032af7c847dd052c0abd6e5eeb843593014c5658c6c85db0df22db28\": rpc error: code = NotFound desc = could not find container \"87b4a6c4032af7c847dd052c0abd6e5eeb843593014c5658c6c85db0df22db28\": container with ID starting with 87b4a6c4032af7c847dd052c0abd6e5eeb843593014c5658c6c85db0df22db28 not found: ID does not exist" Oct 30 21:16:42.258230 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:16:42.258208 5378 scope.go:122] "RemoveContainer" containerID="bedb5cdbeaa14b70dc1b81bc8a66c6dc91881456249eb23ac3104f92190d47ee" Oct 30 21:16:42.267648 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:16:42.267627 5378 scope.go:122] "RemoveContainer" containerID="bedb5cdbeaa14b70dc1b81bc8a66c6dc91881456249eb23ac3104f92190d47ee" Oct 30 21:16:42.267960 ip-10-0-138-149 kubenswrapper[5378]: E1030 21:16:42.267939 5378 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bedb5cdbeaa14b70dc1b81bc8a66c6dc91881456249eb23ac3104f92190d47ee\": container with ID starting with bedb5cdbeaa14b70dc1b81bc8a66c6dc91881456249eb23ac3104f92190d47ee not found: ID does not exist" containerID="bedb5cdbeaa14b70dc1b81bc8a66c6dc91881456249eb23ac3104f92190d47ee" Oct 30 21:16:42.268010 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:16:42.267969 5378 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bedb5cdbeaa14b70dc1b81bc8a66c6dc91881456249eb23ac3104f92190d47ee"} err="failed to get container status \"bedb5cdbeaa14b70dc1b81bc8a66c6dc91881456249eb23ac3104f92190d47ee\": rpc error: code = NotFound desc = could not find container \"bedb5cdbeaa14b70dc1b81bc8a66c6dc91881456249eb23ac3104f92190d47ee\": container with ID starting with bedb5cdbeaa14b70dc1b81bc8a66c6dc91881456249eb23ac3104f92190d47ee not found: ID does not exist" Oct 30 21:16:42.331673 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:16:42.331581 5378 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dce080ad-1859-4657-aea5-9a2749ac1866-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dce080ad-1859-4657-aea5-9a2749ac1866-istio-data\") pod \"dce080ad-1859-4657-aea5-9a2749ac1866\" (UID: \"dce080ad-1859-4657-aea5-9a2749ac1866\") " Oct 30 21:16:42.331673 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:16:42.331663 5378 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ffcf4858-801d-4c04-be59-2c2492f56947-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ffcf4858-801d-4c04-be59-2c2492f56947-workload-certs\") pod \"ffcf4858-801d-4c04-be59-2c2492f56947\" (UID: \"ffcf4858-801d-4c04-be59-2c2492f56947\") " Oct 30 21:16:42.331954 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:16:42.331702 5378 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dce080ad-1859-4657-aea5-9a2749ac1866-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dce080ad-1859-4657-aea5-9a2749ac1866-credential-socket\") pod \"dce080ad-1859-4657-aea5-9a2749ac1866\" (UID: \"dce080ad-1859-4657-aea5-9a2749ac1866\") " Oct 30 21:16:42.331954 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:16:42.331752 5378 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ffcf4858-801d-4c04-be59-2c2492f56947-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ffcf4858-801d-4c04-be59-2c2492f56947-istio-envoy\") pod \"ffcf4858-801d-4c04-be59-2c2492f56947\" (UID: \"ffcf4858-801d-4c04-be59-2c2492f56947\") " Oct 30 21:16:42.331954 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:16:42.331792 5378 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dce080ad-1859-4657-aea5-9a2749ac1866-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dce080ad-1859-4657-aea5-9a2749ac1866-workload-socket\") pod \"dce080ad-1859-4657-aea5-9a2749ac1866\" (UID: \"dce080ad-1859-4657-aea5-9a2749ac1866\") " Oct 30 21:16:42.331954 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:16:42.331826 5378 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dce080ad-1859-4657-aea5-9a2749ac1866-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dce080ad-1859-4657-aea5-9a2749ac1866-istio-envoy\") pod \"dce080ad-1859-4657-aea5-9a2749ac1866\" (UID: \"dce080ad-1859-4657-aea5-9a2749ac1866\") " Oct 30 21:16:42.331954 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:16:42.331855 5378 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ffcf4858-801d-4c04-be59-2c2492f56947-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ffcf4858-801d-4c04-be59-2c2492f56947-credential-socket\") pod \"ffcf4858-801d-4c04-be59-2c2492f56947\" (UID: \"ffcf4858-801d-4c04-be59-2c2492f56947\") " Oct 30 21:16:42.332538 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:16:42.332009 5378 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ffcf4858-801d-4c04-be59-2c2492f56947-istio-token\" (UniqueName: \"kubernetes.io/projected/ffcf4858-801d-4c04-be59-2c2492f56947-istio-token\") pod \"ffcf4858-801d-4c04-be59-2c2492f56947\" (UID: \"ffcf4858-801d-4c04-be59-2c2492f56947\") " Oct 30 21:16:42.332538 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:16:42.332046 5378 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dce080ad-1859-4657-aea5-9a2749ac1866-istio-token\" (UniqueName: \"kubernetes.io/projected/dce080ad-1859-4657-aea5-9a2749ac1866-istio-token\") pod \"dce080ad-1859-4657-aea5-9a2749ac1866\" (UID: \"dce080ad-1859-4657-aea5-9a2749ac1866\") " Oct 30 21:16:42.332538 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:16:42.332092 5378 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ffcf4858-801d-4c04-be59-2c2492f56947-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ffcf4858-801d-4c04-be59-2c2492f56947-istio-data\") pod \"ffcf4858-801d-4c04-be59-2c2492f56947\" (UID: \"ffcf4858-801d-4c04-be59-2c2492f56947\") " Oct 30 21:16:42.332538 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:16:42.332185 5378 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dce080ad-1859-4657-aea5-9a2749ac1866-kube-api-access-p958m\" (UniqueName: \"kubernetes.io/projected/dce080ad-1859-4657-aea5-9a2749ac1866-kube-api-access-p958m\") pod \"dce080ad-1859-4657-aea5-9a2749ac1866\" (UID: \"dce080ad-1859-4657-aea5-9a2749ac1866\") " Oct 30 21:16:42.332538 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:16:42.332223 5378 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dce080ad-1859-4657-aea5-9a2749ac1866-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dce080ad-1859-4657-aea5-9a2749ac1866-workload-certs\") pod \"dce080ad-1859-4657-aea5-9a2749ac1866\" (UID: \"dce080ad-1859-4657-aea5-9a2749ac1866\") " Oct 30 21:16:42.332538 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:16:42.332256 5378 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/ffcf4858-801d-4c04-be59-2c2492f56947-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ffcf4858-801d-4c04-be59-2c2492f56947-istio-podinfo\") pod \"ffcf4858-801d-4c04-be59-2c2492f56947\" (UID: \"ffcf4858-801d-4c04-be59-2c2492f56947\") " Oct 30 21:16:42.332538 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:16:42.332330 5378 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ffcf4858-801d-4c04-be59-2c2492f56947-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ffcf4858-801d-4c04-be59-2c2492f56947-istiod-ca-cert\") pod \"ffcf4858-801d-4c04-be59-2c2492f56947\" (UID: \"ffcf4858-801d-4c04-be59-2c2492f56947\") " Oct 30 21:16:42.332538 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:16:42.332408 5378 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/dce080ad-1859-4657-aea5-9a2749ac1866-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dce080ad-1859-4657-aea5-9a2749ac1866-istio-podinfo\") pod \"dce080ad-1859-4657-aea5-9a2749ac1866\" (UID: \"dce080ad-1859-4657-aea5-9a2749ac1866\") " Oct 30 21:16:42.332538 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:16:42.332437 5378 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ffcf4858-801d-4c04-be59-2c2492f56947-kube-api-access-bn4v4\" (UniqueName: \"kubernetes.io/projected/ffcf4858-801d-4c04-be59-2c2492f56947-kube-api-access-bn4v4\") pod \"ffcf4858-801d-4c04-be59-2c2492f56947\" (UID: \"ffcf4858-801d-4c04-be59-2c2492f56947\") " Oct 30 21:16:42.332538 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:16:42.332466 5378 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ffcf4858-801d-4c04-be59-2c2492f56947-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ffcf4858-801d-4c04-be59-2c2492f56947-workload-socket\") pod \"ffcf4858-801d-4c04-be59-2c2492f56947\" (UID: \"ffcf4858-801d-4c04-be59-2c2492f56947\") " Oct 30 21:16:42.333237 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:16:42.332554 5378 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dce080ad-1859-4657-aea5-9a2749ac1866-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dce080ad-1859-4657-aea5-9a2749ac1866-istiod-ca-cert\") pod \"dce080ad-1859-4657-aea5-9a2749ac1866\" (UID: \"dce080ad-1859-4657-aea5-9a2749ac1866\") " Oct 30 21:16:42.333237 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:16:42.331848 5378 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dce080ad-1859-4657-aea5-9a2749ac1866-istio-data" pod "dce080ad-1859-4657-aea5-9a2749ac1866" (UID: "dce080ad-1859-4657-aea5-9a2749ac1866"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:16:42.333237 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:16:42.332275 5378 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dce080ad-1859-4657-aea5-9a2749ac1866-credential-socket" pod "dce080ad-1859-4657-aea5-9a2749ac1866" (UID: "dce080ad-1859-4657-aea5-9a2749ac1866"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:16:42.333237 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:16:42.332434 5378 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ffcf4858-801d-4c04-be59-2c2492f56947-credential-socket" pod "ffcf4858-801d-4c04-be59-2c2492f56947" (UID: "ffcf4858-801d-4c04-be59-2c2492f56947"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:16:42.333237 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:16:42.332437 5378 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dce080ad-1859-4657-aea5-9a2749ac1866-workload-socket" pod "dce080ad-1859-4657-aea5-9a2749ac1866" (UID: "dce080ad-1859-4657-aea5-9a2749ac1866"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:16:42.333237 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:16:42.332829 5378 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ffcf4858-801d-4c04-be59-2c2492f56947-istio-data" pod "ffcf4858-801d-4c04-be59-2c2492f56947" (UID: "ffcf4858-801d-4c04-be59-2c2492f56947"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:16:42.333237 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:16:42.333114 5378 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ffcf4858-801d-4c04-be59-2c2492f56947-workload-certs" pod "ffcf4858-801d-4c04-be59-2c2492f56947" (UID: "ffcf4858-801d-4c04-be59-2c2492f56947"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:16:42.333982 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:16:42.333956 5378 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ffcf4858-801d-4c04-be59-2c2492f56947-workload-socket" pod "ffcf4858-801d-4c04-be59-2c2492f56947" (UID: "ffcf4858-801d-4c04-be59-2c2492f56947"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:16:42.334110 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:16:42.334072 5378 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dce080ad-1859-4657-aea5-9a2749ac1866-workload-certs" pod "dce080ad-1859-4657-aea5-9a2749ac1866" (UID: "dce080ad-1859-4657-aea5-9a2749ac1866"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:16:42.334356 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:16:42.334325 5378 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dce080ad-1859-4657-aea5-9a2749ac1866-istiod-ca-cert" pod "dce080ad-1859-4657-aea5-9a2749ac1866" (UID: "dce080ad-1859-4657-aea5-9a2749ac1866"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:16:42.334457 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:16:42.334381 5378 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ffcf4858-801d-4c04-be59-2c2492f56947-istiod-ca-cert" pod "ffcf4858-801d-4c04-be59-2c2492f56947" (UID: "ffcf4858-801d-4c04-be59-2c2492f56947"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:16:42.334916 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:16:42.334871 5378 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ffcf4858-801d-4c04-be59-2c2492f56947-istio-token" pod "ffcf4858-801d-4c04-be59-2c2492f56947" (UID: "ffcf4858-801d-4c04-be59-2c2492f56947"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:16:42.335815 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:16:42.335780 5378 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ffcf4858-801d-4c04-be59-2c2492f56947-kube-api-access-bn4v4" pod "ffcf4858-801d-4c04-be59-2c2492f56947" (UID: "ffcf4858-801d-4c04-be59-2c2492f56947"). InnerVolumeSpecName "kube-api-access-bn4v4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:16:42.336480 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:16:42.336458 5378 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dce080ad-1859-4657-aea5-9a2749ac1866-istio-envoy" pod "dce080ad-1859-4657-aea5-9a2749ac1866" (UID: "dce080ad-1859-4657-aea5-9a2749ac1866"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:16:42.336823 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:16:42.336796 5378 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/dce080ad-1859-4657-aea5-9a2749ac1866-istio-podinfo" pod "dce080ad-1859-4657-aea5-9a2749ac1866" (UID: "dce080ad-1859-4657-aea5-9a2749ac1866"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 21:16:42.337037 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:16:42.337016 5378 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ffcf4858-801d-4c04-be59-2c2492f56947-istio-envoy" pod "ffcf4858-801d-4c04-be59-2c2492f56947" (UID: "ffcf4858-801d-4c04-be59-2c2492f56947"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:16:42.337393 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:16:42.337376 5378 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dce080ad-1859-4657-aea5-9a2749ac1866-kube-api-access-p958m" pod "dce080ad-1859-4657-aea5-9a2749ac1866" (UID: "dce080ad-1859-4657-aea5-9a2749ac1866"). InnerVolumeSpecName "kube-api-access-p958m". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:16:42.337629 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:16:42.337614 5378 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ffcf4858-801d-4c04-be59-2c2492f56947-istio-podinfo" pod "ffcf4858-801d-4c04-be59-2c2492f56947" (UID: "ffcf4858-801d-4c04-be59-2c2492f56947"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 21:16:42.338434 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:16:42.338414 5378 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dce080ad-1859-4657-aea5-9a2749ac1866-istio-token" pod "dce080ad-1859-4657-aea5-9a2749ac1866" (UID: "dce080ad-1859-4657-aea5-9a2749ac1866"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:16:42.434230 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:16:42.434194 5378 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dce080ad-1859-4657-aea5-9a2749ac1866-istio-data\") on node \"ip-10-0-138-149.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:16:42.434230 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:16:42.434224 5378 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ffcf4858-801d-4c04-be59-2c2492f56947-workload-certs\") on node \"ip-10-0-138-149.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:16:42.434230 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:16:42.434235 5378 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dce080ad-1859-4657-aea5-9a2749ac1866-credential-socket\") on node \"ip-10-0-138-149.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:16:42.434528 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:16:42.434244 5378 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ffcf4858-801d-4c04-be59-2c2492f56947-istio-envoy\") on node \"ip-10-0-138-149.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:16:42.434528 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:16:42.434254 5378 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dce080ad-1859-4657-aea5-9a2749ac1866-workload-socket\") on node \"ip-10-0-138-149.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:16:42.434528 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:16:42.434263 5378 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dce080ad-1859-4657-aea5-9a2749ac1866-istio-envoy\") on node \"ip-10-0-138-149.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:16:42.434528 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:16:42.434273 5378 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ffcf4858-801d-4c04-be59-2c2492f56947-credential-socket\") on node \"ip-10-0-138-149.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:16:42.434528 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:16:42.434286 5378 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ffcf4858-801d-4c04-be59-2c2492f56947-istio-token\") on node \"ip-10-0-138-149.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:16:42.434528 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:16:42.434294 5378 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dce080ad-1859-4657-aea5-9a2749ac1866-istio-token\") on node \"ip-10-0-138-149.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:16:42.434528 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:16:42.434302 5378 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ffcf4858-801d-4c04-be59-2c2492f56947-istio-data\") on node \"ip-10-0-138-149.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:16:42.434528 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:16:42.434311 5378 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p958m\" (UniqueName: \"kubernetes.io/projected/dce080ad-1859-4657-aea5-9a2749ac1866-kube-api-access-p958m\") on node \"ip-10-0-138-149.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:16:42.434528 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:16:42.434320 5378 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dce080ad-1859-4657-aea5-9a2749ac1866-workload-certs\") on node \"ip-10-0-138-149.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:16:42.434528 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:16:42.434329 5378 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ffcf4858-801d-4c04-be59-2c2492f56947-istio-podinfo\") on node \"ip-10-0-138-149.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:16:42.434528 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:16:42.434339 5378 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ffcf4858-801d-4c04-be59-2c2492f56947-istiod-ca-cert\") on node \"ip-10-0-138-149.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:16:42.434528 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:16:42.434356 5378 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dce080ad-1859-4657-aea5-9a2749ac1866-istio-podinfo\") on node \"ip-10-0-138-149.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:16:42.434528 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:16:42.434366 5378 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bn4v4\" (UniqueName: \"kubernetes.io/projected/ffcf4858-801d-4c04-be59-2c2492f56947-kube-api-access-bn4v4\") on node \"ip-10-0-138-149.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:16:42.434528 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:16:42.434375 5378 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ffcf4858-801d-4c04-be59-2c2492f56947-workload-socket\") on node \"ip-10-0-138-149.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:16:42.434528 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:16:42.434383 5378 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dce080ad-1859-4657-aea5-9a2749ac1866-istiod-ca-cert\") on node \"ip-10-0-138-149.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:16:42.676843 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:16:42.676803 5378 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-b888d0-openshift-default-65d785fbfc-7lvf5"] Oct 30 21:16:42.681373 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:16:42.681344 5378 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-a-b888d0-openshift-default-65d785fbfc-7lvf5"] Oct 30 21:16:42.796164 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:16:42.796136 5378 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-b888d0-openshift-default-65d785fbfc-m9ld8"] Oct 30 21:16:42.801213 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:16:42.801182 5378 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-a-b888d0-openshift-default-65d785fbfc-m9ld8"] Oct 30 21:16:42.856706 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:16:42.856667 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:16:43.741666 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:16:43.741631 5378 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dce080ad-1859-4657-aea5-9a2749ac1866" path="/var/lib/kubelet/pods/dce080ad-1859-4657-aea5-9a2749ac1866/volumes" Oct 30 21:16:43.742429 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:16:43.742407 5378 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ffcf4858-801d-4c04-be59-2c2492f56947" path="/var/lib/kubelet/pods/ffcf4858-801d-4c04-be59-2c2492f56947/volumes" Oct 30 21:16:44.170376 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:16:44.170336 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:16:44.637356 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:16:44.637310 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:16:45.127190 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:16:45.127153 5378 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-639b07-openshift-default-795b89858c-9q4rj"] Oct 30 21:16:45.127974 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:16:45.127951 5378 cpu_manager.go:414] "containerMap: removing container" podUID="ffcf4858-801d-4c04-be59-2c2492f56947" containerName="istio-proxy" Oct 30 21:16:45.128076 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:16:45.127975 5378 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ffcf4858-801d-4c04-be59-2c2492f56947" containerName="istio-proxy" Oct 30 21:16:45.128076 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:16:45.127995 5378 cpu_manager.go:414] "containerMap: removing container" podUID="3559349f-6be4-4ef6-8061-59c7d1a8b678" containerName="istio-proxy" Oct 30 21:16:45.128076 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:16:45.128003 5378 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3559349f-6be4-4ef6-8061-59c7d1a8b678" containerName="istio-proxy" Oct 30 21:16:45.128076 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:16:45.128014 5378 cpu_manager.go:414] "containerMap: removing container" podUID="dce080ad-1859-4657-aea5-9a2749ac1866" containerName="istio-proxy" Oct 30 21:16:45.128076 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:16:45.128021 5378 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dce080ad-1859-4657-aea5-9a2749ac1866" containerName="istio-proxy" Oct 30 21:16:45.128076 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:16:45.128050 5378 cpu_manager.go:414] "containerMap: removing container" podUID="1eb6161d-5e96-4dd5-bbe1-68300df5c425" containerName="istio-proxy" Oct 30 21:16:45.128076 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:16:45.128057 5378 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1eb6161d-5e96-4dd5-bbe1-68300df5c425" containerName="istio-proxy" Oct 30 21:16:45.128454 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:16:45.128084 5378 cpu_manager.go:414] "containerMap: removing container" podUID="4521c8f6-928b-47a2-8457-de63ee943c09" containerName="istio-proxy" Oct 30 21:16:45.128454 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:16:45.128091 5378 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4521c8f6-928b-47a2-8457-de63ee943c09" containerName="istio-proxy" Oct 30 21:16:45.128454 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:16:45.128103 5378 cpu_manager.go:414] "containerMap: removing container" podUID="ddde4251-43ac-4def-b4d4-e1f9a2a7dc30" containerName="istio-proxy" Oct 30 21:16:45.128454 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:16:45.128110 5378 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ddde4251-43ac-4def-b4d4-e1f9a2a7dc30" containerName="istio-proxy" Oct 30 21:16:45.128454 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:16:45.128121 5378 cpu_manager.go:414] "containerMap: removing container" podUID="3559349f-6be4-4ef6-8061-59c7d1a8b678" containerName="istio-proxy" Oct 30 21:16:45.128454 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:16:45.128128 5378 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3559349f-6be4-4ef6-8061-59c7d1a8b678" containerName="istio-proxy" Oct 30 21:16:45.128454 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:16:45.128142 5378 cpu_manager.go:414] "containerMap: removing container" podUID="dd395160-55d3-4dfb-a2b1-6e68c62f32b5" containerName="istio-proxy" Oct 30 21:16:45.128454 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:16:45.128149 5378 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dd395160-55d3-4dfb-a2b1-6e68c62f32b5" containerName="istio-proxy" Oct 30 21:16:45.128454 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:16:45.128170 5378 cpu_manager.go:414] "containerMap: removing container" podUID="1a34eaf4-4b44-4e51-b497-eada6dd5a4fd" containerName="maas-api" Oct 30 21:16:45.128454 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:16:45.128178 5378 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1a34eaf4-4b44-4e51-b497-eada6dd5a4fd" containerName="maas-api" Oct 30 21:16:45.128454 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:16:45.128194 5378 cpu_manager.go:414] "containerMap: removing container" podUID="dd395160-55d3-4dfb-a2b1-6e68c62f32b5" containerName="istio-proxy" Oct 30 21:16:45.128454 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:16:45.128201 5378 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dd395160-55d3-4dfb-a2b1-6e68c62f32b5" containerName="istio-proxy" Oct 30 21:16:45.128454 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:16:45.128375 5378 memory_manager.go:365] "RemoveStaleState removing state" podUID="dd395160-55d3-4dfb-a2b1-6e68c62f32b5" containerName="istio-proxy" Oct 30 21:16:45.128454 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:16:45.128394 5378 memory_manager.go:365] "RemoveStaleState removing state" podUID="1a34eaf4-4b44-4e51-b497-eada6dd5a4fd" containerName="maas-api" Oct 30 21:16:45.128454 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:16:45.128405 5378 memory_manager.go:365] "RemoveStaleState removing state" podUID="dd395160-55d3-4dfb-a2b1-6e68c62f32b5" containerName="istio-proxy" Oct 30 21:16:45.128454 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:16:45.128417 5378 memory_manager.go:365] "RemoveStaleState removing state" podUID="ffcf4858-801d-4c04-be59-2c2492f56947" containerName="istio-proxy" Oct 30 21:16:45.128454 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:16:45.128426 5378 memory_manager.go:365] "RemoveStaleState removing state" podUID="3559349f-6be4-4ef6-8061-59c7d1a8b678" containerName="istio-proxy" Oct 30 21:16:45.128454 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:16:45.128436 5378 memory_manager.go:365] "RemoveStaleState removing state" podUID="dce080ad-1859-4657-aea5-9a2749ac1866" containerName="istio-proxy" Oct 30 21:16:45.128454 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:16:45.128448 5378 memory_manager.go:365] "RemoveStaleState removing state" podUID="1eb6161d-5e96-4dd5-bbe1-68300df5c425" containerName="istio-proxy" Oct 30 21:16:45.128454 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:16:45.128459 5378 memory_manager.go:365] "RemoveStaleState removing state" podUID="4521c8f6-928b-47a2-8457-de63ee943c09" containerName="istio-proxy" Oct 30 21:16:45.129402 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:16:45.128468 5378 memory_manager.go:365] "RemoveStaleState removing state" podUID="ddde4251-43ac-4def-b4d4-e1f9a2a7dc30" containerName="istio-proxy" Oct 30 21:16:45.129402 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:16:45.128476 5378 memory_manager.go:365] "RemoveStaleState removing state" podUID="3559349f-6be4-4ef6-8061-59c7d1a8b678" containerName="istio-proxy" Oct 30 21:16:45.133358 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:16:45.133337 5378 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-639b07-openshift-default-795b89858c-9q4rj" Oct 30 21:16:45.192823 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:16:45.192015 5378 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-639b07-openshift-default-dockercfg-65ccp\"" Oct 30 21:16:45.271597 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:16:45.263467 5378 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-639b07-openshift-default-795b89858c-9q4rj"] Oct 30 21:16:45.337238 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:16:45.290631 5378 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6614eb26-439d-464e-a7c2-b95744976905-workload-socket\") pod \"e2e-inf-b-639b07-openshift-default-795b89858c-9q4rj\" (UID: \"6614eb26-439d-464e-a7c2-b95744976905\") " pod="openshift-ingress/e2e-inf-b-639b07-openshift-default-795b89858c-9q4rj" Oct 30 21:16:45.337238 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:16:45.290762 5378 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6614eb26-439d-464e-a7c2-b95744976905-istiod-ca-cert\") pod \"e2e-inf-b-639b07-openshift-default-795b89858c-9q4rj\" (UID: \"6614eb26-439d-464e-a7c2-b95744976905\") " pod="openshift-ingress/e2e-inf-b-639b07-openshift-default-795b89858c-9q4rj" Oct 30 21:16:45.337238 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:16:45.290799 5378 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6614eb26-439d-464e-a7c2-b95744976905-istio-data\") pod \"e2e-inf-b-639b07-openshift-default-795b89858c-9q4rj\" (UID: \"6614eb26-439d-464e-a7c2-b95744976905\") " pod="openshift-ingress/e2e-inf-b-639b07-openshift-default-795b89858c-9q4rj" Oct 30 21:16:45.337238 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:16:45.290828 5378 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6614eb26-439d-464e-a7c2-b95744976905-workload-certs\") pod \"e2e-inf-b-639b07-openshift-default-795b89858c-9q4rj\" (UID: \"6614eb26-439d-464e-a7c2-b95744976905\") " pod="openshift-ingress/e2e-inf-b-639b07-openshift-default-795b89858c-9q4rj" Oct 30 21:16:45.337238 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:16:45.290923 5378 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c5br9\" (UniqueName: \"kubernetes.io/projected/6614eb26-439d-464e-a7c2-b95744976905-kube-api-access-c5br9\") pod \"e2e-inf-b-639b07-openshift-default-795b89858c-9q4rj\" (UID: \"6614eb26-439d-464e-a7c2-b95744976905\") " pod="openshift-ingress/e2e-inf-b-639b07-openshift-default-795b89858c-9q4rj" Oct 30 21:16:45.337238 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:16:45.291009 5378 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6614eb26-439d-464e-a7c2-b95744976905-credential-socket\") pod \"e2e-inf-b-639b07-openshift-default-795b89858c-9q4rj\" (UID: \"6614eb26-439d-464e-a7c2-b95744976905\") " pod="openshift-ingress/e2e-inf-b-639b07-openshift-default-795b89858c-9q4rj" Oct 30 21:16:45.337238 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:16:45.291040 5378 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6614eb26-439d-464e-a7c2-b95744976905-istio-podinfo\") pod \"e2e-inf-b-639b07-openshift-default-795b89858c-9q4rj\" (UID: \"6614eb26-439d-464e-a7c2-b95744976905\") " pod="openshift-ingress/e2e-inf-b-639b07-openshift-default-795b89858c-9q4rj" Oct 30 21:16:45.337238 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:16:45.291111 5378 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6614eb26-439d-464e-a7c2-b95744976905-istio-envoy\") pod \"e2e-inf-b-639b07-openshift-default-795b89858c-9q4rj\" (UID: \"6614eb26-439d-464e-a7c2-b95744976905\") " pod="openshift-ingress/e2e-inf-b-639b07-openshift-default-795b89858c-9q4rj" Oct 30 21:16:45.337238 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:16:45.291194 5378 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6614eb26-439d-464e-a7c2-b95744976905-istio-token\") pod \"e2e-inf-b-639b07-openshift-default-795b89858c-9q4rj\" (UID: \"6614eb26-439d-464e-a7c2-b95744976905\") " pod="openshift-ingress/e2e-inf-b-639b07-openshift-default-795b89858c-9q4rj" Oct 30 21:16:45.353106 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:16:45.352892 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:16:45.392558 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:16:45.392458 5378 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6614eb26-439d-464e-a7c2-b95744976905-istio-token\") pod \"e2e-inf-b-639b07-openshift-default-795b89858c-9q4rj\" (UID: \"6614eb26-439d-464e-a7c2-b95744976905\") " pod="openshift-ingress/e2e-inf-b-639b07-openshift-default-795b89858c-9q4rj" Oct 30 21:16:45.392558 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:16:45.392501 5378 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6614eb26-439d-464e-a7c2-b95744976905-workload-socket\") pod \"e2e-inf-b-639b07-openshift-default-795b89858c-9q4rj\" (UID: \"6614eb26-439d-464e-a7c2-b95744976905\") " pod="openshift-ingress/e2e-inf-b-639b07-openshift-default-795b89858c-9q4rj" Oct 30 21:16:45.392558 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:16:45.392536 5378 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6614eb26-439d-464e-a7c2-b95744976905-istiod-ca-cert\") pod \"e2e-inf-b-639b07-openshift-default-795b89858c-9q4rj\" (UID: \"6614eb26-439d-464e-a7c2-b95744976905\") " pod="openshift-ingress/e2e-inf-b-639b07-openshift-default-795b89858c-9q4rj" Oct 30 21:16:45.392558 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:16:45.392555 5378 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6614eb26-439d-464e-a7c2-b95744976905-istio-data\") pod \"e2e-inf-b-639b07-openshift-default-795b89858c-9q4rj\" (UID: \"6614eb26-439d-464e-a7c2-b95744976905\") " pod="openshift-ingress/e2e-inf-b-639b07-openshift-default-795b89858c-9q4rj" Oct 30 21:16:45.392964 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:16:45.392772 5378 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6614eb26-439d-464e-a7c2-b95744976905-workload-certs\") pod \"e2e-inf-b-639b07-openshift-default-795b89858c-9q4rj\" (UID: \"6614eb26-439d-464e-a7c2-b95744976905\") " pod="openshift-ingress/e2e-inf-b-639b07-openshift-default-795b89858c-9q4rj" Oct 30 21:16:45.392964 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:16:45.392917 5378 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c5br9\" (UniqueName: \"kubernetes.io/projected/6614eb26-439d-464e-a7c2-b95744976905-kube-api-access-c5br9\") pod \"e2e-inf-b-639b07-openshift-default-795b89858c-9q4rj\" (UID: \"6614eb26-439d-464e-a7c2-b95744976905\") " pod="openshift-ingress/e2e-inf-b-639b07-openshift-default-795b89858c-9q4rj" Oct 30 21:16:45.393080 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:16:45.393040 5378 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6614eb26-439d-464e-a7c2-b95744976905-workload-socket\") pod \"e2e-inf-b-639b07-openshift-default-795b89858c-9q4rj\" (UID: \"6614eb26-439d-464e-a7c2-b95744976905\") " pod="openshift-ingress/e2e-inf-b-639b07-openshift-default-795b89858c-9q4rj" Oct 30 21:16:45.393145 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:16:45.393110 5378 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6614eb26-439d-464e-a7c2-b95744976905-istio-data\") pod \"e2e-inf-b-639b07-openshift-default-795b89858c-9q4rj\" (UID: \"6614eb26-439d-464e-a7c2-b95744976905\") " pod="openshift-ingress/e2e-inf-b-639b07-openshift-default-795b89858c-9q4rj" Oct 30 21:16:45.393236 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:16:45.393207 5378 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6614eb26-439d-464e-a7c2-b95744976905-workload-certs\") pod \"e2e-inf-b-639b07-openshift-default-795b89858c-9q4rj\" (UID: \"6614eb26-439d-464e-a7c2-b95744976905\") " pod="openshift-ingress/e2e-inf-b-639b07-openshift-default-795b89858c-9q4rj" Oct 30 21:16:45.393370 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:16:45.393320 5378 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6614eb26-439d-464e-a7c2-b95744976905-credential-socket\") pod \"e2e-inf-b-639b07-openshift-default-795b89858c-9q4rj\" (UID: \"6614eb26-439d-464e-a7c2-b95744976905\") " pod="openshift-ingress/e2e-inf-b-639b07-openshift-default-795b89858c-9q4rj" Oct 30 21:16:45.393370 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:16:45.393352 5378 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6614eb26-439d-464e-a7c2-b95744976905-istiod-ca-cert\") pod \"e2e-inf-b-639b07-openshift-default-795b89858c-9q4rj\" (UID: \"6614eb26-439d-464e-a7c2-b95744976905\") " pod="openshift-ingress/e2e-inf-b-639b07-openshift-default-795b89858c-9q4rj" Oct 30 21:16:45.393501 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:16:45.393360 5378 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6614eb26-439d-464e-a7c2-b95744976905-istio-podinfo\") pod \"e2e-inf-b-639b07-openshift-default-795b89858c-9q4rj\" (UID: \"6614eb26-439d-464e-a7c2-b95744976905\") " pod="openshift-ingress/e2e-inf-b-639b07-openshift-default-795b89858c-9q4rj" Oct 30 21:16:45.393501 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:16:45.393452 5378 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6614eb26-439d-464e-a7c2-b95744976905-istio-envoy\") pod \"e2e-inf-b-639b07-openshift-default-795b89858c-9q4rj\" (UID: \"6614eb26-439d-464e-a7c2-b95744976905\") " pod="openshift-ingress/e2e-inf-b-639b07-openshift-default-795b89858c-9q4rj" Oct 30 21:16:45.393659 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:16:45.393632 5378 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6614eb26-439d-464e-a7c2-b95744976905-credential-socket\") pod \"e2e-inf-b-639b07-openshift-default-795b89858c-9q4rj\" (UID: \"6614eb26-439d-464e-a7c2-b95744976905\") " pod="openshift-ingress/e2e-inf-b-639b07-openshift-default-795b89858c-9q4rj" Oct 30 21:16:45.396057 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:16:45.396036 5378 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6614eb26-439d-464e-a7c2-b95744976905-istio-podinfo\") pod \"e2e-inf-b-639b07-openshift-default-795b89858c-9q4rj\" (UID: \"6614eb26-439d-464e-a7c2-b95744976905\") " pod="openshift-ingress/e2e-inf-b-639b07-openshift-default-795b89858c-9q4rj" Oct 30 21:16:45.396179 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:16:45.396130 5378 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6614eb26-439d-464e-a7c2-b95744976905-istio-envoy\") pod \"e2e-inf-b-639b07-openshift-default-795b89858c-9q4rj\" (UID: \"6614eb26-439d-464e-a7c2-b95744976905\") " pod="openshift-ingress/e2e-inf-b-639b07-openshift-default-795b89858c-9q4rj" Oct 30 21:16:45.452794 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:16:45.452763 5378 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6614eb26-439d-464e-a7c2-b95744976905-istio-token\") pod \"e2e-inf-b-639b07-openshift-default-795b89858c-9q4rj\" (UID: \"6614eb26-439d-464e-a7c2-b95744976905\") " pod="openshift-ingress/e2e-inf-b-639b07-openshift-default-795b89858c-9q4rj" Oct 30 21:16:45.452983 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:16:45.452841 5378 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c5br9\" (UniqueName: \"kubernetes.io/projected/6614eb26-439d-464e-a7c2-b95744976905-kube-api-access-c5br9\") pod \"e2e-inf-b-639b07-openshift-default-795b89858c-9q4rj\" (UID: \"6614eb26-439d-464e-a7c2-b95744976905\") " pod="openshift-ingress/e2e-inf-b-639b07-openshift-default-795b89858c-9q4rj" Oct 30 21:16:45.662995 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:16:45.662910 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:16:45.747190 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:16:45.747163 5378 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-639b07-openshift-default-795b89858c-9q4rj" Oct 30 21:16:45.823898 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:16:45.823722 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:16:46.209557 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:16:46.209524 5378 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-639b07-openshift-default-795b89858c-9q4rj"] Oct 30 21:16:46.210470 ip-10-0-138-149 kubenswrapper[5378]: W1030 21:16:46.210446 5378 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6614eb26_439d_464e_a7c2_b95744976905.slice/crio-34c44600a35387ef6b926480c71f4279df9258b1059604e0b044e9f8e3e80d0e WatchSource:0}: Error finding container 34c44600a35387ef6b926480c71f4279df9258b1059604e0b044e9f8e3e80d0e: Status 404 returned error can't find the container with id 34c44600a35387ef6b926480c71f4279df9258b1059604e0b044e9f8e3e80d0e Oct 30 21:16:46.213177 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:16:46.213139 5378 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 21:16:46.213264 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:16:46.213227 5378 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 21:16:46.213301 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:16:46.213269 5378 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 21:16:46.360107 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:16:46.360068 5378 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-639b07-openshift-default-795b89858c-9q4rj" event={"ID":"6614eb26-439d-464e-a7c2-b95744976905","Type":"ContainerStarted","Data":"a76a56d4d1a7618806474066a486dd2bd3c8f5287141fddadcc8dfb270bdd372"} Oct 30 21:16:46.360308 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:16:46.360124 5378 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-639b07-openshift-default-795b89858c-9q4rj" event={"ID":"6614eb26-439d-464e-a7c2-b95744976905","Type":"ContainerStarted","Data":"34c44600a35387ef6b926480c71f4279df9258b1059604e0b044e9f8e3e80d0e"} Oct 30 21:16:46.488773 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:16:46.488662 5378 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-639b07-openshift-default-795b89858c-9q4rj" podStartSLOduration=1.488645841 podStartE2EDuration="1.488645841s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:16:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:16:46.487327036 +0000 UTC m=+2296.915440955" watchObservedRunningTime="2026-10-30 21:16:46.488645841 +0000 UTC m=+2296.916759762" Oct 30 21:16:46.747900 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:16:46.747821 5378 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-639b07-openshift-default-795b89858c-9q4rj" Oct 30 21:16:47.754582 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:16:47.754534 5378 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-639b07-openshift-default-795b89858c-9q4rj" Oct 30 21:16:48.373060 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:16:48.373028 5378 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-639b07-openshift-default-795b89858c-9q4rj" Oct 30 21:16:48.374035 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:16:48.374015 5378 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-639b07-openshift-default-795b89858c-9q4rj" Oct 30 21:16:48.838636 ip-10-0-138-149 kubenswrapper[5378]: E1030 21:16:48.838533 5378 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q6mx7" containerName="extractor" Oct 30 21:16:49.130680 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:16:49.130649 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:16:49.297171 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:16:49.297139 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:16:49.509699 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:16:49.509612 5378 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w5-2b425d-74d89cbc4b-rdcqc"] Oct 30 21:16:49.648958 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:16:49.648918 5378 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w5-2b425d-59c9dc557f-sd78q"] Oct 30 21:16:49.683354 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:16:49.683324 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:16:49.833847 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:16:49.833754 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:16:50.017297 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:16:50.017247 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:16:50.162914 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:16:50.161759 5378 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-769df7846c-r5j2r"] Oct 30 21:16:50.168481 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:16:50.168450 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:16:50.171680 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:16:50.171647 5378 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-r5j2r" Oct 30 21:16:50.239142 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:16:50.239109 5378 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-769df7846c-r5j2r"] Oct 30 21:16:50.264196 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:16:50.264155 5378 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/04d5e742-36a9-41a8-9bb7-c9c8808d74cc-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-769df7846c-r5j2r\" (UID: \"04d5e742-36a9-41a8-9bb7-c9c8808d74cc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-r5j2r" Oct 30 21:16:50.264417 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:16:50.264209 5378 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/04d5e742-36a9-41a8-9bb7-c9c8808d74cc-istio-envoy\") pod \"maas-default-gateway-openshift-default-769df7846c-r5j2r\" (UID: \"04d5e742-36a9-41a8-9bb7-c9c8808d74cc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-r5j2r" Oct 30 21:16:50.264417 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:16:50.264376 5378 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/04d5e742-36a9-41a8-9bb7-c9c8808d74cc-workload-certs\") pod \"maas-default-gateway-openshift-default-769df7846c-r5j2r\" (UID: \"04d5e742-36a9-41a8-9bb7-c9c8808d74cc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-r5j2r" Oct 30 21:16:50.264609 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:16:50.264488 5378 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/04d5e742-36a9-41a8-9bb7-c9c8808d74cc-istio-podinfo\") pod \"maas-default-gateway-openshift-default-769df7846c-r5j2r\" (UID: \"04d5e742-36a9-41a8-9bb7-c9c8808d74cc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-r5j2r" Oct 30 21:16:50.264609 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:16:50.264580 5378 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/04d5e742-36a9-41a8-9bb7-c9c8808d74cc-workload-socket\") pod \"maas-default-gateway-openshift-default-769df7846c-r5j2r\" (UID: \"04d5e742-36a9-41a8-9bb7-c9c8808d74cc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-r5j2r" Oct 30 21:16:50.264747 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:16:50.264657 5378 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lp65v\" (UniqueName: \"kubernetes.io/projected/04d5e742-36a9-41a8-9bb7-c9c8808d74cc-kube-api-access-lp65v\") pod \"maas-default-gateway-openshift-default-769df7846c-r5j2r\" (UID: \"04d5e742-36a9-41a8-9bb7-c9c8808d74cc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-r5j2r" Oct 30 21:16:50.264747 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:16:50.264695 5378 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/04d5e742-36a9-41a8-9bb7-c9c8808d74cc-istio-data\") pod \"maas-default-gateway-openshift-default-769df7846c-r5j2r\" (UID: \"04d5e742-36a9-41a8-9bb7-c9c8808d74cc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-r5j2r" Oct 30 21:16:50.264929 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:16:50.264809 5378 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/04d5e742-36a9-41a8-9bb7-c9c8808d74cc-istio-token\") pod \"maas-default-gateway-openshift-default-769df7846c-r5j2r\" (UID: \"04d5e742-36a9-41a8-9bb7-c9c8808d74cc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-r5j2r" Oct 30 21:16:50.264929 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:16:50.264835 5378 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/04d5e742-36a9-41a8-9bb7-c9c8808d74cc-credential-socket\") pod \"maas-default-gateway-openshift-default-769df7846c-r5j2r\" (UID: \"04d5e742-36a9-41a8-9bb7-c9c8808d74cc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-r5j2r" Oct 30 21:16:50.365887 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:16:50.365840 5378 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/04d5e742-36a9-41a8-9bb7-c9c8808d74cc-istio-token\") pod \"maas-default-gateway-openshift-default-769df7846c-r5j2r\" (UID: \"04d5e742-36a9-41a8-9bb7-c9c8808d74cc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-r5j2r" Oct 30 21:16:50.366073 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:16:50.365894 5378 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/04d5e742-36a9-41a8-9bb7-c9c8808d74cc-credential-socket\") pod \"maas-default-gateway-openshift-default-769df7846c-r5j2r\" (UID: \"04d5e742-36a9-41a8-9bb7-c9c8808d74cc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-r5j2r" Oct 30 21:16:50.366073 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:16:50.365937 5378 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/04d5e742-36a9-41a8-9bb7-c9c8808d74cc-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-769df7846c-r5j2r\" (UID: \"04d5e742-36a9-41a8-9bb7-c9c8808d74cc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-r5j2r" Oct 30 21:16:50.366073 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:16:50.365979 5378 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/04d5e742-36a9-41a8-9bb7-c9c8808d74cc-istio-envoy\") pod \"maas-default-gateway-openshift-default-769df7846c-r5j2r\" (UID: \"04d5e742-36a9-41a8-9bb7-c9c8808d74cc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-r5j2r" Oct 30 21:16:50.366073 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:16:50.366019 5378 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/04d5e742-36a9-41a8-9bb7-c9c8808d74cc-workload-certs\") pod \"maas-default-gateway-openshift-default-769df7846c-r5j2r\" (UID: \"04d5e742-36a9-41a8-9bb7-c9c8808d74cc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-r5j2r" Oct 30 21:16:50.366326 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:16:50.366089 5378 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/04d5e742-36a9-41a8-9bb7-c9c8808d74cc-istio-podinfo\") pod \"maas-default-gateway-openshift-default-769df7846c-r5j2r\" (UID: \"04d5e742-36a9-41a8-9bb7-c9c8808d74cc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-r5j2r" Oct 30 21:16:50.366326 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:16:50.366145 5378 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/04d5e742-36a9-41a8-9bb7-c9c8808d74cc-workload-socket\") pod \"maas-default-gateway-openshift-default-769df7846c-r5j2r\" (UID: \"04d5e742-36a9-41a8-9bb7-c9c8808d74cc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-r5j2r" Oct 30 21:16:50.366326 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:16:50.366210 5378 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lp65v\" (UniqueName: \"kubernetes.io/projected/04d5e742-36a9-41a8-9bb7-c9c8808d74cc-kube-api-access-lp65v\") pod \"maas-default-gateway-openshift-default-769df7846c-r5j2r\" (UID: \"04d5e742-36a9-41a8-9bb7-c9c8808d74cc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-r5j2r" Oct 30 21:16:50.366326 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:16:50.366239 5378 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/04d5e742-36a9-41a8-9bb7-c9c8808d74cc-istio-data\") pod \"maas-default-gateway-openshift-default-769df7846c-r5j2r\" (UID: \"04d5e742-36a9-41a8-9bb7-c9c8808d74cc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-r5j2r" Oct 30 21:16:50.366549 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:16:50.366449 5378 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/04d5e742-36a9-41a8-9bb7-c9c8808d74cc-credential-socket\") pod \"maas-default-gateway-openshift-default-769df7846c-r5j2r\" (UID: \"04d5e742-36a9-41a8-9bb7-c9c8808d74cc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-r5j2r" Oct 30 21:16:50.366549 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:16:50.366510 5378 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/04d5e742-36a9-41a8-9bb7-c9c8808d74cc-workload-certs\") pod \"maas-default-gateway-openshift-default-769df7846c-r5j2r\" (UID: \"04d5e742-36a9-41a8-9bb7-c9c8808d74cc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-r5j2r" Oct 30 21:16:50.366695 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:16:50.366577 5378 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/04d5e742-36a9-41a8-9bb7-c9c8808d74cc-workload-socket\") pod \"maas-default-gateway-openshift-default-769df7846c-r5j2r\" (UID: \"04d5e742-36a9-41a8-9bb7-c9c8808d74cc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-r5j2r" Oct 30 21:16:50.366767 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:16:50.366728 5378 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/04d5e742-36a9-41a8-9bb7-c9c8808d74cc-istio-data\") pod \"maas-default-gateway-openshift-default-769df7846c-r5j2r\" (UID: \"04d5e742-36a9-41a8-9bb7-c9c8808d74cc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-r5j2r" Oct 30 21:16:50.366831 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:16:50.366809 5378 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/04d5e742-36a9-41a8-9bb7-c9c8808d74cc-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-769df7846c-r5j2r\" (UID: \"04d5e742-36a9-41a8-9bb7-c9c8808d74cc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-r5j2r" Oct 30 21:16:50.368950 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:16:50.368933 5378 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/04d5e742-36a9-41a8-9bb7-c9c8808d74cc-istio-envoy\") pod \"maas-default-gateway-openshift-default-769df7846c-r5j2r\" (UID: \"04d5e742-36a9-41a8-9bb7-c9c8808d74cc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-r5j2r" Oct 30 21:16:50.369274 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:16:50.369254 5378 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/04d5e742-36a9-41a8-9bb7-c9c8808d74cc-istio-podinfo\") pod \"maas-default-gateway-openshift-default-769df7846c-r5j2r\" (UID: \"04d5e742-36a9-41a8-9bb7-c9c8808d74cc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-r5j2r" Oct 30 21:16:50.432957 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:16:50.432887 5378 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/04d5e742-36a9-41a8-9bb7-c9c8808d74cc-istio-token\") pod \"maas-default-gateway-openshift-default-769df7846c-r5j2r\" (UID: \"04d5e742-36a9-41a8-9bb7-c9c8808d74cc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-r5j2r" Oct 30 21:16:50.435318 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:16:50.435289 5378 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lp65v\" (UniqueName: \"kubernetes.io/projected/04d5e742-36a9-41a8-9bb7-c9c8808d74cc-kube-api-access-lp65v\") pod \"maas-default-gateway-openshift-default-769df7846c-r5j2r\" (UID: \"04d5e742-36a9-41a8-9bb7-c9c8808d74cc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-r5j2r" Oct 30 21:16:50.491593 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:16:50.491521 5378 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-r5j2r" Oct 30 21:16:50.926561 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:16:50.926525 5378 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-769df7846c-r5j2r"] Oct 30 21:16:50.963553 ip-10-0-138-149 kubenswrapper[5378]: W1030 21:16:50.963518 5378 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod04d5e742_36a9_41a8_9bb7_c9c8808d74cc.slice/crio-94734d844da343f0b612d6fc696ffa29b3a3cea6c636cc6b5e0b9ba48ff23f1b WatchSource:0}: Error finding container 94734d844da343f0b612d6fc696ffa29b3a3cea6c636cc6b5e0b9ba48ff23f1b: Status 404 returned error can't find the container with id 94734d844da343f0b612d6fc696ffa29b3a3cea6c636cc6b5e0b9ba48ff23f1b Oct 30 21:16:50.966154 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:16:50.966120 5378 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 21:16:50.966262 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:16:50.966202 5378 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 21:16:50.966262 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:16:50.966245 5378 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 21:16:51.175868 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:16:51.175832 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:16:51.336483 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:16:51.336381 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:16:51.395072 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:16:51.395037 5378 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-r5j2r" event={"ID":"04d5e742-36a9-41a8-9bb7-c9c8808d74cc","Type":"ContainerStarted","Data":"7bdaccbcc89ad3f46ed294c698f27162cbb4a16505169bcb29858637febdb929"} Oct 30 21:16:51.395259 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:16:51.395093 5378 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-r5j2r" event={"ID":"04d5e742-36a9-41a8-9bb7-c9c8808d74cc","Type":"ContainerStarted","Data":"94734d844da343f0b612d6fc696ffa29b3a3cea6c636cc6b5e0b9ba48ff23f1b"} Oct 30 21:16:51.493296 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:16:51.493241 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:16:51.495200 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:16:51.495175 5378 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-r5j2r" Oct 30 21:16:51.525250 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:16:51.525191 5378 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-r5j2r" podStartSLOduration=1.525174708 podStartE2EDuration="1.525174708s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:16:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:16:51.522128352 +0000 UTC m=+2301.950242270" watchObservedRunningTime="2026-10-30 21:16:51.525174708 +0000 UTC m=+2301.953288734" Oct 30 21:16:52.493961 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:16:52.493924 5378 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-769df7846c-r5j2r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.161:15021/healthz/ready\": dial tcp 10.133.0.161:15021: connect: connection refused" start-of-body= Oct 30 21:16:52.494401 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:16:52.493982 5378 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-r5j2r" podUID="04d5e742-36a9-41a8-9bb7-c9c8808d74cc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.161:15021/healthz/ready\": dial tcp 10.133.0.161:15021: connect: connection refused" Oct 30 21:16:53.492119 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:16:53.492079 5378 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-769df7846c-r5j2r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.161:15021/healthz/ready\": dial tcp 10.133.0.161:15021: connect: connection refused" start-of-body= Oct 30 21:16:53.492306 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:16:53.492148 5378 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-r5j2r" podUID="04d5e742-36a9-41a8-9bb7-c9c8808d74cc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.161:15021/healthz/ready\": dial tcp 10.133.0.161:15021: connect: connection refused" Oct 30 21:16:54.492298 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:16:54.492257 5378 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-769df7846c-r5j2r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.161:15021/healthz/ready\": dial tcp 10.133.0.161:15021: connect: connection refused" start-of-body= Oct 30 21:16:54.492780 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:16:54.492331 5378 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-r5j2r" podUID="04d5e742-36a9-41a8-9bb7-c9c8808d74cc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.161:15021/healthz/ready\": dial tcp 10.133.0.161:15021: connect: connection refused" Oct 30 21:16:54.501286 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:16:54.501249 5378 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-cb6404-5f8b4fc56b-twpl5"] Oct 30 21:16:54.510533 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:16:54.510496 5378 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w5-2b425d-74d89cbc4b-rdcqc" podUID="2d52d866-9725-4d46-9aa5-adcc43e87bbd" containerName="payload-processing" containerID="cri-o://2bb51a5abb388bb86b1f21c45ba6d808e1a095613b591fbaa4c4213cffa4d783" gracePeriod=25 Oct 30 21:16:54.649651 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:16:54.649605 5378 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-2b425d-59c9dc557f-sd78q" podUID="7715f68b-9673-4119-a43d-05372a096116" containerName="payload-pre-processing" containerID="cri-o://984fb00d89aa672a0cf010cd330a7da88d6a067932dfb5b87c0f9f5843f90b01" gracePeriod=25 Oct 30 21:16:54.880406 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:16:54.880368 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:16:55.024722 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:16:55.024685 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:16:55.492163 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:16:55.492124 5378 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-769df7846c-r5j2r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.161:15021/healthz/ready\": dial tcp 10.133.0.161:15021: connect: connection refused" start-of-body= Oct 30 21:16:55.492338 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:16:55.492187 5378 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-r5j2r" podUID="04d5e742-36a9-41a8-9bb7-c9c8808d74cc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.161:15021/healthz/ready\": dial tcp 10.133.0.161:15021: connect: connection refused" Oct 30 21:16:55.541196 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:16:55.541160 5378 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w5-2b425d-74d89cbc4b-rdcqc" podUID="2d52d866-9725-4d46-9aa5-adcc43e87bbd" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 21:16:55.541344 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:16:55.541169 5378 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-2b425d-59c9dc557f-sd78q" podUID="7715f68b-9673-4119-a43d-05372a096116" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 21:16:56.492134 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:16:56.492092 5378 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-769df7846c-r5j2r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.161:15021/healthz/ready\": dial tcp 10.133.0.161:15021: connect: connection refused" start-of-body= Oct 30 21:16:56.492352 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:16:56.492159 5378 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-r5j2r" podUID="04d5e742-36a9-41a8-9bb7-c9c8808d74cc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.161:15021/healthz/ready\": dial tcp 10.133.0.161:15021: connect: connection refused" Oct 30 21:16:56.817830 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:16:56.817730 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:16:57.501284 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:16:57.501243 5378 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-769df7846c-r5j2r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.161:15021/healthz/ready\": dial tcp 10.133.0.161:15021: connect: connection refused" start-of-body= Oct 30 21:16:57.501811 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:16:57.501310 5378 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-r5j2r" podUID="04d5e742-36a9-41a8-9bb7-c9c8808d74cc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.161:15021/healthz/ready\": dial tcp 10.133.0.161:15021: connect: connection refused" Oct 30 21:16:58.492859 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:16:58.492825 5378 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-769df7846c-r5j2r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.161:15021/healthz/ready\": dial tcp 10.133.0.161:15021: connect: connection refused" start-of-body= Oct 30 21:16:58.493110 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:16:58.492883 5378 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-r5j2r" podUID="04d5e742-36a9-41a8-9bb7-c9c8808d74cc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.161:15021/healthz/ready\": dial tcp 10.133.0.161:15021: connect: connection refused" Oct 30 21:16:58.504852 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:16:58.504821 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:16:58.723878 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:16:58.723843 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:16:59.502165 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:16:59.502110 5378 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-b-cb6404-5f8b4fc56b-twpl5" podUID="3f616067-e750-4f5c-8fd4-d5a9cf5cf9ec" containerName="payload-processing" containerID="cri-o://b34c7e2ac1472859f0876d7263bbf4bd775388913a492eaeda5fb814a4dfc1a5" gracePeriod=25 Oct 30 21:16:59.811409 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:16:59.811384 5378 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-cb6404-5f8b4fc56b-twpl5" Oct 30 21:16:59.854230 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:16:59.854197 5378 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-r5j2r" Oct 30 21:16:59.854531 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:16:59.854510 5378 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-r5j2r" Oct 30 21:16:59.855819 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:16:59.855798 5378 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-r5j2r" Oct 30 21:16:59.897888 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:16:59.897850 5378 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3f616067-e750-4f5c-8fd4-d5a9cf5cf9ec-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3f616067-e750-4f5c-8fd4-d5a9cf5cf9ec-plugins-config-volume\") pod \"3f616067-e750-4f5c-8fd4-d5a9cf5cf9ec\" (UID: \"3f616067-e750-4f5c-8fd4-d5a9cf5cf9ec\") " Oct 30 21:16:59.898081 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:16:59.897996 5378 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3f616067-e750-4f5c-8fd4-d5a9cf5cf9ec-kube-api-access-rdqh9\" (UniqueName: \"kubernetes.io/projected/3f616067-e750-4f5c-8fd4-d5a9cf5cf9ec-kube-api-access-rdqh9\") pod \"3f616067-e750-4f5c-8fd4-d5a9cf5cf9ec\" (UID: \"3f616067-e750-4f5c-8fd4-d5a9cf5cf9ec\") " Oct 30 21:16:59.898259 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:16:59.898237 5378 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3f616067-e750-4f5c-8fd4-d5a9cf5cf9ec-plugins-config-volume" pod "3f616067-e750-4f5c-8fd4-d5a9cf5cf9ec" (UID: "3f616067-e750-4f5c-8fd4-d5a9cf5cf9ec"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:16:59.898549 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:16:59.898523 5378 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3f616067-e750-4f5c-8fd4-d5a9cf5cf9ec-plugins-config-volume\") on node \"ip-10-0-138-149.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:16:59.900085 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:16:59.900058 5378 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3f616067-e750-4f5c-8fd4-d5a9cf5cf9ec-kube-api-access-rdqh9" pod "3f616067-e750-4f5c-8fd4-d5a9cf5cf9ec" (UID: "3f616067-e750-4f5c-8fd4-d5a9cf5cf9ec"). InnerVolumeSpecName "kube-api-access-rdqh9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:17:00.000515 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:17:00.000478 5378 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rdqh9\" (UniqueName: \"kubernetes.io/projected/3f616067-e750-4f5c-8fd4-d5a9cf5cf9ec-kube-api-access-rdqh9\") on node \"ip-10-0-138-149.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:17:00.482169 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:17:00.482137 5378 generic.go:356] "Generic (PLEG): container finished" podID="3f616067-e750-4f5c-8fd4-d5a9cf5cf9ec" containerID="b34c7e2ac1472859f0876d7263bbf4bd775388913a492eaeda5fb814a4dfc1a5" exitCode=0 Oct 30 21:17:00.482393 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:17:00.482186 5378 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-cb6404-5f8b4fc56b-twpl5" Oct 30 21:17:00.482393 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:17:00.482236 5378 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-cb6404-5f8b4fc56b-twpl5" event={"ID":"3f616067-e750-4f5c-8fd4-d5a9cf5cf9ec","Type":"ContainerDied","Data":"b34c7e2ac1472859f0876d7263bbf4bd775388913a492eaeda5fb814a4dfc1a5"} Oct 30 21:17:00.482393 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:17:00.482303 5378 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-cb6404-5f8b4fc56b-twpl5" event={"ID":"3f616067-e750-4f5c-8fd4-d5a9cf5cf9ec","Type":"ContainerDied","Data":"1966d90e947d46dbf86e4762183c5f682ecccbc99e621d0ee0f27421b4bb89c0"} Oct 30 21:17:00.482393 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:17:00.482345 5378 scope.go:122] "RemoveContainer" containerID="b34c7e2ac1472859f0876d7263bbf4bd775388913a492eaeda5fb814a4dfc1a5" Oct 30 21:17:00.494313 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:17:00.494294 5378 scope.go:122] "RemoveContainer" containerID="b34c7e2ac1472859f0876d7263bbf4bd775388913a492eaeda5fb814a4dfc1a5" Oct 30 21:17:00.494615 ip-10-0-138-149 kubenswrapper[5378]: E1030 21:17:00.494595 5378 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b34c7e2ac1472859f0876d7263bbf4bd775388913a492eaeda5fb814a4dfc1a5\": container with ID starting with b34c7e2ac1472859f0876d7263bbf4bd775388913a492eaeda5fb814a4dfc1a5 not found: ID does not exist" containerID="b34c7e2ac1472859f0876d7263bbf4bd775388913a492eaeda5fb814a4dfc1a5" Oct 30 21:17:00.494691 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:17:00.494623 5378 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b34c7e2ac1472859f0876d7263bbf4bd775388913a492eaeda5fb814a4dfc1a5"} err="failed to get container status \"b34c7e2ac1472859f0876d7263bbf4bd775388913a492eaeda5fb814a4dfc1a5\": rpc error: code = NotFound desc = could not find container \"b34c7e2ac1472859f0876d7263bbf4bd775388913a492eaeda5fb814a4dfc1a5\": container with ID starting with b34c7e2ac1472859f0876d7263bbf4bd775388913a492eaeda5fb814a4dfc1a5 not found: ID does not exist" Oct 30 21:17:00.648378 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:17:00.648348 5378 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-cb6404-5f8b4fc56b-twpl5"] Oct 30 21:17:00.651554 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:17:00.651528 5378 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-cb6404-5f8b4fc56b-twpl5"] Oct 30 21:17:01.560471 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:17:01.560439 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:17:01.742797 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:17:01.742766 5378 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3f616067-e750-4f5c-8fd4-d5a9cf5cf9ec" path="/var/lib/kubelet/pods/3f616067-e750-4f5c-8fd4-d5a9cf5cf9ec/volumes" Oct 30 21:17:05.540758 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:17:05.540715 5378 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-2b425d-59c9dc557f-sd78q" podUID="7715f68b-9673-4119-a43d-05372a096116" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 21:17:05.541268 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:17:05.540721 5378 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w5-2b425d-74d89cbc4b-rdcqc" podUID="2d52d866-9725-4d46-9aa5-adcc43e87bbd" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 21:17:06.372974 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:17:06.372936 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:17:11.413819 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:17:11.413776 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:17:14.794324 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:17:14.794292 5378 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w5-2b425d-74d89cbc4b-rdcqc" Oct 30 21:17:14.902231 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:17:14.902203 5378 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2d52d866-9725-4d46-9aa5-adcc43e87bbd-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2d52d866-9725-4d46-9aa5-adcc43e87bbd-plugins-config-volume\") pod \"2d52d866-9725-4d46-9aa5-adcc43e87bbd\" (UID: \"2d52d866-9725-4d46-9aa5-adcc43e87bbd\") " Oct 30 21:17:14.902393 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:17:14.902307 5378 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2d52d866-9725-4d46-9aa5-adcc43e87bbd-kube-api-access-9lrmh\" (UniqueName: \"kubernetes.io/projected/2d52d866-9725-4d46-9aa5-adcc43e87bbd-kube-api-access-9lrmh\") pod \"2d52d866-9725-4d46-9aa5-adcc43e87bbd\" (UID: \"2d52d866-9725-4d46-9aa5-adcc43e87bbd\") " Oct 30 21:17:14.902612 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:17:14.902550 5378 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2d52d866-9725-4d46-9aa5-adcc43e87bbd-plugins-config-volume" pod "2d52d866-9725-4d46-9aa5-adcc43e87bbd" (UID: "2d52d866-9725-4d46-9aa5-adcc43e87bbd"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:17:14.903234 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:17:14.903200 5378 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2d52d866-9725-4d46-9aa5-adcc43e87bbd-plugins-config-volume\") on node \"ip-10-0-138-149.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:17:14.904339 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:17:14.904319 5378 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2d52d866-9725-4d46-9aa5-adcc43e87bbd-kube-api-access-9lrmh" pod "2d52d866-9725-4d46-9aa5-adcc43e87bbd" (UID: "2d52d866-9725-4d46-9aa5-adcc43e87bbd"). InnerVolumeSpecName "kube-api-access-9lrmh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:17:14.919629 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:17:14.919601 5378 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-2b425d-59c9dc557f-sd78q" Oct 30 21:17:15.004752 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:17:15.004722 5378 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7715f68b-9673-4119-a43d-05372a096116-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7715f68b-9673-4119-a43d-05372a096116-plugins-config-volume\") pod \"7715f68b-9673-4119-a43d-05372a096116\" (UID: \"7715f68b-9673-4119-a43d-05372a096116\") " Oct 30 21:17:15.004937 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:17:15.004868 5378 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7715f68b-9673-4119-a43d-05372a096116-kube-api-access-vcs7j\" (UniqueName: \"kubernetes.io/projected/7715f68b-9673-4119-a43d-05372a096116-kube-api-access-vcs7j\") pod \"7715f68b-9673-4119-a43d-05372a096116\" (UID: \"7715f68b-9673-4119-a43d-05372a096116\") " Oct 30 21:17:15.005144 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:17:15.005121 5378 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7715f68b-9673-4119-a43d-05372a096116-plugins-config-volume" pod "7715f68b-9673-4119-a43d-05372a096116" (UID: "7715f68b-9673-4119-a43d-05372a096116"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:17:15.005318 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:17:15.005303 5378 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9lrmh\" (UniqueName: \"kubernetes.io/projected/2d52d866-9725-4d46-9aa5-adcc43e87bbd-kube-api-access-9lrmh\") on node \"ip-10-0-138-149.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:17:15.005377 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:17:15.005321 5378 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7715f68b-9673-4119-a43d-05372a096116-plugins-config-volume\") on node \"ip-10-0-138-149.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:17:15.007194 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:17:15.007170 5378 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7715f68b-9673-4119-a43d-05372a096116-kube-api-access-vcs7j" pod "7715f68b-9673-4119-a43d-05372a096116" (UID: "7715f68b-9673-4119-a43d-05372a096116"). InnerVolumeSpecName "kube-api-access-vcs7j". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:17:15.107079 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:17:15.107032 5378 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vcs7j\" (UniqueName: \"kubernetes.io/projected/7715f68b-9673-4119-a43d-05372a096116-kube-api-access-vcs7j\") on node \"ip-10-0-138-149.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:17:15.581751 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:17:15.581718 5378 generic.go:356] "Generic (PLEG): container finished" podID="7715f68b-9673-4119-a43d-05372a096116" containerID="984fb00d89aa672a0cf010cd330a7da88d6a067932dfb5b87c0f9f5843f90b01" exitCode=0 Oct 30 21:17:15.581949 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:17:15.581763 5378 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-2b425d-59c9dc557f-sd78q" event={"ID":"7715f68b-9673-4119-a43d-05372a096116","Type":"ContainerDied","Data":"984fb00d89aa672a0cf010cd330a7da88d6a067932dfb5b87c0f9f5843f90b01"} Oct 30 21:17:15.581949 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:17:15.581785 5378 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-2b425d-59c9dc557f-sd78q" Oct 30 21:17:15.581949 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:17:15.581853 5378 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-2b425d-59c9dc557f-sd78q" event={"ID":"7715f68b-9673-4119-a43d-05372a096116","Type":"ContainerDied","Data":"a2f226d103b2f262c28ca9cb80de67103a06892eb0938155e3068f68ac6cc8d3"} Oct 30 21:17:15.581949 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:17:15.581885 5378 scope.go:122] "RemoveContainer" containerID="984fb00d89aa672a0cf010cd330a7da88d6a067932dfb5b87c0f9f5843f90b01" Oct 30 21:17:15.583960 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:17:15.583935 5378 generic.go:356] "Generic (PLEG): container finished" podID="2d52d866-9725-4d46-9aa5-adcc43e87bbd" containerID="2bb51a5abb388bb86b1f21c45ba6d808e1a095613b591fbaa4c4213cffa4d783" exitCode=0 Oct 30 21:17:15.584105 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:17:15.583987 5378 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w5-2b425d-74d89cbc4b-rdcqc" Oct 30 21:17:15.584105 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:17:15.584004 5378 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w5-2b425d-74d89cbc4b-rdcqc" event={"ID":"2d52d866-9725-4d46-9aa5-adcc43e87bbd","Type":"ContainerDied","Data":"2bb51a5abb388bb86b1f21c45ba6d808e1a095613b591fbaa4c4213cffa4d783"} Oct 30 21:17:15.584105 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:17:15.584049 5378 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w5-2b425d-74d89cbc4b-rdcqc" event={"ID":"2d52d866-9725-4d46-9aa5-adcc43e87bbd","Type":"ContainerDied","Data":"127a3deaf6fc4aee396aee0e233b8b59463ba110ab61995339f74d392d23e0a3"} Oct 30 21:17:15.593153 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:17:15.593132 5378 scope.go:122] "RemoveContainer" containerID="984fb00d89aa672a0cf010cd330a7da88d6a067932dfb5b87c0f9f5843f90b01" Oct 30 21:17:15.593532 ip-10-0-138-149 kubenswrapper[5378]: E1030 21:17:15.593506 5378 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"984fb00d89aa672a0cf010cd330a7da88d6a067932dfb5b87c0f9f5843f90b01\": container with ID starting with 984fb00d89aa672a0cf010cd330a7da88d6a067932dfb5b87c0f9f5843f90b01 not found: ID does not exist" containerID="984fb00d89aa672a0cf010cd330a7da88d6a067932dfb5b87c0f9f5843f90b01" Oct 30 21:17:15.593700 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:17:15.593540 5378 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"984fb00d89aa672a0cf010cd330a7da88d6a067932dfb5b87c0f9f5843f90b01"} err="failed to get container status \"984fb00d89aa672a0cf010cd330a7da88d6a067932dfb5b87c0f9f5843f90b01\": rpc error: code = NotFound desc = could not find container \"984fb00d89aa672a0cf010cd330a7da88d6a067932dfb5b87c0f9f5843f90b01\": container with ID starting with 984fb00d89aa672a0cf010cd330a7da88d6a067932dfb5b87c0f9f5843f90b01 not found: ID does not exist" Oct 30 21:17:15.593700 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:17:15.593559 5378 scope.go:122] "RemoveContainer" containerID="2bb51a5abb388bb86b1f21c45ba6d808e1a095613b591fbaa4c4213cffa4d783" Oct 30 21:17:15.603432 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:17:15.603411 5378 scope.go:122] "RemoveContainer" containerID="2bb51a5abb388bb86b1f21c45ba6d808e1a095613b591fbaa4c4213cffa4d783" Oct 30 21:17:15.603751 ip-10-0-138-149 kubenswrapper[5378]: E1030 21:17:15.603727 5378 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2bb51a5abb388bb86b1f21c45ba6d808e1a095613b591fbaa4c4213cffa4d783\": container with ID starting with 2bb51a5abb388bb86b1f21c45ba6d808e1a095613b591fbaa4c4213cffa4d783 not found: ID does not exist" containerID="2bb51a5abb388bb86b1f21c45ba6d808e1a095613b591fbaa4c4213cffa4d783" Oct 30 21:17:15.603829 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:17:15.603758 5378 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2bb51a5abb388bb86b1f21c45ba6d808e1a095613b591fbaa4c4213cffa4d783"} err="failed to get container status \"2bb51a5abb388bb86b1f21c45ba6d808e1a095613b591fbaa4c4213cffa4d783\": rpc error: code = NotFound desc = could not find container \"2bb51a5abb388bb86b1f21c45ba6d808e1a095613b591fbaa4c4213cffa4d783\": container with ID starting with 2bb51a5abb388bb86b1f21c45ba6d808e1a095613b591fbaa4c4213cffa4d783 not found: ID does not exist" Oct 30 21:17:15.747826 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:17:15.747786 5378 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w5-2b425d-74d89cbc4b-rdcqc"] Oct 30 21:17:15.798306 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:17:15.798219 5378 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w5-2b425d-74d89cbc4b-rdcqc"] Oct 30 21:17:15.908271 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:17:15.908236 5378 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w5-2b425d-59c9dc557f-sd78q"] Oct 30 21:17:15.916064 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:17:15.916029 5378 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w5-2b425d-59c9dc557f-sd78q"] Oct 30 21:17:15.928944 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:17:15.928913 5378 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-08df98-dc45f5c6-6hbng"] Oct 30 21:17:15.929530 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:17:15.929517 5378 cpu_manager.go:414] "containerMap: removing container" podUID="2d52d866-9725-4d46-9aa5-adcc43e87bbd" containerName="payload-processing" Oct 30 21:17:15.929611 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:17:15.929532 5378 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2d52d866-9725-4d46-9aa5-adcc43e87bbd" containerName="payload-processing" Oct 30 21:17:15.929611 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:17:15.929553 5378 cpu_manager.go:414] "containerMap: removing container" podUID="7715f68b-9673-4119-a43d-05372a096116" containerName="payload-pre-processing" Oct 30 21:17:15.929611 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:17:15.929580 5378 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7715f68b-9673-4119-a43d-05372a096116" containerName="payload-pre-processing" Oct 30 21:17:15.929611 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:17:15.929609 5378 cpu_manager.go:414] "containerMap: removing container" podUID="3f616067-e750-4f5c-8fd4-d5a9cf5cf9ec" containerName="payload-processing" Oct 30 21:17:15.929762 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:17:15.929614 5378 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3f616067-e750-4f5c-8fd4-d5a9cf5cf9ec" containerName="payload-processing" Oct 30 21:17:15.929803 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:17:15.929796 5378 memory_manager.go:365] "RemoveStaleState removing state" podUID="2d52d866-9725-4d46-9aa5-adcc43e87bbd" containerName="payload-processing" Oct 30 21:17:15.929851 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:17:15.929809 5378 memory_manager.go:365] "RemoveStaleState removing state" podUID="7715f68b-9673-4119-a43d-05372a096116" containerName="payload-pre-processing" Oct 30 21:17:15.929851 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:17:15.929821 5378 memory_manager.go:365] "RemoveStaleState removing state" podUID="3f616067-e750-4f5c-8fd4-d5a9cf5cf9ec" containerName="payload-processing" Oct 30 21:17:15.934508 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:17:15.934490 5378 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-08df98-dc45f5c6-6hbng" Oct 30 21:17:15.988791 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:17:15.988760 5378 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-b-08df98-serving-cert\"" Oct 30 21:17:15.990166 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:17:15.990133 5378 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-08df98-dc45f5c6-6hbng"] Oct 30 21:17:16.018924 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:17:16.018886 5378 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0acdf79f-8312-4818-a1fb-8e50c7948634-metrics-certs\") pod \"maas-api-e2e-inf-b-08df98-dc45f5c6-6hbng\" (UID: \"0acdf79f-8312-4818-a1fb-8e50c7948634\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-08df98-dc45f5c6-6hbng" Oct 30 21:17:16.019198 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:17:16.019171 5378 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0acdf79f-8312-4818-a1fb-8e50c7948634-maas-api-tls\") pod \"maas-api-e2e-inf-b-08df98-dc45f5c6-6hbng\" (UID: \"0acdf79f-8312-4818-a1fb-8e50c7948634\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-08df98-dc45f5c6-6hbng" Oct 30 21:17:16.019327 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:17:16.019302 5378 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0acdf79f-8312-4818-a1fb-8e50c7948634-openshift-service-ca\") pod \"maas-api-e2e-inf-b-08df98-dc45f5c6-6hbng\" (UID: \"0acdf79f-8312-4818-a1fb-8e50c7948634\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-08df98-dc45f5c6-6hbng" Oct 30 21:17:16.019411 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:17:16.019388 5378 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zm8h5\" (UniqueName: \"kubernetes.io/projected/0acdf79f-8312-4818-a1fb-8e50c7948634-kube-api-access-zm8h5\") pod \"maas-api-e2e-inf-b-08df98-dc45f5c6-6hbng\" (UID: \"0acdf79f-8312-4818-a1fb-8e50c7948634\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-08df98-dc45f5c6-6hbng" Oct 30 21:17:16.056470 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:17:16.056395 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:17:16.120195 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:17:16.120162 5378 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0acdf79f-8312-4818-a1fb-8e50c7948634-maas-api-tls\") pod \"maas-api-e2e-inf-b-08df98-dc45f5c6-6hbng\" (UID: \"0acdf79f-8312-4818-a1fb-8e50c7948634\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-08df98-dc45f5c6-6hbng" Oct 30 21:17:16.120385 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:17:16.120216 5378 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0acdf79f-8312-4818-a1fb-8e50c7948634-openshift-service-ca\") pod \"maas-api-e2e-inf-b-08df98-dc45f5c6-6hbng\" (UID: \"0acdf79f-8312-4818-a1fb-8e50c7948634\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-08df98-dc45f5c6-6hbng" Oct 30 21:17:16.120385 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:17:16.120269 5378 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zm8h5\" (UniqueName: \"kubernetes.io/projected/0acdf79f-8312-4818-a1fb-8e50c7948634-kube-api-access-zm8h5\") pod \"maas-api-e2e-inf-b-08df98-dc45f5c6-6hbng\" (UID: \"0acdf79f-8312-4818-a1fb-8e50c7948634\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-08df98-dc45f5c6-6hbng" Oct 30 21:17:16.120385 ip-10-0-138-149 kubenswrapper[5378]: E1030 21:17:16.120318 5378 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-inf-b-08df98-serving-cert: secret "maas-api-e2e-inf-b-08df98-serving-cert" not found Oct 30 21:17:16.120385 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:17:16.120328 5378 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0acdf79f-8312-4818-a1fb-8e50c7948634-metrics-certs\") pod \"maas-api-e2e-inf-b-08df98-dc45f5c6-6hbng\" (UID: \"0acdf79f-8312-4818-a1fb-8e50c7948634\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-08df98-dc45f5c6-6hbng" Oct 30 21:17:16.120385 ip-10-0-138-149 kubenswrapper[5378]: E1030 21:17:16.120384 5378 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0acdf79f-8312-4818-a1fb-8e50c7948634-maas-api-tls podName:0acdf79f-8312-4818-a1fb-8e50c7948634 nodeName:}" failed. No retries permitted until 2026-10-30 21:17:16.620367291 +0000 UTC m=+2327.048481188 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/0acdf79f-8312-4818-a1fb-8e50c7948634-maas-api-tls") pod "maas-api-e2e-inf-b-08df98-dc45f5c6-6hbng" (UID: "0acdf79f-8312-4818-a1fb-8e50c7948634") : secret "maas-api-e2e-inf-b-08df98-serving-cert" not found Oct 30 21:17:16.121029 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:17:16.121010 5378 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0acdf79f-8312-4818-a1fb-8e50c7948634-openshift-service-ca\") pod \"maas-api-e2e-inf-b-08df98-dc45f5c6-6hbng\" (UID: \"0acdf79f-8312-4818-a1fb-8e50c7948634\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-08df98-dc45f5c6-6hbng" Oct 30 21:17:16.122787 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:17:16.122771 5378 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0acdf79f-8312-4818-a1fb-8e50c7948634-metrics-certs\") pod \"maas-api-e2e-inf-b-08df98-dc45f5c6-6hbng\" (UID: \"0acdf79f-8312-4818-a1fb-8e50c7948634\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-08df98-dc45f5c6-6hbng" Oct 30 21:17:16.208635 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:17:16.208409 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:17:16.218550 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:17:16.212223 5378 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zm8h5\" (UniqueName: \"kubernetes.io/projected/0acdf79f-8312-4818-a1fb-8e50c7948634-kube-api-access-zm8h5\") pod \"maas-api-e2e-inf-b-08df98-dc45f5c6-6hbng\" (UID: \"0acdf79f-8312-4818-a1fb-8e50c7948634\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-08df98-dc45f5c6-6hbng" Oct 30 21:17:16.343776 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:17:16.343559 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:17:16.628640 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:17:16.628610 5378 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0acdf79f-8312-4818-a1fb-8e50c7948634-maas-api-tls\") pod \"maas-api-e2e-inf-b-08df98-dc45f5c6-6hbng\" (UID: \"0acdf79f-8312-4818-a1fb-8e50c7948634\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-08df98-dc45f5c6-6hbng" Oct 30 21:17:16.631097 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:17:16.631071 5378 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0acdf79f-8312-4818-a1fb-8e50c7948634-maas-api-tls\") pod \"maas-api-e2e-inf-b-08df98-dc45f5c6-6hbng\" (UID: \"0acdf79f-8312-4818-a1fb-8e50c7948634\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-08df98-dc45f5c6-6hbng" Oct 30 21:17:16.849378 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:17:16.849341 5378 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-08df98-dc45f5c6-6hbng" Oct 30 21:17:17.227753 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:17:17.227715 5378 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-08df98-dc45f5c6-6hbng"] Oct 30 21:17:17.228758 ip-10-0-138-149 kubenswrapper[5378]: W1030 21:17:17.228728 5378 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0acdf79f_8312_4818_a1fb_8e50c7948634.slice/crio-f843f373ec6dc9d712644fea33c0b3deeca7ab1e3c2a37e42c0bac4ea5701754 WatchSource:0}: Error finding container f843f373ec6dc9d712644fea33c0b3deeca7ab1e3c2a37e42c0bac4ea5701754: Status 404 returned error can't find the container with id f843f373ec6dc9d712644fea33c0b3deeca7ab1e3c2a37e42c0bac4ea5701754 Oct 30 21:17:17.295944 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:17:17.295910 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:17:17.601900 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:17:17.601866 5378 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-08df98-dc45f5c6-6hbng" event={"ID":"0acdf79f-8312-4818-a1fb-8e50c7948634","Type":"ContainerStarted","Data":"f843f373ec6dc9d712644fea33c0b3deeca7ab1e3c2a37e42c0bac4ea5701754"} Oct 30 21:17:17.742137 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:17:17.742100 5378 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2d52d866-9725-4d46-9aa5-adcc43e87bbd" path="/var/lib/kubelet/pods/2d52d866-9725-4d46-9aa5-adcc43e87bbd/volumes" Oct 30 21:17:17.742725 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:17:17.742696 5378 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7715f68b-9673-4119-a43d-05372a096116" path="/var/lib/kubelet/pods/7715f68b-9673-4119-a43d-05372a096116/volumes" Oct 30 21:17:18.610436 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:17:18.610399 5378 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-08df98-dc45f5c6-6hbng" event={"ID":"0acdf79f-8312-4818-a1fb-8e50c7948634","Type":"ContainerStarted","Data":"e2e1204fc7b8f93163152667ed40a0c4e55182dd1010853dfd0bf8c317eccb01"} Oct 30 21:17:18.610948 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:17:18.610630 5378 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-08df98-dc45f5c6-6hbng" Oct 30 21:17:18.726609 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:17:18.726538 5378 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-08df98-dc45f5c6-6hbng" podStartSLOduration=2.882070699 podStartE2EDuration="3.7265222s" totalImagesPullingTime="844.451501ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:17:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:17:18.724883207 +0000 UTC m=+2329.152997126" watchObservedRunningTime="2026-10-30 21:17:18.7265222 +0000 UTC m=+2329.154636119" Oct 30 21:17:18.841702 ip-10-0-138-149 kubenswrapper[5378]: E1030 21:17:18.841666 5378 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q6mx7" containerName="extractor" Oct 30 21:17:19.496644 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:17:19.496601 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:17:19.628189 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:17:19.627909 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:17:21.044802 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:17:21.044770 5378 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-f9f277/distinct-2-w1-f9f277-kserve-6c49cf5dcb-h4x74"] Oct 30 21:17:21.096374 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:17:21.096340 5378 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-f9f277/simulator-w1-f9f277-kserve-5d9685f85c-2h78x"] Oct 30 21:17:21.233614 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:17:21.233556 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:17:21.351182 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:17:21.351138 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:17:21.841374 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:17:21.841290 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:17:21.970230 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:17:21.970191 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:17:23.782089 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:17:23.782039 5378 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-r9g59"] Oct 30 21:17:24.636428 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:17:24.636398 5378 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-08df98-dc45f5c6-6hbng" Oct 30 21:17:27.174662 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:17:27.174630 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:17:27.315546 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:17:27.315508 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:17:29.005909 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:17:29.005876 5378 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w1-f9f277-cf5c8b8bb-st9wf"] Oct 30 21:17:29.110079 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:17:29.110037 5378 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w1-f9f277-68cfd5cbbc-bz22r"] Oct 30 21:17:29.354617 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:17:29.354509 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:17:29.494392 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:17:29.494349 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:17:31.948374 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:17:31.948256 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:17:34.006464 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:17:34.006423 5378 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w1-f9f277-cf5c8b8bb-st9wf" podUID="c360b269-e417-49b5-9482-34e06924da36" containerName="payload-processing" containerID="cri-o://1daf7b461607444f74486b20584ab7362cb4557e5de900fcb167398d1572f567" gracePeriod=25 Oct 30 21:17:34.110940 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:17:34.110902 5378 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-f9f277-68cfd5cbbc-bz22r" podUID="76bab521-3612-401b-a99a-8e8be187baef" containerName="payload-pre-processing" containerID="cri-o://8768a90ec84008cf853ee0a21042fb377db792be4f738b5293ea1577e3bf0022" gracePeriod=25 Oct 30 21:17:36.066046 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:17:36.066000 5378 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w1-f9f277/distinct-2-w1-f9f277-kserve-6c49cf5dcb-h4x74" podUID="9eb6ff54-b77b-453c-a166-988b98cf8f49" containerName="main" containerID="cri-o://ac0610f90854b97e1bc8b4176433cada60c54acd61664c390b22d899d59017e0" gracePeriod=45 Oct 30 21:17:36.116640 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:17:36.116553 5378 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w1-f9f277/simulator-w1-f9f277-kserve-5d9685f85c-2h78x" podUID="dc662e9d-914c-4dfb-a86f-0fb048f7b690" containerName="main" containerID="cri-o://a6d16baadcb84866120ccf6156a24bd15e3f789892937fa292bcb2462dde73a0" gracePeriod=45 Oct 30 21:17:36.150346 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:17:36.150279 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:17:36.479624 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:17:36.479232 5378 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-f9f277/distinct-2-w1-f9f277-kserve-6c49cf5dcb-h4x74" Oct 30 21:17:36.512199 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:17:36.512164 5378 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9eb6ff54-b77b-453c-a166-988b98cf8f49-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9eb6ff54-b77b-453c-a166-988b98cf8f49-tmp-dir\") pod \"9eb6ff54-b77b-453c-a166-988b98cf8f49\" (UID: \"9eb6ff54-b77b-453c-a166-988b98cf8f49\") " Oct 30 21:17:36.512511 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:17:36.512480 5378 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9eb6ff54-b77b-453c-a166-988b98cf8f49-tmp-dir" pod "9eb6ff54-b77b-453c-a166-988b98cf8f49" (UID: "9eb6ff54-b77b-453c-a166-988b98cf8f49"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:17:36.512698 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:17:36.512504 5378 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9eb6ff54-b77b-453c-a166-988b98cf8f49-dshm\" (UniqueName: \"kubernetes.io/empty-dir/9eb6ff54-b77b-453c-a166-988b98cf8f49-dshm\") pod \"9eb6ff54-b77b-453c-a166-988b98cf8f49\" (UID: \"9eb6ff54-b77b-453c-a166-988b98cf8f49\") " Oct 30 21:17:36.513529 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:17:36.513506 5378 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9eb6ff54-b77b-453c-a166-988b98cf8f49-tls-certs\" (UniqueName: \"kubernetes.io/secret/9eb6ff54-b77b-453c-a166-988b98cf8f49-tls-certs\") pod \"9eb6ff54-b77b-453c-a166-988b98cf8f49\" (UID: \"9eb6ff54-b77b-453c-a166-988b98cf8f49\") " Oct 30 21:17:36.513778 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:17:36.513760 5378 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9eb6ff54-b77b-453c-a166-988b98cf8f49-home\" (UniqueName: \"kubernetes.io/empty-dir/9eb6ff54-b77b-453c-a166-988b98cf8f49-home\") pod \"9eb6ff54-b77b-453c-a166-988b98cf8f49\" (UID: \"9eb6ff54-b77b-453c-a166-988b98cf8f49\") " Oct 30 21:17:36.515767 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:17:36.515744 5378 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9eb6ff54-b77b-453c-a166-988b98cf8f49-kube-api-access-fbz2k\" (UniqueName: \"kubernetes.io/projected/9eb6ff54-b77b-453c-a166-988b98cf8f49-kube-api-access-fbz2k\") pod \"9eb6ff54-b77b-453c-a166-988b98cf8f49\" (UID: \"9eb6ff54-b77b-453c-a166-988b98cf8f49\") " Oct 30 21:17:36.516467 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:17:36.515589 5378 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-f9f277/simulator-w1-f9f277-kserve-5d9685f85c-2h78x" Oct 30 21:17:36.516720 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:17:36.515268 5378 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9eb6ff54-b77b-453c-a166-988b98cf8f49-home" pod "9eb6ff54-b77b-453c-a166-988b98cf8f49" (UID: "9eb6ff54-b77b-453c-a166-988b98cf8f49"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:17:36.516967 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:17:36.516923 5378 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9eb6ff54-b77b-453c-a166-988b98cf8f49-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9eb6ff54-b77b-453c-a166-988b98cf8f49-model-cache\") pod \"9eb6ff54-b77b-453c-a166-988b98cf8f49\" (UID: \"9eb6ff54-b77b-453c-a166-988b98cf8f49\") " Oct 30 21:17:36.519728 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:17:36.517355 5378 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9eb6ff54-b77b-453c-a166-988b98cf8f49-model-cache" pod "9eb6ff54-b77b-453c-a166-988b98cf8f49" (UID: "9eb6ff54-b77b-453c-a166-988b98cf8f49"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:17:36.519728 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:17:36.519100 5378 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9eb6ff54-b77b-453c-a166-988b98cf8f49-kube-api-access-fbz2k" pod "9eb6ff54-b77b-453c-a166-988b98cf8f49" (UID: "9eb6ff54-b77b-453c-a166-988b98cf8f49"). InnerVolumeSpecName "kube-api-access-fbz2k". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:17:36.520287 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:17:36.520271 5378 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fbz2k\" (UniqueName: \"kubernetes.io/projected/9eb6ff54-b77b-453c-a166-988b98cf8f49-kube-api-access-fbz2k\") on node \"ip-10-0-138-149.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:17:36.520371 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:17:36.520361 5378 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9eb6ff54-b77b-453c-a166-988b98cf8f49-model-cache\") on node \"ip-10-0-138-149.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:17:36.520425 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:17:36.520417 5378 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9eb6ff54-b77b-453c-a166-988b98cf8f49-tmp-dir\") on node \"ip-10-0-138-149.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:17:36.520482 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:17:36.520473 5378 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9eb6ff54-b77b-453c-a166-988b98cf8f49-home\") on node \"ip-10-0-138-149.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:17:36.523452 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:17:36.523414 5378 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9eb6ff54-b77b-453c-a166-988b98cf8f49-tls-certs" pod "9eb6ff54-b77b-453c-a166-988b98cf8f49" (UID: "9eb6ff54-b77b-453c-a166-988b98cf8f49"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:17:36.523982 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:17:36.523959 5378 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9eb6ff54-b77b-453c-a166-988b98cf8f49-dshm" pod "9eb6ff54-b77b-453c-a166-988b98cf8f49" (UID: "9eb6ff54-b77b-453c-a166-988b98cf8f49"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:17:36.620991 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:17:36.620956 5378 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dc662e9d-914c-4dfb-a86f-0fb048f7b690-home\" (UniqueName: \"kubernetes.io/empty-dir/dc662e9d-914c-4dfb-a86f-0fb048f7b690-home\") pod \"dc662e9d-914c-4dfb-a86f-0fb048f7b690\" (UID: \"dc662e9d-914c-4dfb-a86f-0fb048f7b690\") " Oct 30 21:17:36.621231 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:17:36.621151 5378 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dc662e9d-914c-4dfb-a86f-0fb048f7b690-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dc662e9d-914c-4dfb-a86f-0fb048f7b690-tmp-dir\") pod \"dc662e9d-914c-4dfb-a86f-0fb048f7b690\" (UID: \"dc662e9d-914c-4dfb-a86f-0fb048f7b690\") " Oct 30 21:17:36.621231 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:17:36.621192 5378 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dc662e9d-914c-4dfb-a86f-0fb048f7b690-dshm\" (UniqueName: \"kubernetes.io/empty-dir/dc662e9d-914c-4dfb-a86f-0fb048f7b690-dshm\") pod \"dc662e9d-914c-4dfb-a86f-0fb048f7b690\" (UID: \"dc662e9d-914c-4dfb-a86f-0fb048f7b690\") " Oct 30 21:17:36.621497 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:17:36.621240 5378 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/dc662e9d-914c-4dfb-a86f-0fb048f7b690-tls-certs\" (UniqueName: \"kubernetes.io/secret/dc662e9d-914c-4dfb-a86f-0fb048f7b690-tls-certs\") pod \"dc662e9d-914c-4dfb-a86f-0fb048f7b690\" (UID: \"dc662e9d-914c-4dfb-a86f-0fb048f7b690\") " Oct 30 21:17:36.621497 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:17:36.621274 5378 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dc662e9d-914c-4dfb-a86f-0fb048f7b690-home" pod "dc662e9d-914c-4dfb-a86f-0fb048f7b690" (UID: "dc662e9d-914c-4dfb-a86f-0fb048f7b690"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:17:36.621497 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:17:36.621342 5378 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dc662e9d-914c-4dfb-a86f-0fb048f7b690-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/dc662e9d-914c-4dfb-a86f-0fb048f7b690-model-cache\") pod \"dc662e9d-914c-4dfb-a86f-0fb048f7b690\" (UID: \"dc662e9d-914c-4dfb-a86f-0fb048f7b690\") " Oct 30 21:17:36.621497 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:17:36.621372 5378 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dc662e9d-914c-4dfb-a86f-0fb048f7b690-kube-api-access-msjgx\" (UniqueName: \"kubernetes.io/projected/dc662e9d-914c-4dfb-a86f-0fb048f7b690-kube-api-access-msjgx\") pod \"dc662e9d-914c-4dfb-a86f-0fb048f7b690\" (UID: \"dc662e9d-914c-4dfb-a86f-0fb048f7b690\") " Oct 30 21:17:36.621497 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:17:36.621481 5378 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dc662e9d-914c-4dfb-a86f-0fb048f7b690-tmp-dir" pod "dc662e9d-914c-4dfb-a86f-0fb048f7b690" (UID: "dc662e9d-914c-4dfb-a86f-0fb048f7b690"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:17:36.622043 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:17:36.621968 5378 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/dc662e9d-914c-4dfb-a86f-0fb048f7b690-home\") on node \"ip-10-0-138-149.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:17:36.622043 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:17:36.621990 5378 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9eb6ff54-b77b-453c-a166-988b98cf8f49-dshm\") on node \"ip-10-0-138-149.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:17:36.622043 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:17:36.622004 5378 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9eb6ff54-b77b-453c-a166-988b98cf8f49-tls-certs\") on node \"ip-10-0-138-149.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:17:36.622043 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:17:36.622016 5378 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dc662e9d-914c-4dfb-a86f-0fb048f7b690-tmp-dir\") on node \"ip-10-0-138-149.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:17:36.622313 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:17:36.622276 5378 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dc662e9d-914c-4dfb-a86f-0fb048f7b690-model-cache" pod "dc662e9d-914c-4dfb-a86f-0fb048f7b690" (UID: "dc662e9d-914c-4dfb-a86f-0fb048f7b690"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:17:36.624463 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:17:36.624432 5378 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dc662e9d-914c-4dfb-a86f-0fb048f7b690-dshm" pod "dc662e9d-914c-4dfb-a86f-0fb048f7b690" (UID: "dc662e9d-914c-4dfb-a86f-0fb048f7b690"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:17:36.624599 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:17:36.624465 5378 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dc662e9d-914c-4dfb-a86f-0fb048f7b690-kube-api-access-msjgx" pod "dc662e9d-914c-4dfb-a86f-0fb048f7b690" (UID: "dc662e9d-914c-4dfb-a86f-0fb048f7b690"). InnerVolumeSpecName "kube-api-access-msjgx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:17:36.624828 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:17:36.624804 5378 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dc662e9d-914c-4dfb-a86f-0fb048f7b690-tls-certs" pod "dc662e9d-914c-4dfb-a86f-0fb048f7b690" (UID: "dc662e9d-914c-4dfb-a86f-0fb048f7b690"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:17:36.723269 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:17:36.723230 5378 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/dc662e9d-914c-4dfb-a86f-0fb048f7b690-dshm\") on node \"ip-10-0-138-149.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:17:36.723269 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:17:36.723268 5378 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dc662e9d-914c-4dfb-a86f-0fb048f7b690-tls-certs\") on node \"ip-10-0-138-149.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:17:36.723694 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:17:36.723285 5378 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/dc662e9d-914c-4dfb-a86f-0fb048f7b690-model-cache\") on node \"ip-10-0-138-149.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:17:36.723694 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:17:36.723302 5378 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-msjgx\" (UniqueName: \"kubernetes.io/projected/dc662e9d-914c-4dfb-a86f-0fb048f7b690-kube-api-access-msjgx\") on node \"ip-10-0-138-149.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:17:36.770425 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:17:36.770387 5378 generic.go:356] "Generic (PLEG): container finished" podID="9eb6ff54-b77b-453c-a166-988b98cf8f49" containerID="ac0610f90854b97e1bc8b4176433cada60c54acd61664c390b22d899d59017e0" exitCode=0 Oct 30 21:17:36.770619 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:17:36.770428 5378 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-f9f277/distinct-2-w1-f9f277-kserve-6c49cf5dcb-h4x74" Oct 30 21:17:36.770619 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:17:36.770467 5378 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-f9f277/distinct-2-w1-f9f277-kserve-6c49cf5dcb-h4x74" event={"ID":"9eb6ff54-b77b-453c-a166-988b98cf8f49","Type":"ContainerDied","Data":"ac0610f90854b97e1bc8b4176433cada60c54acd61664c390b22d899d59017e0"} Oct 30 21:17:36.770619 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:17:36.770508 5378 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-f9f277/distinct-2-w1-f9f277-kserve-6c49cf5dcb-h4x74" event={"ID":"9eb6ff54-b77b-453c-a166-988b98cf8f49","Type":"ContainerDied","Data":"f345bf1c4e602939b92c2bade819f1827576f5bdaa1049a3ce7065c10153ba43"} Oct 30 21:17:36.770619 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:17:36.770529 5378 scope.go:122] "RemoveContainer" containerID="ac0610f90854b97e1bc8b4176433cada60c54acd61664c390b22d899d59017e0" Oct 30 21:17:36.772134 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:17:36.772109 5378 generic.go:356] "Generic (PLEG): container finished" podID="dc662e9d-914c-4dfb-a86f-0fb048f7b690" containerID="a6d16baadcb84866120ccf6156a24bd15e3f789892937fa292bcb2462dde73a0" exitCode=0 Oct 30 21:17:36.772270 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:17:36.772157 5378 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-f9f277/simulator-w1-f9f277-kserve-5d9685f85c-2h78x" Oct 30 21:17:36.772270 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:17:36.772207 5378 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-f9f277/simulator-w1-f9f277-kserve-5d9685f85c-2h78x" event={"ID":"dc662e9d-914c-4dfb-a86f-0fb048f7b690","Type":"ContainerDied","Data":"a6d16baadcb84866120ccf6156a24bd15e3f789892937fa292bcb2462dde73a0"} Oct 30 21:17:36.772270 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:17:36.772239 5378 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-f9f277/simulator-w1-f9f277-kserve-5d9685f85c-2h78x" event={"ID":"dc662e9d-914c-4dfb-a86f-0fb048f7b690","Type":"ContainerDied","Data":"91e64839ddba643b5f406aacc92dcab4f72496ae37403f03c17fda8e0a1f174f"} Oct 30 21:17:36.782794 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:17:36.782774 5378 scope.go:122] "RemoveContainer" containerID="ac0610f90854b97e1bc8b4176433cada60c54acd61664c390b22d899d59017e0" Oct 30 21:17:36.783098 ip-10-0-138-149 kubenswrapper[5378]: E1030 21:17:36.783078 5378 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ac0610f90854b97e1bc8b4176433cada60c54acd61664c390b22d899d59017e0\": container with ID starting with ac0610f90854b97e1bc8b4176433cada60c54acd61664c390b22d899d59017e0 not found: ID does not exist" containerID="ac0610f90854b97e1bc8b4176433cada60c54acd61664c390b22d899d59017e0" Oct 30 21:17:36.783250 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:17:36.783106 5378 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ac0610f90854b97e1bc8b4176433cada60c54acd61664c390b22d899d59017e0"} err="failed to get container status \"ac0610f90854b97e1bc8b4176433cada60c54acd61664c390b22d899d59017e0\": rpc error: code = NotFound desc = could not find container \"ac0610f90854b97e1bc8b4176433cada60c54acd61664c390b22d899d59017e0\": container with ID starting with ac0610f90854b97e1bc8b4176433cada60c54acd61664c390b22d899d59017e0 not found: ID does not exist" Oct 30 21:17:36.783250 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:17:36.783128 5378 scope.go:122] "RemoveContainer" containerID="a6d16baadcb84866120ccf6156a24bd15e3f789892937fa292bcb2462dde73a0" Oct 30 21:17:36.794967 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:17:36.794840 5378 scope.go:122] "RemoveContainer" containerID="a6d16baadcb84866120ccf6156a24bd15e3f789892937fa292bcb2462dde73a0" Oct 30 21:17:36.795216 ip-10-0-138-149 kubenswrapper[5378]: E1030 21:17:36.795195 5378 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a6d16baadcb84866120ccf6156a24bd15e3f789892937fa292bcb2462dde73a0\": container with ID starting with a6d16baadcb84866120ccf6156a24bd15e3f789892937fa292bcb2462dde73a0 not found: ID does not exist" containerID="a6d16baadcb84866120ccf6156a24bd15e3f789892937fa292bcb2462dde73a0" Oct 30 21:17:36.795279 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:17:36.795222 5378 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a6d16baadcb84866120ccf6156a24bd15e3f789892937fa292bcb2462dde73a0"} err="failed to get container status \"a6d16baadcb84866120ccf6156a24bd15e3f789892937fa292bcb2462dde73a0\": rpc error: code = NotFound desc = could not find container \"a6d16baadcb84866120ccf6156a24bd15e3f789892937fa292bcb2462dde73a0\": container with ID starting with a6d16baadcb84866120ccf6156a24bd15e3f789892937fa292bcb2462dde73a0 not found: ID does not exist" Oct 30 21:17:36.935758 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:17:36.935676 5378 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-f9f277/simulator-w1-f9f277-kserve-5d9685f85c-2h78x"] Oct 30 21:17:36.939529 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:17:36.939504 5378 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w1-f9f277/simulator-w1-f9f277-kserve-5d9685f85c-2h78x"] Oct 30 21:17:37.072836 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:17:37.072795 5378 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-f9f277/distinct-2-w1-f9f277-kserve-6c49cf5dcb-h4x74"] Oct 30 21:17:37.093787 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:17:37.093742 5378 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w1-f9f277/distinct-2-w1-f9f277-kserve-6c49cf5dcb-h4x74"] Oct 30 21:17:37.495140 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:17:37.495107 5378 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w1-f9f277-cf5c8b8bb-st9wf" podUID="c360b269-e417-49b5-9482-34e06924da36" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 21:17:37.495311 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:17:37.495112 5378 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-f9f277-68cfd5cbbc-bz22r" podUID="76bab521-3612-401b-a99a-8e8be187baef" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 21:17:37.696458 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:17:37.696420 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:17:37.741163 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:17:37.741129 5378 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9eb6ff54-b77b-453c-a166-988b98cf8f49" path="/var/lib/kubelet/pods/9eb6ff54-b77b-453c-a166-988b98cf8f49/volumes" Oct 30 21:17:37.741617 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:17:37.741603 5378 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dc662e9d-914c-4dfb-a86f-0fb048f7b690" path="/var/lib/kubelet/pods/dc662e9d-914c-4dfb-a86f-0fb048f7b690/volumes" Oct 30 21:17:37.957618 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:17:37.957579 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:17:38.089269 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:17:38.089234 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:17:38.648531 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:17:38.648451 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:17:43.239309 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:17:43.239276 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:17:44.523990 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:17:44.523953 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:17:44.660641 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:17:44.660610 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:17:44.886579 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:17:44.886541 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:17:45.445399 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:17:45.445365 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:17:47.495419 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:17:47.495377 5378 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-f9f277-68cfd5cbbc-bz22r" podUID="76bab521-3612-401b-a99a-8e8be187baef" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 21:17:47.495848 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:17:47.495432 5378 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w1-f9f277-cf5c8b8bb-st9wf" podUID="c360b269-e417-49b5-9482-34e06924da36" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 21:17:48.498753 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:17:48.498720 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:17:48.837543 ip-10-0-138-149 kubenswrapper[5378]: E1030 21:17:48.837452 5378 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q6mx7" containerName="extractor" Oct 30 21:17:49.467234 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:17:49.467193 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:17:50.397033 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:17:50.397004 5378 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-781058/distinct-w0-781058-kserve-8bdf9bfcd-hzdfk"] Oct 30 21:17:50.536751 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:17:50.536720 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:17:53.648980 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:17:53.648944 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:17:53.808771 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:17:53.808737 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:17:53.935307 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:17:53.935220 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:17:54.281155 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:17:54.281127 5378 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w1-f9f277-cf5c8b8bb-st9wf" Oct 30 21:17:54.367732 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:17:54.367704 5378 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-f9f277-68cfd5cbbc-bz22r" Oct 30 21:17:54.393903 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:17:54.393880 5378 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c360b269-e417-49b5-9482-34e06924da36-kube-api-access-fs2fl\" (UniqueName: \"kubernetes.io/projected/c360b269-e417-49b5-9482-34e06924da36-kube-api-access-fs2fl\") pod \"c360b269-e417-49b5-9482-34e06924da36\" (UID: \"c360b269-e417-49b5-9482-34e06924da36\") " Oct 30 21:17:54.394044 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:17:54.393916 5378 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c360b269-e417-49b5-9482-34e06924da36-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c360b269-e417-49b5-9482-34e06924da36-plugins-config-volume\") pod \"c360b269-e417-49b5-9482-34e06924da36\" (UID: \"c360b269-e417-49b5-9482-34e06924da36\") " Oct 30 21:17:54.394360 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:17:54.394329 5378 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c360b269-e417-49b5-9482-34e06924da36-plugins-config-volume" pod "c360b269-e417-49b5-9482-34e06924da36" (UID: "c360b269-e417-49b5-9482-34e06924da36"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:17:54.394651 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:17:54.394628 5378 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c360b269-e417-49b5-9482-34e06924da36-plugins-config-volume\") on node \"ip-10-0-138-149.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:17:54.396161 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:17:54.396140 5378 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c360b269-e417-49b5-9482-34e06924da36-kube-api-access-fs2fl" pod "c360b269-e417-49b5-9482-34e06924da36" (UID: "c360b269-e417-49b5-9482-34e06924da36"). InnerVolumeSpecName "kube-api-access-fs2fl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:17:54.495868 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:17:54.495787 5378 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/76bab521-3612-401b-a99a-8e8be187baef-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/76bab521-3612-401b-a99a-8e8be187baef-plugins-config-volume\") pod \"76bab521-3612-401b-a99a-8e8be187baef\" (UID: \"76bab521-3612-401b-a99a-8e8be187baef\") " Oct 30 21:17:54.495868 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:17:54.495827 5378 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/76bab521-3612-401b-a99a-8e8be187baef-kube-api-access-2k45l\" (UniqueName: \"kubernetes.io/projected/76bab521-3612-401b-a99a-8e8be187baef-kube-api-access-2k45l\") pod \"76bab521-3612-401b-a99a-8e8be187baef\" (UID: \"76bab521-3612-401b-a99a-8e8be187baef\") " Oct 30 21:17:54.496246 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:17:54.496220 5378 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/76bab521-3612-401b-a99a-8e8be187baef-plugins-config-volume" pod "76bab521-3612-401b-a99a-8e8be187baef" (UID: "76bab521-3612-401b-a99a-8e8be187baef"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:17:54.496323 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:17:54.496288 5378 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fs2fl\" (UniqueName: \"kubernetes.io/projected/c360b269-e417-49b5-9482-34e06924da36-kube-api-access-fs2fl\") on node \"ip-10-0-138-149.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:17:54.498067 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:17:54.498046 5378 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/76bab521-3612-401b-a99a-8e8be187baef-kube-api-access-2k45l" pod "76bab521-3612-401b-a99a-8e8be187baef" (UID: "76bab521-3612-401b-a99a-8e8be187baef"). InnerVolumeSpecName "kube-api-access-2k45l". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:17:54.597249 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:17:54.597215 5378 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/76bab521-3612-401b-a99a-8e8be187baef-plugins-config-volume\") on node \"ip-10-0-138-149.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:17:54.597249 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:17:54.597241 5378 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2k45l\" (UniqueName: \"kubernetes.io/projected/76bab521-3612-401b-a99a-8e8be187baef-kube-api-access-2k45l\") on node \"ip-10-0-138-149.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:17:54.938073 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:17:54.938040 5378 generic.go:356] "Generic (PLEG): container finished" podID="76bab521-3612-401b-a99a-8e8be187baef" containerID="8768a90ec84008cf853ee0a21042fb377db792be4f738b5293ea1577e3bf0022" exitCode=0 Oct 30 21:17:54.938539 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:17:54.938086 5378 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-f9f277-68cfd5cbbc-bz22r" Oct 30 21:17:54.938539 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:17:54.938121 5378 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-f9f277-68cfd5cbbc-bz22r" event={"ID":"76bab521-3612-401b-a99a-8e8be187baef","Type":"ContainerDied","Data":"8768a90ec84008cf853ee0a21042fb377db792be4f738b5293ea1577e3bf0022"} Oct 30 21:17:54.938539 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:17:54.938176 5378 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-f9f277-68cfd5cbbc-bz22r" event={"ID":"76bab521-3612-401b-a99a-8e8be187baef","Type":"ContainerDied","Data":"12abefd12686e7ca0b2ff5e26f696f601eef2a0c4d7930f22720a4c6f88f7cc7"} Oct 30 21:17:54.938539 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:17:54.938203 5378 scope.go:122] "RemoveContainer" containerID="8768a90ec84008cf853ee0a21042fb377db792be4f738b5293ea1577e3bf0022" Oct 30 21:17:54.939608 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:17:54.939550 5378 generic.go:356] "Generic (PLEG): container finished" podID="c360b269-e417-49b5-9482-34e06924da36" containerID="1daf7b461607444f74486b20584ab7362cb4557e5de900fcb167398d1572f567" exitCode=0 Oct 30 21:17:54.939731 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:17:54.939603 5378 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w1-f9f277-cf5c8b8bb-st9wf" event={"ID":"c360b269-e417-49b5-9482-34e06924da36","Type":"ContainerDied","Data":"1daf7b461607444f74486b20584ab7362cb4557e5de900fcb167398d1572f567"} Oct 30 21:17:54.939731 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:17:54.939623 5378 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w1-f9f277-cf5c8b8bb-st9wf" Oct 30 21:17:54.939731 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:17:54.939653 5378 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w1-f9f277-cf5c8b8bb-st9wf" event={"ID":"c360b269-e417-49b5-9482-34e06924da36","Type":"ContainerDied","Data":"3952316435df6a8cf30b4e17a86b1216ac2389cfc6216f21059889374f939d17"} Oct 30 21:17:54.949040 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:17:54.949023 5378 scope.go:122] "RemoveContainer" containerID="8768a90ec84008cf853ee0a21042fb377db792be4f738b5293ea1577e3bf0022" Oct 30 21:17:54.949311 ip-10-0-138-149 kubenswrapper[5378]: E1030 21:17:54.949288 5378 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8768a90ec84008cf853ee0a21042fb377db792be4f738b5293ea1577e3bf0022\": container with ID starting with 8768a90ec84008cf853ee0a21042fb377db792be4f738b5293ea1577e3bf0022 not found: ID does not exist" containerID="8768a90ec84008cf853ee0a21042fb377db792be4f738b5293ea1577e3bf0022" Oct 30 21:17:54.949400 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:17:54.949319 5378 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8768a90ec84008cf853ee0a21042fb377db792be4f738b5293ea1577e3bf0022"} err="failed to get container status \"8768a90ec84008cf853ee0a21042fb377db792be4f738b5293ea1577e3bf0022\": rpc error: code = NotFound desc = could not find container \"8768a90ec84008cf853ee0a21042fb377db792be4f738b5293ea1577e3bf0022\": container with ID starting with 8768a90ec84008cf853ee0a21042fb377db792be4f738b5293ea1577e3bf0022 not found: ID does not exist" Oct 30 21:17:54.949400 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:17:54.949335 5378 scope.go:122] "RemoveContainer" containerID="1daf7b461607444f74486b20584ab7362cb4557e5de900fcb167398d1572f567" Oct 30 21:17:54.958025 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:17:54.958009 5378 scope.go:122] "RemoveContainer" containerID="1daf7b461607444f74486b20584ab7362cb4557e5de900fcb167398d1572f567" Oct 30 21:17:54.958302 ip-10-0-138-149 kubenswrapper[5378]: E1030 21:17:54.958283 5378 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1daf7b461607444f74486b20584ab7362cb4557e5de900fcb167398d1572f567\": container with ID starting with 1daf7b461607444f74486b20584ab7362cb4557e5de900fcb167398d1572f567 not found: ID does not exist" containerID="1daf7b461607444f74486b20584ab7362cb4557e5de900fcb167398d1572f567" Oct 30 21:17:54.958380 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:17:54.958312 5378 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1daf7b461607444f74486b20584ab7362cb4557e5de900fcb167398d1572f567"} err="failed to get container status \"1daf7b461607444f74486b20584ab7362cb4557e5de900fcb167398d1572f567\": rpc error: code = NotFound desc = could not find container \"1daf7b461607444f74486b20584ab7362cb4557e5de900fcb167398d1572f567\": container with ID starting with 1daf7b461607444f74486b20584ab7362cb4557e5de900fcb167398d1572f567 not found: ID does not exist" Oct 30 21:17:55.104349 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:17:55.104310 5378 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w1-f9f277-68cfd5cbbc-bz22r"] Oct 30 21:17:55.154672 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:17:55.154639 5378 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w1-f9f277-68cfd5cbbc-bz22r"] Oct 30 21:17:55.264773 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:17:55.264695 5378 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w1-f9f277-cf5c8b8bb-st9wf"] Oct 30 21:17:55.268197 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:17:55.268169 5378 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w1-f9f277-cf5c8b8bb-st9wf"] Oct 30 21:17:55.741519 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:17:55.741490 5378 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="76bab521-3612-401b-a99a-8e8be187baef" path="/var/lib/kubelet/pods/76bab521-3612-401b-a99a-8e8be187baef/volumes" Oct 30 21:17:55.741942 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:17:55.741928 5378 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c360b269-e417-49b5-9482-34e06924da36" path="/var/lib/kubelet/pods/c360b269-e417-49b5-9482-34e06924da36/volumes" Oct 30 21:17:57.561453 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:17:57.561412 5378 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-781058-6d5676f8cb-fkrgx"] Oct 30 21:17:57.981081 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:17:57.981050 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:17:58.115981 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:17:58.115944 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:17:59.169630 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:17:59.158457 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:17:59.374472 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:17:59.374349 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:17:59.855646 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:17:59.848270 5378 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-1fdaa7/simulator-w4-1fdaa7-kserve-9866d468c-fwmsk" podUID="09219832-010e-4ba8-bfb9-5a2884b358e1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 21:18:00.052586 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:18:00.052541 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:18:00.904818 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:18:00.904767 5378 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-769df7846c-r5j2r container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.161:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 21:18:00.905223 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:18:00.904835 5378 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-r5j2r" podUID="04d5e742-36a9-41a8-9bb7-c9c8808d74cc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.161:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 21:18:02.562466 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:18:02.562425 5378 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w0-781058-6d5676f8cb-fkrgx" podUID="9b4bb6e8-85e8-44a9-aaaa-485f6c6b0b79" containerName="payload-processing" containerID="cri-o://3e25b19e11cd021147005f7589ac25af8e75d13b095e83dd162a5d0e96347bb1" gracePeriod=25 Oct 30 21:18:04.902704 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:18:04.902668 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:18:05.200620 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:18:05.200508 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:18:05.417907 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:18:05.417843 5378 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w0-781058/distinct-w0-781058-kserve-8bdf9bfcd-hzdfk" podUID="a9678fc2-1c56-4d45-9007-eee5049e0f48" containerName="main" containerID="cri-o://210dcceffce8b917b42e961a01e6a86c470e71b5535de0b9ccc82c420fc7ee88" gracePeriod=45 Oct 30 21:18:05.685624 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:18:05.685600 5378 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-781058/distinct-w0-781058-kserve-8bdf9bfcd-hzdfk" Oct 30 21:18:05.791922 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:18:05.791855 5378 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a9678fc2-1c56-4d45-9007-eee5049e0f48-kube-api-access-6v9tk\" (UniqueName: \"kubernetes.io/projected/a9678fc2-1c56-4d45-9007-eee5049e0f48-kube-api-access-6v9tk\") pod \"a9678fc2-1c56-4d45-9007-eee5049e0f48\" (UID: \"a9678fc2-1c56-4d45-9007-eee5049e0f48\") " Oct 30 21:18:05.792072 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:18:05.791973 5378 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a9678fc2-1c56-4d45-9007-eee5049e0f48-home\" (UniqueName: \"kubernetes.io/empty-dir/a9678fc2-1c56-4d45-9007-eee5049e0f48-home\") pod \"a9678fc2-1c56-4d45-9007-eee5049e0f48\" (UID: \"a9678fc2-1c56-4d45-9007-eee5049e0f48\") " Oct 30 21:18:05.792143 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:18:05.792108 5378 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a9678fc2-1c56-4d45-9007-eee5049e0f48-dshm\" (UniqueName: \"kubernetes.io/empty-dir/a9678fc2-1c56-4d45-9007-eee5049e0f48-dshm\") pod \"a9678fc2-1c56-4d45-9007-eee5049e0f48\" (UID: \"a9678fc2-1c56-4d45-9007-eee5049e0f48\") " Oct 30 21:18:05.792200 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:18:05.792154 5378 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a9678fc2-1c56-4d45-9007-eee5049e0f48-tls-certs\" (UniqueName: \"kubernetes.io/secret/a9678fc2-1c56-4d45-9007-eee5049e0f48-tls-certs\") pod \"a9678fc2-1c56-4d45-9007-eee5049e0f48\" (UID: \"a9678fc2-1c56-4d45-9007-eee5049e0f48\") " Oct 30 21:18:05.792261 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:18:05.792199 5378 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a9678fc2-1c56-4d45-9007-eee5049e0f48-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a9678fc2-1c56-4d45-9007-eee5049e0f48-tmp-dir\") pod \"a9678fc2-1c56-4d45-9007-eee5049e0f48\" (UID: \"a9678fc2-1c56-4d45-9007-eee5049e0f48\") " Oct 30 21:18:05.792323 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:18:05.792248 5378 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a9678fc2-1c56-4d45-9007-eee5049e0f48-home" pod "a9678fc2-1c56-4d45-9007-eee5049e0f48" (UID: "a9678fc2-1c56-4d45-9007-eee5049e0f48"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:18:05.792323 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:18:05.792295 5378 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a9678fc2-1c56-4d45-9007-eee5049e0f48-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a9678fc2-1c56-4d45-9007-eee5049e0f48-model-cache\") pod \"a9678fc2-1c56-4d45-9007-eee5049e0f48\" (UID: \"a9678fc2-1c56-4d45-9007-eee5049e0f48\") " Oct 30 21:18:05.792580 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:18:05.792531 5378 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a9678fc2-1c56-4d45-9007-eee5049e0f48-tmp-dir" pod "a9678fc2-1c56-4d45-9007-eee5049e0f48" (UID: "a9678fc2-1c56-4d45-9007-eee5049e0f48"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:18:05.792715 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:18:05.792686 5378 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a9678fc2-1c56-4d45-9007-eee5049e0f48-model-cache" pod "a9678fc2-1c56-4d45-9007-eee5049e0f48" (UID: "a9678fc2-1c56-4d45-9007-eee5049e0f48"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:18:05.793068 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:18:05.793046 5378 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a9678fc2-1c56-4d45-9007-eee5049e0f48-home\") on node \"ip-10-0-138-149.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:18:05.793186 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:18:05.793073 5378 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a9678fc2-1c56-4d45-9007-eee5049e0f48-tmp-dir\") on node \"ip-10-0-138-149.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:18:05.793186 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:18:05.793089 5378 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a9678fc2-1c56-4d45-9007-eee5049e0f48-model-cache\") on node \"ip-10-0-138-149.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:18:05.794359 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:18:05.794323 5378 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a9678fc2-1c56-4d45-9007-eee5049e0f48-dshm" pod "a9678fc2-1c56-4d45-9007-eee5049e0f48" (UID: "a9678fc2-1c56-4d45-9007-eee5049e0f48"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:18:05.794591 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:18:05.794554 5378 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a9678fc2-1c56-4d45-9007-eee5049e0f48-kube-api-access-6v9tk" pod "a9678fc2-1c56-4d45-9007-eee5049e0f48" (UID: "a9678fc2-1c56-4d45-9007-eee5049e0f48"). InnerVolumeSpecName "kube-api-access-6v9tk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:18:05.794680 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:18:05.794665 5378 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a9678fc2-1c56-4d45-9007-eee5049e0f48-tls-certs" pod "a9678fc2-1c56-4d45-9007-eee5049e0f48" (UID: "a9678fc2-1c56-4d45-9007-eee5049e0f48"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:18:05.894832 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:18:05.894800 5378 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a9678fc2-1c56-4d45-9007-eee5049e0f48-dshm\") on node \"ip-10-0-138-149.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:18:05.894832 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:18:05.894830 5378 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a9678fc2-1c56-4d45-9007-eee5049e0f48-tls-certs\") on node \"ip-10-0-138-149.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:18:05.895021 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:18:05.894841 5378 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6v9tk\" (UniqueName: \"kubernetes.io/projected/a9678fc2-1c56-4d45-9007-eee5049e0f48-kube-api-access-6v9tk\") on node \"ip-10-0-138-149.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:18:06.090208 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:18:06.090122 5378 generic.go:356] "Generic (PLEG): container finished" podID="a9678fc2-1c56-4d45-9007-eee5049e0f48" containerID="210dcceffce8b917b42e961a01e6a86c470e71b5535de0b9ccc82c420fc7ee88" exitCode=0 Oct 30 21:18:06.090208 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:18:06.090171 5378 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-781058/distinct-w0-781058-kserve-8bdf9bfcd-hzdfk" Oct 30 21:18:06.090718 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:18:06.090206 5378 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-781058/distinct-w0-781058-kserve-8bdf9bfcd-hzdfk" event={"ID":"a9678fc2-1c56-4d45-9007-eee5049e0f48","Type":"ContainerDied","Data":"210dcceffce8b917b42e961a01e6a86c470e71b5535de0b9ccc82c420fc7ee88"} Oct 30 21:18:06.090718 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:18:06.090252 5378 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-781058/distinct-w0-781058-kserve-8bdf9bfcd-hzdfk" event={"ID":"a9678fc2-1c56-4d45-9007-eee5049e0f48","Type":"ContainerDied","Data":"36d0e4149bbdc27aae8da13c2bbbb753bfbf44b9a837844b6cdee806f3309ad4"} Oct 30 21:18:06.090718 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:18:06.090269 5378 scope.go:122] "RemoveContainer" containerID="210dcceffce8b917b42e961a01e6a86c470e71b5535de0b9ccc82c420fc7ee88" Oct 30 21:18:06.101006 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:18:06.100989 5378 scope.go:122] "RemoveContainer" containerID="210dcceffce8b917b42e961a01e6a86c470e71b5535de0b9ccc82c420fc7ee88" Oct 30 21:18:06.101264 ip-10-0-138-149 kubenswrapper[5378]: E1030 21:18:06.101247 5378 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"210dcceffce8b917b42e961a01e6a86c470e71b5535de0b9ccc82c420fc7ee88\": container with ID starting with 210dcceffce8b917b42e961a01e6a86c470e71b5535de0b9ccc82c420fc7ee88 not found: ID does not exist" containerID="210dcceffce8b917b42e961a01e6a86c470e71b5535de0b9ccc82c420fc7ee88" Oct 30 21:18:06.101330 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:18:06.101270 5378 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"210dcceffce8b917b42e961a01e6a86c470e71b5535de0b9ccc82c420fc7ee88"} err="failed to get container status \"210dcceffce8b917b42e961a01e6a86c470e71b5535de0b9ccc82c420fc7ee88\": rpc error: code = NotFound desc = could not find container \"210dcceffce8b917b42e961a01e6a86c470e71b5535de0b9ccc82c420fc7ee88\": container with ID starting with 210dcceffce8b917b42e961a01e6a86c470e71b5535de0b9ccc82c420fc7ee88 not found: ID does not exist" Oct 30 21:18:06.262689 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:18:06.262636 5378 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-781058/distinct-w0-781058-kserve-8bdf9bfcd-hzdfk"] Oct 30 21:18:06.265370 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:18:06.265347 5378 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w0-781058/distinct-w0-781058-kserve-8bdf9bfcd-hzdfk"] Oct 30 21:18:07.741440 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:18:07.741405 5378 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a9678fc2-1c56-4d45-9007-eee5049e0f48" path="/var/lib/kubelet/pods/a9678fc2-1c56-4d45-9007-eee5049e0f48/volumes" Oct 30 21:18:08.145197 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:18:08.145150 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:18:08.268620 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:18:08.268579 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:18:09.465347 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:18:09.465313 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:18:10.481035 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:18:10.480995 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:18:10.588823 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:18:10.588791 5378 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w0-781058-6d5676f8cb-fkrgx" podUID="9b4bb6e8-85e8-44a9-aaaa-485f6c6b0b79" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 21:18:11.360382 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:18:11.360354 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:18:15.090534 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:18:15.090493 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:18:15.219916 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:18:15.219883 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:18:15.355916 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:18:15.355844 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:18:17.110388 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:18:17.110355 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:18:17.255457 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:18:17.255422 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:18:18.837592 ip-10-0-138-149 kubenswrapper[5378]: E1030 21:18:18.837537 5378 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q6mx7" containerName="extractor" Oct 30 21:18:19.466331 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:18:19.466295 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:18:20.588876 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:18:20.588842 5378 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w0-781058-6d5676f8cb-fkrgx" podUID="9b4bb6e8-85e8-44a9-aaaa-485f6c6b0b79" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 21:18:21.026834 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:18:21.026802 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:18:21.264934 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:18:21.264899 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:18:21.392853 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:18:21.392813 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:18:21.959249 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:18:21.959213 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:18:22.541054 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:18:22.541017 5378 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-r9g59"] Oct 30 21:18:23.088021 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:18:23.087988 5378 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w0-781058-6d5676f8cb-fkrgx" Oct 30 21:18:23.280995 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:18:23.280904 5378 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9b4bb6e8-85e8-44a9-aaaa-485f6c6b0b79-kube-api-access-jfk9j\" (UniqueName: \"kubernetes.io/projected/9b4bb6e8-85e8-44a9-aaaa-485f6c6b0b79-kube-api-access-jfk9j\") pod \"9b4bb6e8-85e8-44a9-aaaa-485f6c6b0b79\" (UID: \"9b4bb6e8-85e8-44a9-aaaa-485f6c6b0b79\") " Oct 30 21:18:23.318739 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:18:23.318709 5378 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9b4bb6e8-85e8-44a9-aaaa-485f6c6b0b79-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9b4bb6e8-85e8-44a9-aaaa-485f6c6b0b79-plugins-config-volume\") pod \"9b4bb6e8-85e8-44a9-aaaa-485f6c6b0b79\" (UID: \"9b4bb6e8-85e8-44a9-aaaa-485f6c6b0b79\") " Oct 30 21:18:23.319817 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:18:23.317665 5378 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9b4bb6e8-85e8-44a9-aaaa-485f6c6b0b79-kube-api-access-jfk9j" pod "9b4bb6e8-85e8-44a9-aaaa-485f6c6b0b79" (UID: "9b4bb6e8-85e8-44a9-aaaa-485f6c6b0b79"). InnerVolumeSpecName "kube-api-access-jfk9j". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:18:23.319817 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:18:23.319338 5378 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9b4bb6e8-85e8-44a9-aaaa-485f6c6b0b79-plugins-config-volume" pod "9b4bb6e8-85e8-44a9-aaaa-485f6c6b0b79" (UID: "9b4bb6e8-85e8-44a9-aaaa-485f6c6b0b79"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:18:23.319817 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:18:23.319781 5378 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jfk9j\" (UniqueName: \"kubernetes.io/projected/9b4bb6e8-85e8-44a9-aaaa-485f6c6b0b79-kube-api-access-jfk9j\") on node \"ip-10-0-138-149.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:18:23.332607 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:18:23.330012 5378 generic.go:356] "Generic (PLEG): container finished" podID="9b4bb6e8-85e8-44a9-aaaa-485f6c6b0b79" containerID="3e25b19e11cd021147005f7589ac25af8e75d13b095e83dd162a5d0e96347bb1" exitCode=0 Oct 30 21:18:23.332607 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:18:23.330198 5378 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w0-781058-6d5676f8cb-fkrgx" event={"ID":"9b4bb6e8-85e8-44a9-aaaa-485f6c6b0b79","Type":"ContainerDied","Data":"3e25b19e11cd021147005f7589ac25af8e75d13b095e83dd162a5d0e96347bb1"} Oct 30 21:18:23.332607 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:18:23.330285 5378 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w0-781058-6d5676f8cb-fkrgx" event={"ID":"9b4bb6e8-85e8-44a9-aaaa-485f6c6b0b79","Type":"ContainerDied","Data":"a5af72ea280f3664443ae07c193dde3dfff2e2022e3277d9d3c283ece25ce541"} Oct 30 21:18:23.332607 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:18:23.330315 5378 scope.go:122] "RemoveContainer" containerID="3e25b19e11cd021147005f7589ac25af8e75d13b095e83dd162a5d0e96347bb1" Oct 30 21:18:23.332607 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:18:23.330480 5378 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w0-781058-6d5676f8cb-fkrgx" Oct 30 21:18:23.409425 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:18:23.409387 5378 scope.go:122] "RemoveContainer" containerID="3e25b19e11cd021147005f7589ac25af8e75d13b095e83dd162a5d0e96347bb1" Oct 30 21:18:23.409966 ip-10-0-138-149 kubenswrapper[5378]: E1030 21:18:23.409937 5378 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3e25b19e11cd021147005f7589ac25af8e75d13b095e83dd162a5d0e96347bb1\": container with ID starting with 3e25b19e11cd021147005f7589ac25af8e75d13b095e83dd162a5d0e96347bb1 not found: ID does not exist" containerID="3e25b19e11cd021147005f7589ac25af8e75d13b095e83dd162a5d0e96347bb1" Oct 30 21:18:23.410076 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:18:23.409979 5378 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3e25b19e11cd021147005f7589ac25af8e75d13b095e83dd162a5d0e96347bb1"} err="failed to get container status \"3e25b19e11cd021147005f7589ac25af8e75d13b095e83dd162a5d0e96347bb1\": rpc error: code = NotFound desc = could not find container \"3e25b19e11cd021147005f7589ac25af8e75d13b095e83dd162a5d0e96347bb1\": container with ID starting with 3e25b19e11cd021147005f7589ac25af8e75d13b095e83dd162a5d0e96347bb1 not found: ID does not exist" Oct 30 21:18:23.423539 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:18:23.423494 5378 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9b4bb6e8-85e8-44a9-aaaa-485f6c6b0b79-plugins-config-volume\") on node \"ip-10-0-138-149.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:18:23.540658 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:18:23.540556 5378 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-781058-6d5676f8cb-fkrgx"] Oct 30 21:18:23.546909 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:18:23.546879 5378 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-781058-6d5676f8cb-fkrgx"] Oct 30 21:18:23.741960 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:18:23.741929 5378 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9b4bb6e8-85e8-44a9-aaaa-485f6c6b0b79" path="/var/lib/kubelet/pods/9b4bb6e8-85e8-44a9-aaaa-485f6c6b0b79/volumes" Oct 30 21:18:26.905331 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:18:26.905299 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:18:28.211577 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:18:28.211540 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:18:28.343284 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:18:28.343241 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:18:28.566645 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:18:28.560045 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:18:30.071928 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:18:30.071893 5378 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-7x9jg_c2ff359c-cf92-4a5b-8610-f6eb41acbe73/ovn-acl-logging/0.log" Oct 30 21:18:30.082939 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:18:30.082914 5378 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-7x9jg_c2ff359c-cf92-4a5b-8610-f6eb41acbe73/ovn-acl-logging/0.log" Oct 30 21:18:32.411881 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:18:32.411854 5378 generic.go:356] "Generic (PLEG): container finished" podID="6614eb26-439d-464e-a7c2-b95744976905" containerID="a76a56d4d1a7618806474066a486dd2bd3c8f5287141fddadcc8dfb270bdd372" exitCode=0 Oct 30 21:18:32.412266 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:18:32.411923 5378 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-639b07-openshift-default-795b89858c-9q4rj" event={"ID":"6614eb26-439d-464e-a7c2-b95744976905","Type":"ContainerDied","Data":"a76a56d4d1a7618806474066a486dd2bd3c8f5287141fddadcc8dfb270bdd372"} Oct 30 21:18:32.412339 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:18:32.412324 5378 scope.go:122] "RemoveContainer" containerID="a76a56d4d1a7618806474066a486dd2bd3c8f5287141fddadcc8dfb270bdd372" Oct 30 21:18:32.413811 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:18:32.413781 5378 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 21:18:32.413907 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:18:32.413852 5378 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 21:18:32.413907 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:18:32.413887 5378 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 21:18:32.672426 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:18:32.672351 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:18:33.374587 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:18:33.373671 5378 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-639b07-openshift-default-795b89858c-9q4rj" Oct 30 21:18:33.420519 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:18:33.420481 5378 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-639b07-openshift-default-795b89858c-9q4rj" event={"ID":"6614eb26-439d-464e-a7c2-b95744976905","Type":"ContainerStarted","Data":"0230011be2a9e2668fc4d0bf5b8c60a9168fb78cfb050d723e26b5415a110000"} Oct 30 21:18:33.747404 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:18:33.747331 5378 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-639b07-openshift-default-795b89858c-9q4rj" Oct 30 21:18:33.748327 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:18:33.748302 5378 patch_prober.go:29] interesting pod/e2e-inf-b-639b07-openshift-default-795b89858c-9q4rj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.160:15021/healthz/ready\": dial tcp 10.133.0.160:15021: connect: connection refused" start-of-body= Oct 30 21:18:33.748463 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:18:33.748368 5378 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-639b07-openshift-default-795b89858c-9q4rj" podUID="6614eb26-439d-464e-a7c2-b95744976905" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.160:15021/healthz/ready\": dial tcp 10.133.0.160:15021: connect: connection refused" Oct 30 21:18:34.748029 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:18:34.747996 5378 patch_prober.go:29] interesting pod/e2e-inf-b-639b07-openshift-default-795b89858c-9q4rj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.160:15021/healthz/ready\": dial tcp 10.133.0.160:15021: connect: connection refused" start-of-body= Oct 30 21:18:34.748427 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:18:34.748056 5378 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-639b07-openshift-default-795b89858c-9q4rj" podUID="6614eb26-439d-464e-a7c2-b95744976905" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.160:15021/healthz/ready\": dial tcp 10.133.0.160:15021: connect: connection refused" Oct 30 21:18:35.748165 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:18:35.748130 5378 patch_prober.go:29] interesting pod/e2e-inf-b-639b07-openshift-default-795b89858c-9q4rj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.160:15021/healthz/ready\": dial tcp 10.133.0.160:15021: connect: connection refused" start-of-body= Oct 30 21:18:35.748552 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:18:35.748191 5378 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-639b07-openshift-default-795b89858c-9q4rj" podUID="6614eb26-439d-464e-a7c2-b95744976905" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.160:15021/healthz/ready\": dial tcp 10.133.0.160:15021: connect: connection refused" Oct 30 21:18:36.748432 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:18:36.748395 5378 patch_prober.go:29] interesting pod/e2e-inf-b-639b07-openshift-default-795b89858c-9q4rj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.160:15021/healthz/ready\": dial tcp 10.133.0.160:15021: connect: connection refused" start-of-body= Oct 30 21:18:36.748876 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:18:36.748452 5378 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-639b07-openshift-default-795b89858c-9q4rj" podUID="6614eb26-439d-464e-a7c2-b95744976905" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.160:15021/healthz/ready\": dial tcp 10.133.0.160:15021: connect: connection refused" Oct 30 21:18:37.596509 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:18:37.596469 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:18:37.748355 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:18:37.748319 5378 patch_prober.go:29] interesting pod/e2e-inf-b-639b07-openshift-default-795b89858c-9q4rj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.160:15021/healthz/ready\": dial tcp 10.133.0.160:15021: connect: connection refused" start-of-body= Oct 30 21:18:37.748532 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:18:37.748375 5378 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-639b07-openshift-default-795b89858c-9q4rj" podUID="6614eb26-439d-464e-a7c2-b95744976905" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.160:15021/healthz/ready\": dial tcp 10.133.0.160:15021: connect: connection refused" Oct 30 21:18:38.752043 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:18:38.752010 5378 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-639b07-openshift-default-795b89858c-9q4rj" Oct 30 21:18:38.752693 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:18:38.752478 5378 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-639b07-openshift-default-795b89858c-9q4rj" Oct 30 21:18:38.753263 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:18:38.753243 5378 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-639b07-openshift-default-795b89858c-9q4rj" Oct 30 21:18:41.106420 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:18:41.106384 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:18:42.399875 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:18:42.399839 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:18:47.292779 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:18:47.292729 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:18:48.841669 ip-10-0-138-149 kubenswrapper[5378]: E1030 21:18:48.841624 5378 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q6mx7" containerName="extractor" Oct 30 21:18:49.805055 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:18:49.805024 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:18:52.064466 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:18:52.064436 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:18:52.193105 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:18:52.193066 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:18:52.717658 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:18:52.717623 5378 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-r9g59"] Oct 30 21:18:55.134586 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:18:55.134531 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:18:56.962647 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:18:56.962614 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:19:00.434200 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:19:00.434161 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:19:01.410334 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:19:01.410294 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:19:02.330264 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:19:02.330220 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:19:03.031277 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:19:03.031235 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:19:03.198912 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:19:03.198875 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:19:03.331578 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:19:03.331476 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:19:03.972028 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:19:03.972001 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:19:08.271508 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:19:08.271447 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:19:10.054968 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:19:10.054930 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:19:10.235243 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:19:10.235187 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:19:10.410927 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:19:10.410896 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:19:13.091023 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:19:13.090990 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:19:17.888888 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:19:17.888856 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:19:18.837395 ip-10-0-138-149 kubenswrapper[5378]: E1030 21:19:18.837355 5378 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q6mx7" containerName="extractor" Oct 30 21:19:19.722513 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:19:19.722479 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:19:22.758743 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:19:22.758690 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:19:27.578280 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:19:27.578243 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:19:28.744977 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:19:28.744938 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:19:30.190689 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:19:30.190654 5378 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-r9g59"] Oct 30 21:19:30.625320 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:19:30.625267 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:19:31.792711 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:19:31.792680 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:19:33.354182 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:19:33.354148 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:19:33.580928 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:19:33.580897 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:19:33.712792 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:19:33.712710 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:19:34.277699 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:19:34.277665 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:19:35.498779 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:19:35.498745 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:19:35.622556 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:19:35.622522 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:19:38.655663 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:19:38.655625 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:19:40.231354 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:19:40.231323 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:19:40.358786 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:19:40.358744 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:19:40.585373 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:19:40.585290 5378 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-f9f277-openshift-default-869d58fb77-p2wtm"] Oct 30 21:19:40.585535 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:19:40.585519 5378 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-f9f277-openshift-default-869d58fb77-p2wtm" podUID="261c5b5a-84f8-4527-9b51-567d6dbf17ca" containerName="istio-proxy" containerID="cri-o://470d851689964db6c4bc6e0521e64737e2b9b70fd5544f90a068a21fbeb8d14f" gracePeriod=30 Oct 30 21:19:40.590546 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:19:40.590519 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:19:40.946923 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:19:40.946891 5378 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-682999-openshift-default-54dcb797b5-r8dcv"] Oct 30 21:19:40.947187 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:19:40.947145 5378 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-682999-openshift-default-54dcb797b5-r8dcv" podUID="9b76fbb2-f5a8-4fea-bdfd-30f40bee7008" containerName="istio-proxy" containerID="cri-o://7966cdde4c6c95b093f0557ff6527111242add1fb6bd6213a8d49c79eea93a0b" gracePeriod=30 Oct 30 21:19:41.613250 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:19:41.613215 5378 patch_prober.go:29] interesting pod/e2e-worker-w1-f9f277-openshift-default-869d58fb77-p2wtm container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 21:19:41.613731 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:19:41.613273 5378 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-f9f277-openshift-default-869d58fb77-p2wtm" podUID="261c5b5a-84f8-4527-9b51-567d6dbf17ca" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 21:19:43.450913 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:19:43.450877 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:19:45.862702 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:19:45.862643 5378 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-f9f277-openshift-default-869d58fb77-p2wtm" Oct 30 21:19:45.885909 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:19:45.885877 5378 generic.go:356] "Generic (PLEG): container finished" podID="261c5b5a-84f8-4527-9b51-567d6dbf17ca" containerID="470d851689964db6c4bc6e0521e64737e2b9b70fd5544f90a068a21fbeb8d14f" exitCode=0 Oct 30 21:19:45.886052 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:19:45.885924 5378 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-f9f277-openshift-default-869d58fb77-p2wtm" Oct 30 21:19:45.886052 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:19:45.885966 5378 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-f9f277-openshift-default-869d58fb77-p2wtm" event={"ID":"261c5b5a-84f8-4527-9b51-567d6dbf17ca","Type":"ContainerDied","Data":"470d851689964db6c4bc6e0521e64737e2b9b70fd5544f90a068a21fbeb8d14f"} Oct 30 21:19:45.886052 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:19:45.886028 5378 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-f9f277-openshift-default-869d58fb77-p2wtm" event={"ID":"261c5b5a-84f8-4527-9b51-567d6dbf17ca","Type":"ContainerDied","Data":"88f7ac03b84fddec1234adb4a47a5f1d4f9cd7ececfa41bd277c72d5720ed713"} Oct 30 21:19:45.886229 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:19:45.886058 5378 scope.go:122] "RemoveContainer" containerID="470d851689964db6c4bc6e0521e64737e2b9b70fd5544f90a068a21fbeb8d14f" Oct 30 21:19:45.896003 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:19:45.895986 5378 scope.go:122] "RemoveContainer" containerID="1ae019c888ef9d4c5b0a8b8d7e7b434bbad1b7c253644c927a9f6aa6276aa201" Oct 30 21:19:45.904787 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:19:45.904721 5378 scope.go:122] "RemoveContainer" containerID="470d851689964db6c4bc6e0521e64737e2b9b70fd5544f90a068a21fbeb8d14f" Oct 30 21:19:45.905010 ip-10-0-138-149 kubenswrapper[5378]: E1030 21:19:45.904987 5378 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"470d851689964db6c4bc6e0521e64737e2b9b70fd5544f90a068a21fbeb8d14f\": container with ID starting with 470d851689964db6c4bc6e0521e64737e2b9b70fd5544f90a068a21fbeb8d14f not found: ID does not exist" containerID="470d851689964db6c4bc6e0521e64737e2b9b70fd5544f90a068a21fbeb8d14f" Oct 30 21:19:45.905094 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:19:45.905016 5378 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"470d851689964db6c4bc6e0521e64737e2b9b70fd5544f90a068a21fbeb8d14f"} err="failed to get container status \"470d851689964db6c4bc6e0521e64737e2b9b70fd5544f90a068a21fbeb8d14f\": rpc error: code = NotFound desc = could not find container \"470d851689964db6c4bc6e0521e64737e2b9b70fd5544f90a068a21fbeb8d14f\": container with ID starting with 470d851689964db6c4bc6e0521e64737e2b9b70fd5544f90a068a21fbeb8d14f not found: ID does not exist" Oct 30 21:19:45.905094 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:19:45.905039 5378 scope.go:122] "RemoveContainer" containerID="1ae019c888ef9d4c5b0a8b8d7e7b434bbad1b7c253644c927a9f6aa6276aa201" Oct 30 21:19:45.905271 ip-10-0-138-149 kubenswrapper[5378]: E1030 21:19:45.905249 5378 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1ae019c888ef9d4c5b0a8b8d7e7b434bbad1b7c253644c927a9f6aa6276aa201\": container with ID starting with 1ae019c888ef9d4c5b0a8b8d7e7b434bbad1b7c253644c927a9f6aa6276aa201 not found: ID does not exist" containerID="1ae019c888ef9d4c5b0a8b8d7e7b434bbad1b7c253644c927a9f6aa6276aa201" Oct 30 21:19:45.905316 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:19:45.905282 5378 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1ae019c888ef9d4c5b0a8b8d7e7b434bbad1b7c253644c927a9f6aa6276aa201"} err="failed to get container status \"1ae019c888ef9d4c5b0a8b8d7e7b434bbad1b7c253644c927a9f6aa6276aa201\": rpc error: code = NotFound desc = could not find container \"1ae019c888ef9d4c5b0a8b8d7e7b434bbad1b7c253644c927a9f6aa6276aa201\": container with ID starting with 1ae019c888ef9d4c5b0a8b8d7e7b434bbad1b7c253644c927a9f6aa6276aa201 not found: ID does not exist" Oct 30 21:19:45.955355 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:19:45.955326 5378 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/261c5b5a-84f8-4527-9b51-567d6dbf17ca-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/261c5b5a-84f8-4527-9b51-567d6dbf17ca-istio-data\") pod \"261c5b5a-84f8-4527-9b51-567d6dbf17ca\" (UID: \"261c5b5a-84f8-4527-9b51-567d6dbf17ca\") " Oct 30 21:19:45.955486 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:19:45.955365 5378 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/261c5b5a-84f8-4527-9b51-567d6dbf17ca-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/261c5b5a-84f8-4527-9b51-567d6dbf17ca-istio-envoy\") pod \"261c5b5a-84f8-4527-9b51-567d6dbf17ca\" (UID: \"261c5b5a-84f8-4527-9b51-567d6dbf17ca\") " Oct 30 21:19:45.955486 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:19:45.955439 5378 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/261c5b5a-84f8-4527-9b51-567d6dbf17ca-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/261c5b5a-84f8-4527-9b51-567d6dbf17ca-istio-podinfo\") pod \"261c5b5a-84f8-4527-9b51-567d6dbf17ca\" (UID: \"261c5b5a-84f8-4527-9b51-567d6dbf17ca\") " Oct 30 21:19:45.955616 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:19:45.955538 5378 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/261c5b5a-84f8-4527-9b51-567d6dbf17ca-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/261c5b5a-84f8-4527-9b51-567d6dbf17ca-workload-socket\") pod \"261c5b5a-84f8-4527-9b51-567d6dbf17ca\" (UID: \"261c5b5a-84f8-4527-9b51-567d6dbf17ca\") " Oct 30 21:19:45.955616 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:19:45.955593 5378 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/261c5b5a-84f8-4527-9b51-567d6dbf17ca-istio-token\" (UniqueName: \"kubernetes.io/projected/261c5b5a-84f8-4527-9b51-567d6dbf17ca-istio-token\") pod \"261c5b5a-84f8-4527-9b51-567d6dbf17ca\" (UID: \"261c5b5a-84f8-4527-9b51-567d6dbf17ca\") " Oct 30 21:19:45.955717 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:19:45.955575 5378 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/261c5b5a-84f8-4527-9b51-567d6dbf17ca-istio-data" pod "261c5b5a-84f8-4527-9b51-567d6dbf17ca" (UID: "261c5b5a-84f8-4527-9b51-567d6dbf17ca"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:19:45.955717 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:19:45.955678 5378 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/261c5b5a-84f8-4527-9b51-567d6dbf17ca-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/261c5b5a-84f8-4527-9b51-567d6dbf17ca-workload-certs\") pod \"261c5b5a-84f8-4527-9b51-567d6dbf17ca\" (UID: \"261c5b5a-84f8-4527-9b51-567d6dbf17ca\") " Oct 30 21:19:45.955823 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:19:45.955716 5378 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/261c5b5a-84f8-4527-9b51-567d6dbf17ca-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/261c5b5a-84f8-4527-9b51-567d6dbf17ca-istiod-ca-cert\") pod \"261c5b5a-84f8-4527-9b51-567d6dbf17ca\" (UID: \"261c5b5a-84f8-4527-9b51-567d6dbf17ca\") " Oct 30 21:19:45.955823 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:19:45.955786 5378 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/261c5b5a-84f8-4527-9b51-567d6dbf17ca-workload-socket" pod "261c5b5a-84f8-4527-9b51-567d6dbf17ca" (UID: "261c5b5a-84f8-4527-9b51-567d6dbf17ca"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:19:45.955823 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:19:45.955789 5378 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/261c5b5a-84f8-4527-9b51-567d6dbf17ca-kube-api-access-2hdv5\" (UniqueName: \"kubernetes.io/projected/261c5b5a-84f8-4527-9b51-567d6dbf17ca-kube-api-access-2hdv5\") pod \"261c5b5a-84f8-4527-9b51-567d6dbf17ca\" (UID: \"261c5b5a-84f8-4527-9b51-567d6dbf17ca\") " Oct 30 21:19:45.955977 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:19:45.955838 5378 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/261c5b5a-84f8-4527-9b51-567d6dbf17ca-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/261c5b5a-84f8-4527-9b51-567d6dbf17ca-credential-socket\") pod \"261c5b5a-84f8-4527-9b51-567d6dbf17ca\" (UID: \"261c5b5a-84f8-4527-9b51-567d6dbf17ca\") " Oct 30 21:19:45.956199 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:19:45.956174 5378 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/261c5b5a-84f8-4527-9b51-567d6dbf17ca-credential-socket" pod "261c5b5a-84f8-4527-9b51-567d6dbf17ca" (UID: "261c5b5a-84f8-4527-9b51-567d6dbf17ca"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:19:45.956261 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:19:45.956217 5378 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/261c5b5a-84f8-4527-9b51-567d6dbf17ca-istiod-ca-cert" pod "261c5b5a-84f8-4527-9b51-567d6dbf17ca" (UID: "261c5b5a-84f8-4527-9b51-567d6dbf17ca"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:19:45.956319 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:19:45.956285 5378 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/261c5b5a-84f8-4527-9b51-567d6dbf17ca-workload-certs" pod "261c5b5a-84f8-4527-9b51-567d6dbf17ca" (UID: "261c5b5a-84f8-4527-9b51-567d6dbf17ca"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:19:45.956554 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:19:45.956509 5378 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/261c5b5a-84f8-4527-9b51-567d6dbf17ca-workload-certs\") on node \"ip-10-0-138-149.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:19:45.956554 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:19:45.956536 5378 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/261c5b5a-84f8-4527-9b51-567d6dbf17ca-istiod-ca-cert\") on node \"ip-10-0-138-149.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:19:45.956554 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:19:45.956551 5378 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/261c5b5a-84f8-4527-9b51-567d6dbf17ca-credential-socket\") on node \"ip-10-0-138-149.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:19:45.956812 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:19:45.956589 5378 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/261c5b5a-84f8-4527-9b51-567d6dbf17ca-istio-data\") on node \"ip-10-0-138-149.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:19:45.956812 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:19:45.956614 5378 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/261c5b5a-84f8-4527-9b51-567d6dbf17ca-workload-socket\") on node \"ip-10-0-138-149.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:19:45.957843 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:19:45.957824 5378 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/261c5b5a-84f8-4527-9b51-567d6dbf17ca-istio-envoy" pod "261c5b5a-84f8-4527-9b51-567d6dbf17ca" (UID: "261c5b5a-84f8-4527-9b51-567d6dbf17ca"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:19:45.958171 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:19:45.958155 5378 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/261c5b5a-84f8-4527-9b51-567d6dbf17ca-istio-token" pod "261c5b5a-84f8-4527-9b51-567d6dbf17ca" (UID: "261c5b5a-84f8-4527-9b51-567d6dbf17ca"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:19:45.958322 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:19:45.958298 5378 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/261c5b5a-84f8-4527-9b51-567d6dbf17ca-istio-podinfo" pod "261c5b5a-84f8-4527-9b51-567d6dbf17ca" (UID: "261c5b5a-84f8-4527-9b51-567d6dbf17ca"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 21:19:45.958428 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:19:45.958322 5378 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/261c5b5a-84f8-4527-9b51-567d6dbf17ca-kube-api-access-2hdv5" pod "261c5b5a-84f8-4527-9b51-567d6dbf17ca" (UID: "261c5b5a-84f8-4527-9b51-567d6dbf17ca"). InnerVolumeSpecName "kube-api-access-2hdv5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:19:46.058506 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:19:46.058473 5378 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2hdv5\" (UniqueName: \"kubernetes.io/projected/261c5b5a-84f8-4527-9b51-567d6dbf17ca-kube-api-access-2hdv5\") on node \"ip-10-0-138-149.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:19:46.058506 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:19:46.058506 5378 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/261c5b5a-84f8-4527-9b51-567d6dbf17ca-istio-envoy\") on node \"ip-10-0-138-149.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:19:46.058506 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:19:46.058522 5378 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/261c5b5a-84f8-4527-9b51-567d6dbf17ca-istio-podinfo\") on node \"ip-10-0-138-149.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:19:46.060091 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:19:46.058534 5378 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/261c5b5a-84f8-4527-9b51-567d6dbf17ca-istio-token\") on node \"ip-10-0-138-149.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:19:46.200885 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:19:46.200863 5378 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-682999-openshift-default-54dcb797b5-r8dcv" Oct 30 21:19:46.309825 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:19:46.309794 5378 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-f9f277-openshift-default-869d58fb77-p2wtm"] Oct 30 21:19:46.313253 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:19:46.313219 5378 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-f9f277-openshift-default-869d58fb77-p2wtm"] Oct 30 21:19:46.361527 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:19:46.361496 5378 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9b76fbb2-f5a8-4fea-bdfd-30f40bee7008-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9b76fbb2-f5a8-4fea-bdfd-30f40bee7008-istio-envoy\") pod \"9b76fbb2-f5a8-4fea-bdfd-30f40bee7008\" (UID: \"9b76fbb2-f5a8-4fea-bdfd-30f40bee7008\") " Oct 30 21:19:46.361720 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:19:46.361547 5378 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9b76fbb2-f5a8-4fea-bdfd-30f40bee7008-kube-api-access-gxkn6\" (UniqueName: \"kubernetes.io/projected/9b76fbb2-f5a8-4fea-bdfd-30f40bee7008-kube-api-access-gxkn6\") pod \"9b76fbb2-f5a8-4fea-bdfd-30f40bee7008\" (UID: \"9b76fbb2-f5a8-4fea-bdfd-30f40bee7008\") " Oct 30 21:19:46.361720 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:19:46.361601 5378 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9b76fbb2-f5a8-4fea-bdfd-30f40bee7008-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9b76fbb2-f5a8-4fea-bdfd-30f40bee7008-workload-certs\") pod \"9b76fbb2-f5a8-4fea-bdfd-30f40bee7008\" (UID: \"9b76fbb2-f5a8-4fea-bdfd-30f40bee7008\") " Oct 30 21:19:46.361720 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:19:46.361628 5378 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9b76fbb2-f5a8-4fea-bdfd-30f40bee7008-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9b76fbb2-f5a8-4fea-bdfd-30f40bee7008-credential-socket\") pod \"9b76fbb2-f5a8-4fea-bdfd-30f40bee7008\" (UID: \"9b76fbb2-f5a8-4fea-bdfd-30f40bee7008\") " Oct 30 21:19:46.361720 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:19:46.361710 5378 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/9b76fbb2-f5a8-4fea-bdfd-30f40bee7008-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9b76fbb2-f5a8-4fea-bdfd-30f40bee7008-istio-podinfo\") pod \"9b76fbb2-f5a8-4fea-bdfd-30f40bee7008\" (UID: \"9b76fbb2-f5a8-4fea-bdfd-30f40bee7008\") " Oct 30 21:19:46.361957 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:19:46.361750 5378 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9b76fbb2-f5a8-4fea-bdfd-30f40bee7008-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9b76fbb2-f5a8-4fea-bdfd-30f40bee7008-istiod-ca-cert\") pod \"9b76fbb2-f5a8-4fea-bdfd-30f40bee7008\" (UID: \"9b76fbb2-f5a8-4fea-bdfd-30f40bee7008\") " Oct 30 21:19:46.361957 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:19:46.361834 5378 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9b76fbb2-f5a8-4fea-bdfd-30f40bee7008-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9b76fbb2-f5a8-4fea-bdfd-30f40bee7008-workload-socket\") pod \"9b76fbb2-f5a8-4fea-bdfd-30f40bee7008\" (UID: \"9b76fbb2-f5a8-4fea-bdfd-30f40bee7008\") " Oct 30 21:19:46.361957 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:19:46.361895 5378 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9b76fbb2-f5a8-4fea-bdfd-30f40bee7008-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9b76fbb2-f5a8-4fea-bdfd-30f40bee7008-istio-data\") pod \"9b76fbb2-f5a8-4fea-bdfd-30f40bee7008\" (UID: \"9b76fbb2-f5a8-4fea-bdfd-30f40bee7008\") " Oct 30 21:19:46.361957 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:19:46.361938 5378 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9b76fbb2-f5a8-4fea-bdfd-30f40bee7008-istio-token\" (UniqueName: \"kubernetes.io/projected/9b76fbb2-f5a8-4fea-bdfd-30f40bee7008-istio-token\") pod \"9b76fbb2-f5a8-4fea-bdfd-30f40bee7008\" (UID: \"9b76fbb2-f5a8-4fea-bdfd-30f40bee7008\") " Oct 30 21:19:46.361957 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:19:46.361933 5378 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9b76fbb2-f5a8-4fea-bdfd-30f40bee7008-credential-socket" pod "9b76fbb2-f5a8-4fea-bdfd-30f40bee7008" (UID: "9b76fbb2-f5a8-4fea-bdfd-30f40bee7008"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:19:46.361957 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:19:46.361947 5378 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9b76fbb2-f5a8-4fea-bdfd-30f40bee7008-workload-certs" pod "9b76fbb2-f5a8-4fea-bdfd-30f40bee7008" (UID: "9b76fbb2-f5a8-4fea-bdfd-30f40bee7008"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:19:46.362338 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:19:46.362180 5378 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9b76fbb2-f5a8-4fea-bdfd-30f40bee7008-istiod-ca-cert" pod "9b76fbb2-f5a8-4fea-bdfd-30f40bee7008" (UID: "9b76fbb2-f5a8-4fea-bdfd-30f40bee7008"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:19:46.362338 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:19:46.362277 5378 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9b76fbb2-f5a8-4fea-bdfd-30f40bee7008-istio-data" pod "9b76fbb2-f5a8-4fea-bdfd-30f40bee7008" (UID: "9b76fbb2-f5a8-4fea-bdfd-30f40bee7008"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:19:46.362338 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:19:46.362289 5378 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9b76fbb2-f5a8-4fea-bdfd-30f40bee7008-workload-socket" pod "9b76fbb2-f5a8-4fea-bdfd-30f40bee7008" (UID: "9b76fbb2-f5a8-4fea-bdfd-30f40bee7008"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:19:46.362619 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:19:46.362597 5378 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9b76fbb2-f5a8-4fea-bdfd-30f40bee7008-workload-socket\") on node \"ip-10-0-138-149.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:19:46.362619 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:19:46.362626 5378 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9b76fbb2-f5a8-4fea-bdfd-30f40bee7008-istio-data\") on node \"ip-10-0-138-149.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:19:46.362791 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:19:46.362643 5378 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9b76fbb2-f5a8-4fea-bdfd-30f40bee7008-workload-certs\") on node \"ip-10-0-138-149.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:19:46.362791 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:19:46.362658 5378 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9b76fbb2-f5a8-4fea-bdfd-30f40bee7008-credential-socket\") on node \"ip-10-0-138-149.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:19:46.362791 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:19:46.362672 5378 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9b76fbb2-f5a8-4fea-bdfd-30f40bee7008-istiod-ca-cert\") on node \"ip-10-0-138-149.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:19:46.363946 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:19:46.363919 5378 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9b76fbb2-f5a8-4fea-bdfd-30f40bee7008-istio-envoy" pod "9b76fbb2-f5a8-4fea-bdfd-30f40bee7008" (UID: "9b76fbb2-f5a8-4fea-bdfd-30f40bee7008"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:19:46.364054 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:19:46.363972 5378 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/9b76fbb2-f5a8-4fea-bdfd-30f40bee7008-istio-podinfo" pod "9b76fbb2-f5a8-4fea-bdfd-30f40bee7008" (UID: "9b76fbb2-f5a8-4fea-bdfd-30f40bee7008"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 21:19:46.364054 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:19:46.364013 5378 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9b76fbb2-f5a8-4fea-bdfd-30f40bee7008-kube-api-access-gxkn6" pod "9b76fbb2-f5a8-4fea-bdfd-30f40bee7008" (UID: "9b76fbb2-f5a8-4fea-bdfd-30f40bee7008"). InnerVolumeSpecName "kube-api-access-gxkn6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:19:46.364150 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:19:46.364120 5378 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9b76fbb2-f5a8-4fea-bdfd-30f40bee7008-istio-token" pod "9b76fbb2-f5a8-4fea-bdfd-30f40bee7008" (UID: "9b76fbb2-f5a8-4fea-bdfd-30f40bee7008"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:19:46.464037 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:19:46.463959 5378 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9b76fbb2-f5a8-4fea-bdfd-30f40bee7008-istio-podinfo\") on node \"ip-10-0-138-149.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:19:46.464037 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:19:46.463990 5378 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9b76fbb2-f5a8-4fea-bdfd-30f40bee7008-istio-token\") on node \"ip-10-0-138-149.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:19:46.464037 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:19:46.463999 5378 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9b76fbb2-f5a8-4fea-bdfd-30f40bee7008-istio-envoy\") on node \"ip-10-0-138-149.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:19:46.464037 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:19:46.464008 5378 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gxkn6\" (UniqueName: \"kubernetes.io/projected/9b76fbb2-f5a8-4fea-bdfd-30f40bee7008-kube-api-access-gxkn6\") on node \"ip-10-0-138-149.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:19:46.894341 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:19:46.894304 5378 generic.go:356] "Generic (PLEG): container finished" podID="9b76fbb2-f5a8-4fea-bdfd-30f40bee7008" containerID="7966cdde4c6c95b093f0557ff6527111242add1fb6bd6213a8d49c79eea93a0b" exitCode=0 Oct 30 21:19:46.894818 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:19:46.894379 5378 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-682999-openshift-default-54dcb797b5-r8dcv" Oct 30 21:19:46.894818 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:19:46.894393 5378 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-682999-openshift-default-54dcb797b5-r8dcv" event={"ID":"9b76fbb2-f5a8-4fea-bdfd-30f40bee7008","Type":"ContainerDied","Data":"7966cdde4c6c95b093f0557ff6527111242add1fb6bd6213a8d49c79eea93a0b"} Oct 30 21:19:46.894818 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:19:46.894449 5378 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-682999-openshift-default-54dcb797b5-r8dcv" event={"ID":"9b76fbb2-f5a8-4fea-bdfd-30f40bee7008","Type":"ContainerDied","Data":"8bc12fb5ff424f5a3e57cd4a0f2736b0b620bb112edb9633333d6ec65b2afd7b"} Oct 30 21:19:46.894818 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:19:46.894471 5378 scope.go:122] "RemoveContainer" containerID="7966cdde4c6c95b093f0557ff6527111242add1fb6bd6213a8d49c79eea93a0b" Oct 30 21:19:46.904558 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:19:46.904540 5378 scope.go:122] "RemoveContainer" containerID="f371283ad7cbbc8f6a41616e270dfc8087188b60af99301046ead6a2cc9070d9" Oct 30 21:19:46.915643 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:19:46.915628 5378 scope.go:122] "RemoveContainer" containerID="7966cdde4c6c95b093f0557ff6527111242add1fb6bd6213a8d49c79eea93a0b" Oct 30 21:19:46.915932 ip-10-0-138-149 kubenswrapper[5378]: E1030 21:19:46.915912 5378 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7966cdde4c6c95b093f0557ff6527111242add1fb6bd6213a8d49c79eea93a0b\": container with ID starting with 7966cdde4c6c95b093f0557ff6527111242add1fb6bd6213a8d49c79eea93a0b not found: ID does not exist" containerID="7966cdde4c6c95b093f0557ff6527111242add1fb6bd6213a8d49c79eea93a0b" Oct 30 21:19:46.915979 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:19:46.915941 5378 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7966cdde4c6c95b093f0557ff6527111242add1fb6bd6213a8d49c79eea93a0b"} err="failed to get container status \"7966cdde4c6c95b093f0557ff6527111242add1fb6bd6213a8d49c79eea93a0b\": rpc error: code = NotFound desc = could not find container \"7966cdde4c6c95b093f0557ff6527111242add1fb6bd6213a8d49c79eea93a0b\": container with ID starting with 7966cdde4c6c95b093f0557ff6527111242add1fb6bd6213a8d49c79eea93a0b not found: ID does not exist" Oct 30 21:19:46.915979 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:19:46.915959 5378 scope.go:122] "RemoveContainer" containerID="f371283ad7cbbc8f6a41616e270dfc8087188b60af99301046ead6a2cc9070d9" Oct 30 21:19:46.916187 ip-10-0-138-149 kubenswrapper[5378]: E1030 21:19:46.916173 5378 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f371283ad7cbbc8f6a41616e270dfc8087188b60af99301046ead6a2cc9070d9\": container with ID starting with f371283ad7cbbc8f6a41616e270dfc8087188b60af99301046ead6a2cc9070d9 not found: ID does not exist" containerID="f371283ad7cbbc8f6a41616e270dfc8087188b60af99301046ead6a2cc9070d9" Oct 30 21:19:46.916233 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:19:46.916191 5378 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f371283ad7cbbc8f6a41616e270dfc8087188b60af99301046ead6a2cc9070d9"} err="failed to get container status \"f371283ad7cbbc8f6a41616e270dfc8087188b60af99301046ead6a2cc9070d9\": rpc error: code = NotFound desc = could not find container \"f371283ad7cbbc8f6a41616e270dfc8087188b60af99301046ead6a2cc9070d9\": container with ID starting with f371283ad7cbbc8f6a41616e270dfc8087188b60af99301046ead6a2cc9070d9 not found: ID does not exist" Oct 30 21:19:47.061223 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:19:47.061186 5378 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-682999-openshift-default-54dcb797b5-r8dcv"] Oct 30 21:19:47.111418 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:19:47.111387 5378 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-682999-openshift-default-54dcb797b5-r8dcv"] Oct 30 21:19:47.741439 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:19:47.741407 5378 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="261c5b5a-84f8-4527-9b51-567d6dbf17ca" path="/var/lib/kubelet/pods/261c5b5a-84f8-4527-9b51-567d6dbf17ca/volumes" Oct 30 21:19:47.741929 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:19:47.741913 5378 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9b76fbb2-f5a8-4fea-bdfd-30f40bee7008" path="/var/lib/kubelet/pods/9b76fbb2-f5a8-4fea-bdfd-30f40bee7008/volumes" Oct 30 21:19:48.452096 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:19:48.452065 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:19:48.836662 ip-10-0-138-149 kubenswrapper[5378]: E1030 21:19:48.836559 5378 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q6mx7" containerName="extractor" Oct 30 21:19:50.155239 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:19:50.155207 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:19:53.241676 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:19:53.241637 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:19:57.821106 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:19:57.821069 5378 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-b-639b07/test-model-639b07-kserve-b8485fd9-ph8tq"] Oct 30 21:19:57.821637 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:19:57.821618 5378 cpu_manager.go:414] "containerMap: removing container" podUID="dc662e9d-914c-4dfb-a86f-0fb048f7b690" containerName="main" Oct 30 21:19:57.821690 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:19:57.821639 5378 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dc662e9d-914c-4dfb-a86f-0fb048f7b690" containerName="main" Oct 30 21:19:57.821690 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:19:57.821657 5378 cpu_manager.go:414] "containerMap: removing container" podUID="c360b269-e417-49b5-9482-34e06924da36" containerName="payload-processing" Oct 30 21:19:57.821690 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:19:57.821663 5378 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c360b269-e417-49b5-9482-34e06924da36" containerName="payload-processing" Oct 30 21:19:57.821690 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:19:57.821683 5378 cpu_manager.go:414] "containerMap: removing container" podUID="9b76fbb2-f5a8-4fea-bdfd-30f40bee7008" containerName="istio-proxy" Oct 30 21:19:57.821690 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:19:57.821688 5378 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9b76fbb2-f5a8-4fea-bdfd-30f40bee7008" containerName="istio-proxy" Oct 30 21:19:57.821856 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:19:57.821697 5378 cpu_manager.go:414] "containerMap: removing container" podUID="261c5b5a-84f8-4527-9b51-567d6dbf17ca" containerName="istio-proxy" Oct 30 21:19:57.821856 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:19:57.821702 5378 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="261c5b5a-84f8-4527-9b51-567d6dbf17ca" containerName="istio-proxy" Oct 30 21:19:57.821856 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:19:57.821710 5378 cpu_manager.go:414] "containerMap: removing container" podUID="a9678fc2-1c56-4d45-9007-eee5049e0f48" containerName="main" Oct 30 21:19:57.821856 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:19:57.821715 5378 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a9678fc2-1c56-4d45-9007-eee5049e0f48" containerName="main" Oct 30 21:19:57.821856 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:19:57.821723 5378 cpu_manager.go:414] "containerMap: removing container" podUID="9b4bb6e8-85e8-44a9-aaaa-485f6c6b0b79" containerName="payload-processing" Oct 30 21:19:57.821856 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:19:57.821727 5378 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9b4bb6e8-85e8-44a9-aaaa-485f6c6b0b79" containerName="payload-processing" Oct 30 21:19:57.821856 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:19:57.821733 5378 cpu_manager.go:414] "containerMap: removing container" podUID="261c5b5a-84f8-4527-9b51-567d6dbf17ca" containerName="istio-proxy" Oct 30 21:19:57.821856 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:19:57.821737 5378 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="261c5b5a-84f8-4527-9b51-567d6dbf17ca" containerName="istio-proxy" Oct 30 21:19:57.821856 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:19:57.821750 5378 cpu_manager.go:414] "containerMap: removing container" podUID="9eb6ff54-b77b-453c-a166-988b98cf8f49" containerName="main" Oct 30 21:19:57.821856 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:19:57.821754 5378 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9eb6ff54-b77b-453c-a166-988b98cf8f49" containerName="main" Oct 30 21:19:57.821856 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:19:57.821768 5378 cpu_manager.go:414] "containerMap: removing container" podUID="76bab521-3612-401b-a99a-8e8be187baef" containerName="payload-pre-processing" Oct 30 21:19:57.821856 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:19:57.821773 5378 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="76bab521-3612-401b-a99a-8e8be187baef" containerName="payload-pre-processing" Oct 30 21:19:57.821856 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:19:57.821860 5378 memory_manager.go:365] "RemoveStaleState removing state" podUID="76bab521-3612-401b-a99a-8e8be187baef" containerName="payload-pre-processing" Oct 30 21:19:57.822244 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:19:57.821868 5378 memory_manager.go:365] "RemoveStaleState removing state" podUID="dc662e9d-914c-4dfb-a86f-0fb048f7b690" containerName="main" Oct 30 21:19:57.822244 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:19:57.821876 5378 memory_manager.go:365] "RemoveStaleState removing state" podUID="c360b269-e417-49b5-9482-34e06924da36" containerName="payload-processing" Oct 30 21:19:57.822244 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:19:57.821885 5378 memory_manager.go:365] "RemoveStaleState removing state" podUID="9b76fbb2-f5a8-4fea-bdfd-30f40bee7008" containerName="istio-proxy" Oct 30 21:19:57.822244 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:19:57.821891 5378 memory_manager.go:365] "RemoveStaleState removing state" podUID="9b76fbb2-f5a8-4fea-bdfd-30f40bee7008" containerName="istio-proxy" Oct 30 21:19:57.822244 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:19:57.821898 5378 memory_manager.go:365] "RemoveStaleState removing state" podUID="261c5b5a-84f8-4527-9b51-567d6dbf17ca" containerName="istio-proxy" Oct 30 21:19:57.822244 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:19:57.821902 5378 memory_manager.go:365] "RemoveStaleState removing state" podUID="a9678fc2-1c56-4d45-9007-eee5049e0f48" containerName="main" Oct 30 21:19:57.822244 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:19:57.821910 5378 memory_manager.go:365] "RemoveStaleState removing state" podUID="9b4bb6e8-85e8-44a9-aaaa-485f6c6b0b79" containerName="payload-processing" Oct 30 21:19:57.822244 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:19:57.821916 5378 memory_manager.go:365] "RemoveStaleState removing state" podUID="9eb6ff54-b77b-453c-a166-988b98cf8f49" containerName="main" Oct 30 21:19:57.825021 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:19:57.825001 5378 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-639b07/test-model-639b07-kserve-b8485fd9-ph8tq" Oct 30 21:19:57.882117 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:19:57.882083 5378 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-639b07/test-model-639b07-kserve-b8485fd9-ph8tq"] Oct 30 21:19:57.894752 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:19:57.894725 5378 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e96f3b44-86e7-4389-a90f-e0fa736cffd1-tmp-dir\") pod \"test-model-639b07-kserve-b8485fd9-ph8tq\" (UID: \"e96f3b44-86e7-4389-a90f-e0fa736cffd1\") " pod="ai-tenant-e2e-inf-b-639b07/test-model-639b07-kserve-b8485fd9-ph8tq" Oct 30 21:19:57.894899 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:19:57.894786 5378 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e96f3b44-86e7-4389-a90f-e0fa736cffd1-home\") pod \"test-model-639b07-kserve-b8485fd9-ph8tq\" (UID: \"e96f3b44-86e7-4389-a90f-e0fa736cffd1\") " pod="ai-tenant-e2e-inf-b-639b07/test-model-639b07-kserve-b8485fd9-ph8tq" Oct 30 21:19:57.894956 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:19:57.894892 5378 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e96f3b44-86e7-4389-a90f-e0fa736cffd1-model-cache\") pod \"test-model-639b07-kserve-b8485fd9-ph8tq\" (UID: \"e96f3b44-86e7-4389-a90f-e0fa736cffd1\") " pod="ai-tenant-e2e-inf-b-639b07/test-model-639b07-kserve-b8485fd9-ph8tq" Oct 30 21:19:57.894956 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:19:57.894949 5378 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b9gn7\" (UniqueName: \"kubernetes.io/projected/e96f3b44-86e7-4389-a90f-e0fa736cffd1-kube-api-access-b9gn7\") pod \"test-model-639b07-kserve-b8485fd9-ph8tq\" (UID: \"e96f3b44-86e7-4389-a90f-e0fa736cffd1\") " pod="ai-tenant-e2e-inf-b-639b07/test-model-639b07-kserve-b8485fd9-ph8tq" Oct 30 21:19:57.895055 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:19:57.894995 5378 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e96f3b44-86e7-4389-a90f-e0fa736cffd1-tls-certs\") pod \"test-model-639b07-kserve-b8485fd9-ph8tq\" (UID: \"e96f3b44-86e7-4389-a90f-e0fa736cffd1\") " pod="ai-tenant-e2e-inf-b-639b07/test-model-639b07-kserve-b8485fd9-ph8tq" Oct 30 21:19:57.895091 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:19:57.895075 5378 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e96f3b44-86e7-4389-a90f-e0fa736cffd1-dshm\") pod \"test-model-639b07-kserve-b8485fd9-ph8tq\" (UID: \"e96f3b44-86e7-4389-a90f-e0fa736cffd1\") " pod="ai-tenant-e2e-inf-b-639b07/test-model-639b07-kserve-b8485fd9-ph8tq" Oct 30 21:19:57.897408 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:19:57.897383 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:19:57.929076 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:19:57.929042 5378 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-639b07\"/\"default-dockercfg-wqp9v\"" Oct 30 21:19:57.929733 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:19:57.929713 5378 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-639b07\"/\"openshift-service-ca.crt\"" Oct 30 21:19:57.929807 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:19:57.929713 5378 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-639b07\"/\"test-model-639b07-kserve-self-signed-certs\"" Oct 30 21:19:57.929807 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:19:57.929715 5378 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-639b07\"/\"kube-root-ca.crt\"" Oct 30 21:19:57.996632 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:19:57.996603 5378 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e96f3b44-86e7-4389-a90f-e0fa736cffd1-tls-certs\") pod \"test-model-639b07-kserve-b8485fd9-ph8tq\" (UID: \"e96f3b44-86e7-4389-a90f-e0fa736cffd1\") " pod="ai-tenant-e2e-inf-b-639b07/test-model-639b07-kserve-b8485fd9-ph8tq" Oct 30 21:19:57.996801 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:19:57.996646 5378 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e96f3b44-86e7-4389-a90f-e0fa736cffd1-dshm\") pod \"test-model-639b07-kserve-b8485fd9-ph8tq\" (UID: \"e96f3b44-86e7-4389-a90f-e0fa736cffd1\") " pod="ai-tenant-e2e-inf-b-639b07/test-model-639b07-kserve-b8485fd9-ph8tq" Oct 30 21:19:57.996801 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:19:57.996703 5378 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e96f3b44-86e7-4389-a90f-e0fa736cffd1-tmp-dir\") pod \"test-model-639b07-kserve-b8485fd9-ph8tq\" (UID: \"e96f3b44-86e7-4389-a90f-e0fa736cffd1\") " pod="ai-tenant-e2e-inf-b-639b07/test-model-639b07-kserve-b8485fd9-ph8tq" Oct 30 21:19:57.996801 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:19:57.996755 5378 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e96f3b44-86e7-4389-a90f-e0fa736cffd1-home\") pod \"test-model-639b07-kserve-b8485fd9-ph8tq\" (UID: \"e96f3b44-86e7-4389-a90f-e0fa736cffd1\") " pod="ai-tenant-e2e-inf-b-639b07/test-model-639b07-kserve-b8485fd9-ph8tq" Oct 30 21:19:57.996801 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:19:57.996774 5378 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e96f3b44-86e7-4389-a90f-e0fa736cffd1-model-cache\") pod \"test-model-639b07-kserve-b8485fd9-ph8tq\" (UID: \"e96f3b44-86e7-4389-a90f-e0fa736cffd1\") " pod="ai-tenant-e2e-inf-b-639b07/test-model-639b07-kserve-b8485fd9-ph8tq" Oct 30 21:19:57.997026 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:19:57.996834 5378 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b9gn7\" (UniqueName: \"kubernetes.io/projected/e96f3b44-86e7-4389-a90f-e0fa736cffd1-kube-api-access-b9gn7\") pod \"test-model-639b07-kserve-b8485fd9-ph8tq\" (UID: \"e96f3b44-86e7-4389-a90f-e0fa736cffd1\") " pod="ai-tenant-e2e-inf-b-639b07/test-model-639b07-kserve-b8485fd9-ph8tq" Oct 30 21:19:57.997165 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:19:57.997138 5378 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e96f3b44-86e7-4389-a90f-e0fa736cffd1-tmp-dir\") pod \"test-model-639b07-kserve-b8485fd9-ph8tq\" (UID: \"e96f3b44-86e7-4389-a90f-e0fa736cffd1\") " pod="ai-tenant-e2e-inf-b-639b07/test-model-639b07-kserve-b8485fd9-ph8tq" Oct 30 21:19:57.997228 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:19:57.997167 5378 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e96f3b44-86e7-4389-a90f-e0fa736cffd1-model-cache\") pod \"test-model-639b07-kserve-b8485fd9-ph8tq\" (UID: \"e96f3b44-86e7-4389-a90f-e0fa736cffd1\") " pod="ai-tenant-e2e-inf-b-639b07/test-model-639b07-kserve-b8485fd9-ph8tq" Oct 30 21:19:57.997270 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:19:57.997239 5378 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e96f3b44-86e7-4389-a90f-e0fa736cffd1-home\") pod \"test-model-639b07-kserve-b8485fd9-ph8tq\" (UID: \"e96f3b44-86e7-4389-a90f-e0fa736cffd1\") " pod="ai-tenant-e2e-inf-b-639b07/test-model-639b07-kserve-b8485fd9-ph8tq" Oct 30 21:19:57.998979 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:19:57.998959 5378 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e96f3b44-86e7-4389-a90f-e0fa736cffd1-dshm\") pod \"test-model-639b07-kserve-b8485fd9-ph8tq\" (UID: \"e96f3b44-86e7-4389-a90f-e0fa736cffd1\") " pod="ai-tenant-e2e-inf-b-639b07/test-model-639b07-kserve-b8485fd9-ph8tq" Oct 30 21:19:57.999235 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:19:57.999217 5378 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e96f3b44-86e7-4389-a90f-e0fa736cffd1-tls-certs\") pod \"test-model-639b07-kserve-b8485fd9-ph8tq\" (UID: \"e96f3b44-86e7-4389-a90f-e0fa736cffd1\") " pod="ai-tenant-e2e-inf-b-639b07/test-model-639b07-kserve-b8485fd9-ph8tq" Oct 30 21:19:58.026863 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:19:58.026828 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:19:58.054409 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:19:58.054377 5378 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b9gn7\" (UniqueName: \"kubernetes.io/projected/e96f3b44-86e7-4389-a90f-e0fa736cffd1-kube-api-access-b9gn7\") pod \"test-model-639b07-kserve-b8485fd9-ph8tq\" (UID: \"e96f3b44-86e7-4389-a90f-e0fa736cffd1\") " pod="ai-tenant-e2e-inf-b-639b07/test-model-639b07-kserve-b8485fd9-ph8tq" Oct 30 21:19:58.136668 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:19:58.136638 5378 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-639b07/test-model-639b07-kserve-b8485fd9-ph8tq" Oct 30 21:19:58.516058 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:19:58.516029 5378 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-639b07/test-model-639b07-kserve-b8485fd9-ph8tq"] Oct 30 21:19:58.517790 ip-10-0-138-149 kubenswrapper[5378]: W1030 21:19:58.517758 5378 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pode96f3b44_86e7_4389_a90f_e0fa736cffd1.slice/crio-1285a171d01d034023fe4b19e27a0a0bf03e34cb45b8757e7cec45de20574735 WatchSource:0}: Error finding container 1285a171d01d034023fe4b19e27a0a0bf03e34cb45b8757e7cec45de20574735: Status 404 returned error can't find the container with id 1285a171d01d034023fe4b19e27a0a0bf03e34cb45b8757e7cec45de20574735 Oct 30 21:19:58.572847 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:19:58.572815 5378 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-r9g59"] Oct 30 21:19:58.962984 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:19:58.962949 5378 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-639b07/test-model-639b07-kserve-b8485fd9-ph8tq" event={"ID":"e96f3b44-86e7-4389-a90f-e0fa736cffd1","Type":"ContainerStarted","Data":"625d0d1e89ac52181a02d4ed91d1426033a5a5abe58ab968521edf2f6cbc969e"} Oct 30 21:19:58.962984 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:19:58.962987 5378 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-639b07/test-model-639b07-kserve-b8485fd9-ph8tq" event={"ID":"e96f3b44-86e7-4389-a90f-e0fa736cffd1","Type":"ContainerStarted","Data":"1285a171d01d034023fe4b19e27a0a0bf03e34cb45b8757e7cec45de20574735"} Oct 30 21:19:59.081156 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:19:59.081091 5378 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-inf-b-639b07/test-model-639b07-kserve-b8485fd9-ph8tq" podStartSLOduration=2.081071561 podStartE2EDuration="2.081071561s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:19:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:19:59.079848902 +0000 UTC m=+2489.507962814" watchObservedRunningTime="2026-10-30 21:19:59.081071561 +0000 UTC m=+2489.509185481" Oct 30 21:19:59.137604 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:19:59.137544 5378 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-b-639b07/test-model-639b07-kserve-b8485fd9-ph8tq" Oct 30 21:20:02.858226 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:20:02.858188 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:20:07.621554 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:20:07.621507 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:20:08.137667 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:20:08.137638 5378 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-b-639b07/test-model-639b07-kserve-b8485fd9-ph8tq" Oct 30 21:20:08.151001 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:20:08.150967 5378 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-b-639b07/test-model-639b07-kserve-b8485fd9-ph8tq" Oct 30 21:20:09.030266 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:20:09.030238 5378 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-b-639b07/test-model-639b07-kserve-b8485fd9-ph8tq" Oct 30 21:20:12.388767 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:20:12.388729 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:20:17.259225 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:20:17.259193 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:20:18.839636 ip-10-0-138-149 kubenswrapper[5378]: E1030 21:20:18.839596 5378 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q6mx7" containerName="extractor" Oct 30 21:20:22.543294 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:20:22.543255 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:20:25.130606 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:20:25.130560 5378 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-r9g59"] Oct 30 21:20:27.304761 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:20:27.304723 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:20:28.819363 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:20:28.819331 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:20:28.944284 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:20:28.944250 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:20:32.065165 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:20:32.065076 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:20:36.844790 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:20:36.844756 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:20:41.639972 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:20:41.639892 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:20:46.586395 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:20:46.586353 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:20:48.837402 ip-10-0-138-149 kubenswrapper[5378]: E1030 21:20:48.837365 5378 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q6mx7" containerName="extractor" Oct 30 21:20:50.477742 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:20:50.477712 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:20:51.378654 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:20:51.378617 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:20:51.672547 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:20:51.672458 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:20:51.797487 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:20:51.797450 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:20:54.450809 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:20:54.450773 5378 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-r9g59"] Oct 30 21:20:56.177912 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:20:56.177870 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:21:00.961603 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:21:00.961554 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:21:05.755680 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:21:05.755640 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:21:10.591045 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:21:10.591008 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:21:15.371746 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:21:15.371713 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:21:18.837796 ip-10-0-138-149 kubenswrapper[5378]: E1030 21:21:18.837758 5378 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q6mx7" containerName="extractor" Oct 30 21:21:20.146975 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:21:20.146940 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:21:24.958865 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:21:24.958829 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:21:27.583123 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:21:27.583091 5378 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-r9g59"] Oct 30 21:21:29.758140 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:21:29.758109 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:21:31.750234 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:21:31.750205 5378 scope.go:122] "RemoveContainer" containerID="65d9e00da5785d63948239191de1cb71acf1464768dbdf42ee5d168029863d49" Oct 30 21:21:34.552272 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:21:34.552230 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:21:38.772196 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:21:38.772160 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:21:40.336615 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:21:40.336550 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:21:40.571239 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:21:40.571204 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:21:40.705254 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:21:40.705210 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:21:44.961484 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:21:44.961454 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:21:45.771819 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:21:45.771782 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:21:48.666673 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:21:48.666634 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:21:48.837894 ip-10-0-138-149 kubenswrapper[5378]: E1030 21:21:48.837855 5378 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q6mx7" containerName="extractor" Oct 30 21:21:48.838092 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:21:48.838060 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:21:50.792092 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:21:50.792059 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:21:50.908961 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:21:50.908926 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:21:51.105372 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:21:51.105336 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:21:51.315459 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:21:51.315429 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:21:55.312255 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:21:55.312223 5378 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-r9g59"] Oct 30 21:21:55.883972 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:21:55.883940 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:21:56.325699 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:21:56.325601 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:22:00.773315 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:22:00.773279 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:22:05.671257 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:22:05.671219 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:22:10.459425 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:22:10.459353 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:22:15.270323 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:22:15.270265 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:22:18.718478 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:22:18.718441 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:22:18.837056 ip-10-0-138-149 kubenswrapper[5378]: E1030 21:22:18.837019 5378 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q6mx7" containerName="extractor" Oct 30 21:22:18.838815 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:22:18.838790 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:22:20.062447 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:22:20.062409 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:22:22.663685 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:22:22.663649 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:22:24.865432 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:22:24.865404 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:22:29.512408 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:22:29.512375 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:22:29.672081 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:22:29.672050 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:22:29.937579 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:22:29.937529 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:22:34.573741 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:22:34.573702 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:22:35.592133 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:22:35.592102 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:22:39.466177 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:22:39.466139 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:22:44.266641 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:22:44.266558 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:22:45.990415 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:22:45.990383 5378 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-r9g59"] Oct 30 21:22:48.836484 ip-10-0-138-149 kubenswrapper[5378]: E1030 21:22:48.836447 5378 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q6mx7" containerName="extractor" Oct 30 21:22:49.282716 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:22:49.282685 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:22:51.158992 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:22:51.158957 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:22:53.860528 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:22:53.860494 5378 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-d1b889bd-openshift-default-566bd56c85-vvvjg"] Oct 30 21:22:53.861052 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:22:53.861037 5378 cpu_manager.go:414] "containerMap: removing container" podUID="9b76fbb2-f5a8-4fea-bdfd-30f40bee7008" containerName="istio-proxy" Oct 30 21:22:53.861052 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:22:53.861052 5378 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9b76fbb2-f5a8-4fea-bdfd-30f40bee7008" containerName="istio-proxy" Oct 30 21:22:53.861159 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:22:53.861150 5378 memory_manager.go:365] "RemoveStaleState removing state" podUID="261c5b5a-84f8-4527-9b51-567d6dbf17ca" containerName="istio-proxy" Oct 30 21:22:53.864639 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:22:53.864618 5378 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-d1b889bd-openshift-default-566bd56c85-vvvjg" Oct 30 21:22:53.917297 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:22:53.917271 5378 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-d1b889bd-openshift-default-dockercfg-h9jb2\"" Oct 30 21:22:53.998874 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:22:53.998841 5378 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8fdfc4fa-d488-4267-bfb0-113448a60fc2-istio-podinfo\") pod \"e2e-mt-d1b889bd-openshift-default-566bd56c85-vvvjg\" (UID: \"8fdfc4fa-d488-4267-bfb0-113448a60fc2\") " pod="openshift-ingress/e2e-mt-d1b889bd-openshift-default-566bd56c85-vvvjg" Oct 30 21:22:53.999035 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:22:53.998879 5378 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8fdfc4fa-d488-4267-bfb0-113448a60fc2-credential-socket\") pod \"e2e-mt-d1b889bd-openshift-default-566bd56c85-vvvjg\" (UID: \"8fdfc4fa-d488-4267-bfb0-113448a60fc2\") " pod="openshift-ingress/e2e-mt-d1b889bd-openshift-default-566bd56c85-vvvjg" Oct 30 21:22:53.999035 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:22:53.998977 5378 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8fdfc4fa-d488-4267-bfb0-113448a60fc2-istiod-ca-cert\") pod \"e2e-mt-d1b889bd-openshift-default-566bd56c85-vvvjg\" (UID: \"8fdfc4fa-d488-4267-bfb0-113448a60fc2\") " pod="openshift-ingress/e2e-mt-d1b889bd-openshift-default-566bd56c85-vvvjg" Oct 30 21:22:53.999035 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:22:53.999007 5378 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8fdfc4fa-d488-4267-bfb0-113448a60fc2-istio-data\") pod \"e2e-mt-d1b889bd-openshift-default-566bd56c85-vvvjg\" (UID: \"8fdfc4fa-d488-4267-bfb0-113448a60fc2\") " pod="openshift-ingress/e2e-mt-d1b889bd-openshift-default-566bd56c85-vvvjg" Oct 30 21:22:53.999225 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:22:53.999112 5378 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8fdfc4fa-d488-4267-bfb0-113448a60fc2-workload-certs\") pod \"e2e-mt-d1b889bd-openshift-default-566bd56c85-vvvjg\" (UID: \"8fdfc4fa-d488-4267-bfb0-113448a60fc2\") " pod="openshift-ingress/e2e-mt-d1b889bd-openshift-default-566bd56c85-vvvjg" Oct 30 21:22:53.999225 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:22:53.999160 5378 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9zsg9\" (UniqueName: \"kubernetes.io/projected/8fdfc4fa-d488-4267-bfb0-113448a60fc2-kube-api-access-9zsg9\") pod \"e2e-mt-d1b889bd-openshift-default-566bd56c85-vvvjg\" (UID: \"8fdfc4fa-d488-4267-bfb0-113448a60fc2\") " pod="openshift-ingress/e2e-mt-d1b889bd-openshift-default-566bd56c85-vvvjg" Oct 30 21:22:53.999225 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:22:53.999201 5378 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8fdfc4fa-d488-4267-bfb0-113448a60fc2-istio-token\") pod \"e2e-mt-d1b889bd-openshift-default-566bd56c85-vvvjg\" (UID: \"8fdfc4fa-d488-4267-bfb0-113448a60fc2\") " pod="openshift-ingress/e2e-mt-d1b889bd-openshift-default-566bd56c85-vvvjg" Oct 30 21:22:53.999349 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:22:53.999269 5378 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8fdfc4fa-d488-4267-bfb0-113448a60fc2-istio-envoy\") pod \"e2e-mt-d1b889bd-openshift-default-566bd56c85-vvvjg\" (UID: \"8fdfc4fa-d488-4267-bfb0-113448a60fc2\") " pod="openshift-ingress/e2e-mt-d1b889bd-openshift-default-566bd56c85-vvvjg" Oct 30 21:22:53.999349 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:22:53.999301 5378 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8fdfc4fa-d488-4267-bfb0-113448a60fc2-workload-socket\") pod \"e2e-mt-d1b889bd-openshift-default-566bd56c85-vvvjg\" (UID: \"8fdfc4fa-d488-4267-bfb0-113448a60fc2\") " pod="openshift-ingress/e2e-mt-d1b889bd-openshift-default-566bd56c85-vvvjg" Oct 30 21:22:54.065626 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:22:54.065600 5378 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-d1b889bd-openshift-default-566bd56c85-vvvjg"] Oct 30 21:22:54.100436 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:22:54.100407 5378 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8fdfc4fa-d488-4267-bfb0-113448a60fc2-istio-podinfo\") pod \"e2e-mt-d1b889bd-openshift-default-566bd56c85-vvvjg\" (UID: \"8fdfc4fa-d488-4267-bfb0-113448a60fc2\") " pod="openshift-ingress/e2e-mt-d1b889bd-openshift-default-566bd56c85-vvvjg" Oct 30 21:22:54.100648 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:22:54.100444 5378 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8fdfc4fa-d488-4267-bfb0-113448a60fc2-credential-socket\") pod \"e2e-mt-d1b889bd-openshift-default-566bd56c85-vvvjg\" (UID: \"8fdfc4fa-d488-4267-bfb0-113448a60fc2\") " pod="openshift-ingress/e2e-mt-d1b889bd-openshift-default-566bd56c85-vvvjg" Oct 30 21:22:54.100648 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:22:54.100489 5378 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8fdfc4fa-d488-4267-bfb0-113448a60fc2-istiod-ca-cert\") pod \"e2e-mt-d1b889bd-openshift-default-566bd56c85-vvvjg\" (UID: \"8fdfc4fa-d488-4267-bfb0-113448a60fc2\") " pod="openshift-ingress/e2e-mt-d1b889bd-openshift-default-566bd56c85-vvvjg" Oct 30 21:22:54.100648 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:22:54.100517 5378 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8fdfc4fa-d488-4267-bfb0-113448a60fc2-istio-data\") pod \"e2e-mt-d1b889bd-openshift-default-566bd56c85-vvvjg\" (UID: \"8fdfc4fa-d488-4267-bfb0-113448a60fc2\") " pod="openshift-ingress/e2e-mt-d1b889bd-openshift-default-566bd56c85-vvvjg" Oct 30 21:22:54.100648 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:22:54.100590 5378 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8fdfc4fa-d488-4267-bfb0-113448a60fc2-workload-certs\") pod \"e2e-mt-d1b889bd-openshift-default-566bd56c85-vvvjg\" (UID: \"8fdfc4fa-d488-4267-bfb0-113448a60fc2\") " pod="openshift-ingress/e2e-mt-d1b889bd-openshift-default-566bd56c85-vvvjg" Oct 30 21:22:54.100648 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:22:54.100621 5378 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9zsg9\" (UniqueName: \"kubernetes.io/projected/8fdfc4fa-d488-4267-bfb0-113448a60fc2-kube-api-access-9zsg9\") pod \"e2e-mt-d1b889bd-openshift-default-566bd56c85-vvvjg\" (UID: \"8fdfc4fa-d488-4267-bfb0-113448a60fc2\") " pod="openshift-ingress/e2e-mt-d1b889bd-openshift-default-566bd56c85-vvvjg" Oct 30 21:22:54.100648 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:22:54.100647 5378 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8fdfc4fa-d488-4267-bfb0-113448a60fc2-istio-token\") pod \"e2e-mt-d1b889bd-openshift-default-566bd56c85-vvvjg\" (UID: \"8fdfc4fa-d488-4267-bfb0-113448a60fc2\") " pod="openshift-ingress/e2e-mt-d1b889bd-openshift-default-566bd56c85-vvvjg" Oct 30 21:22:54.101010 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:22:54.100674 5378 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8fdfc4fa-d488-4267-bfb0-113448a60fc2-istio-envoy\") pod \"e2e-mt-d1b889bd-openshift-default-566bd56c85-vvvjg\" (UID: \"8fdfc4fa-d488-4267-bfb0-113448a60fc2\") " pod="openshift-ingress/e2e-mt-d1b889bd-openshift-default-566bd56c85-vvvjg" Oct 30 21:22:54.101010 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:22:54.100709 5378 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8fdfc4fa-d488-4267-bfb0-113448a60fc2-workload-socket\") pod \"e2e-mt-d1b889bd-openshift-default-566bd56c85-vvvjg\" (UID: \"8fdfc4fa-d488-4267-bfb0-113448a60fc2\") " pod="openshift-ingress/e2e-mt-d1b889bd-openshift-default-566bd56c85-vvvjg" Oct 30 21:22:54.101010 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:22:54.100890 5378 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8fdfc4fa-d488-4267-bfb0-113448a60fc2-credential-socket\") pod \"e2e-mt-d1b889bd-openshift-default-566bd56c85-vvvjg\" (UID: \"8fdfc4fa-d488-4267-bfb0-113448a60fc2\") " pod="openshift-ingress/e2e-mt-d1b889bd-openshift-default-566bd56c85-vvvjg" Oct 30 21:22:54.101174 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:22:54.101110 5378 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8fdfc4fa-d488-4267-bfb0-113448a60fc2-workload-socket\") pod \"e2e-mt-d1b889bd-openshift-default-566bd56c85-vvvjg\" (UID: \"8fdfc4fa-d488-4267-bfb0-113448a60fc2\") " pod="openshift-ingress/e2e-mt-d1b889bd-openshift-default-566bd56c85-vvvjg" Oct 30 21:22:54.101594 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:22:54.101554 5378 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8fdfc4fa-d488-4267-bfb0-113448a60fc2-workload-certs\") pod \"e2e-mt-d1b889bd-openshift-default-566bd56c85-vvvjg\" (UID: \"8fdfc4fa-d488-4267-bfb0-113448a60fc2\") " pod="openshift-ingress/e2e-mt-d1b889bd-openshift-default-566bd56c85-vvvjg" Oct 30 21:22:54.101705 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:22:54.101592 5378 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8fdfc4fa-d488-4267-bfb0-113448a60fc2-istio-data\") pod \"e2e-mt-d1b889bd-openshift-default-566bd56c85-vvvjg\" (UID: \"8fdfc4fa-d488-4267-bfb0-113448a60fc2\") " pod="openshift-ingress/e2e-mt-d1b889bd-openshift-default-566bd56c85-vvvjg" Oct 30 21:22:54.102474 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:22:54.102448 5378 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8fdfc4fa-d488-4267-bfb0-113448a60fc2-istiod-ca-cert\") pod \"e2e-mt-d1b889bd-openshift-default-566bd56c85-vvvjg\" (UID: \"8fdfc4fa-d488-4267-bfb0-113448a60fc2\") " pod="openshift-ingress/e2e-mt-d1b889bd-openshift-default-566bd56c85-vvvjg" Oct 30 21:22:54.103153 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:22:54.103130 5378 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8fdfc4fa-d488-4267-bfb0-113448a60fc2-istio-envoy\") pod \"e2e-mt-d1b889bd-openshift-default-566bd56c85-vvvjg\" (UID: \"8fdfc4fa-d488-4267-bfb0-113448a60fc2\") " pod="openshift-ingress/e2e-mt-d1b889bd-openshift-default-566bd56c85-vvvjg" Oct 30 21:22:54.103252 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:22:54.103230 5378 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8fdfc4fa-d488-4267-bfb0-113448a60fc2-istio-podinfo\") pod \"e2e-mt-d1b889bd-openshift-default-566bd56c85-vvvjg\" (UID: \"8fdfc4fa-d488-4267-bfb0-113448a60fc2\") " pod="openshift-ingress/e2e-mt-d1b889bd-openshift-default-566bd56c85-vvvjg" Oct 30 21:22:54.147266 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:22:54.147196 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:22:54.158188 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:22:54.158164 5378 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8fdfc4fa-d488-4267-bfb0-113448a60fc2-istio-token\") pod \"e2e-mt-d1b889bd-openshift-default-566bd56c85-vvvjg\" (UID: \"8fdfc4fa-d488-4267-bfb0-113448a60fc2\") " pod="openshift-ingress/e2e-mt-d1b889bd-openshift-default-566bd56c85-vvvjg" Oct 30 21:22:54.158432 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:22:54.158414 5378 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9zsg9\" (UniqueName: \"kubernetes.io/projected/8fdfc4fa-d488-4267-bfb0-113448a60fc2-kube-api-access-9zsg9\") pod \"e2e-mt-d1b889bd-openshift-default-566bd56c85-vvvjg\" (UID: \"8fdfc4fa-d488-4267-bfb0-113448a60fc2\") " pod="openshift-ingress/e2e-mt-d1b889bd-openshift-default-566bd56c85-vvvjg" Oct 30 21:22:54.178361 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:22:54.178339 5378 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-d1b889bd-openshift-default-566bd56c85-vvvjg" Oct 30 21:22:54.656259 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:22:54.656234 5378 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-d1b889bd-openshift-default-566bd56c85-vvvjg"] Oct 30 21:22:54.657983 ip-10-0-138-149 kubenswrapper[5378]: W1030 21:22:54.657946 5378 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8fdfc4fa_d488_4267_bfb0_113448a60fc2.slice/crio-76671a323c11e92a6e51dc9c1d16d5c8dc9efb5b4fef8f9b07294e10489b5420 WatchSource:0}: Error finding container 76671a323c11e92a6e51dc9c1d16d5c8dc9efb5b4fef8f9b07294e10489b5420: Status 404 returned error can't find the container with id 76671a323c11e92a6e51dc9c1d16d5c8dc9efb5b4fef8f9b07294e10489b5420 Oct 30 21:22:54.660336 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:22:54.660306 5378 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 21:22:54.660409 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:22:54.660368 5378 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 21:22:54.660409 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:22:54.660398 5378 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 21:22:55.009899 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:22:55.009809 5378 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-d1b889bd-openshift-default-566bd56c85-vvvjg" event={"ID":"8fdfc4fa-d488-4267-bfb0-113448a60fc2","Type":"ContainerStarted","Data":"113d795cb55b47f8a9085116bac2ee5acc5574596f0bb731e74e78f2c6e73b21"} Oct 30 21:22:55.009899 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:22:55.009875 5378 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-d1b889bd-openshift-default-566bd56c85-vvvjg" event={"ID":"8fdfc4fa-d488-4267-bfb0-113448a60fc2","Type":"ContainerStarted","Data":"76671a323c11e92a6e51dc9c1d16d5c8dc9efb5b4fef8f9b07294e10489b5420"} Oct 30 21:22:55.130055 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:22:55.130007 5378 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-d1b889bd-openshift-default-566bd56c85-vvvjg" podStartSLOduration=2.129991293 podStartE2EDuration="2.129991293s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:22:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:22:55.129155209 +0000 UTC m=+2665.557269129" watchObservedRunningTime="2026-10-30 21:22:55.129991293 +0000 UTC m=+2665.558105212" Oct 30 21:22:55.178906 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:22:55.178878 5378 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-d1b889bd-openshift-default-566bd56c85-vvvjg" Oct 30 21:22:55.180804 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:22:55.180778 5378 patch_prober.go:29] interesting pod/e2e-mt-d1b889bd-openshift-default-566bd56c85-vvvjg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.164:15021/healthz/ready\": dial tcp 10.133.0.164:15021: connect: connection refused" start-of-body= Oct 30 21:22:55.180983 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:22:55.180824 5378 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-d1b889bd-openshift-default-566bd56c85-vvvjg" podUID="8fdfc4fa-d488-4267-bfb0-113448a60fc2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.164:15021/healthz/ready\": dial tcp 10.133.0.164:15021: connect: connection refused" Oct 30 21:22:56.178971 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:22:56.178938 5378 patch_prober.go:29] interesting pod/e2e-mt-d1b889bd-openshift-default-566bd56c85-vvvjg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.164:15021/healthz/ready\": dial tcp 10.133.0.164:15021: connect: connection refused" start-of-body= Oct 30 21:22:56.179467 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:22:56.178990 5378 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-d1b889bd-openshift-default-566bd56c85-vvvjg" podUID="8fdfc4fa-d488-4267-bfb0-113448a60fc2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.164:15021/healthz/ready\": dial tcp 10.133.0.164:15021: connect: connection refused" Oct 30 21:22:57.179625 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:22:57.179592 5378 patch_prober.go:29] interesting pod/e2e-mt-d1b889bd-openshift-default-566bd56c85-vvvjg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.164:15021/healthz/ready\": dial tcp 10.133.0.164:15021: connect: connection refused" start-of-body= Oct 30 21:22:57.180024 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:22:57.179646 5378 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-d1b889bd-openshift-default-566bd56c85-vvvjg" podUID="8fdfc4fa-d488-4267-bfb0-113448a60fc2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.164:15021/healthz/ready\": dial tcp 10.133.0.164:15021: connect: connection refused" Oct 30 21:22:58.178796 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:22:58.178754 5378 patch_prober.go:29] interesting pod/e2e-mt-d1b889bd-openshift-default-566bd56c85-vvvjg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.164:15021/healthz/ready\": dial tcp 10.133.0.164:15021: connect: connection refused" start-of-body= Oct 30 21:22:58.179010 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:22:58.178809 5378 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-d1b889bd-openshift-default-566bd56c85-vvvjg" podUID="8fdfc4fa-d488-4267-bfb0-113448a60fc2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.164:15021/healthz/ready\": dial tcp 10.133.0.164:15021: connect: connection refused" Oct 30 21:22:59.098088 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:22:59.098049 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:22:59.190590 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:22:59.185229 5378 patch_prober.go:29] interesting pod/e2e-mt-d1b889bd-openshift-default-566bd56c85-vvvjg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.164:15021/healthz/ready\": dial tcp 10.133.0.164:15021: connect: connection refused" start-of-body= Oct 30 21:22:59.190590 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:22:59.185291 5378 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-d1b889bd-openshift-default-566bd56c85-vvvjg" podUID="8fdfc4fa-d488-4267-bfb0-113448a60fc2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.164:15021/healthz/ready\": dial tcp 10.133.0.164:15021: connect: connection refused" Oct 30 21:23:00.179367 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:23:00.179331 5378 patch_prober.go:29] interesting pod/e2e-mt-d1b889bd-openshift-default-566bd56c85-vvvjg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.164:15021/healthz/ready\": dial tcp 10.133.0.164:15021: connect: connection refused" start-of-body= Oct 30 21:23:00.179780 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:23:00.179381 5378 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-d1b889bd-openshift-default-566bd56c85-vvvjg" podUID="8fdfc4fa-d488-4267-bfb0-113448a60fc2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.164:15021/healthz/ready\": dial tcp 10.133.0.164:15021: connect: connection refused" Oct 30 21:23:00.290077 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:23:00.290046 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:23:01.254386 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:23:01.254334 5378 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-d1b889bd-openshift-default-566bd56c85-vvvjg" Oct 30 21:23:01.255181 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:23:01.255158 5378 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-d1b889bd-openshift-default-566bd56c85-vvvjg" Oct 30 21:23:01.256355 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:23:01.256331 5378 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-d1b889bd-openshift-default-566bd56c85-vvvjg" Oct 30 21:23:04.101710 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:23:04.101679 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:23:08.972733 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:23:08.972703 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:23:08.985527 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:23:08.985500 5378 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-d1b889bd-openshift-default-566bd56c85-dvzs4"] Oct 30 21:23:08.990119 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:23:08.990101 5378 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-d1b889bd-openshift-default-566bd56c85-dvzs4" Oct 30 21:23:09.088924 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:23:09.088891 5378 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-d1b889bd-openshift-default-566bd56c85-dvzs4"] Oct 30 21:23:09.113771 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:23:09.113732 5378 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d848cad5-c34e-4841-a2a0-e3aab4a69029-workload-socket\") pod \"e2e-mt-d1b889bd-openshift-default-566bd56c85-dvzs4\" (UID: \"d848cad5-c34e-4841-a2a0-e3aab4a69029\") " pod="openshift-ingress/e2e-mt-d1b889bd-openshift-default-566bd56c85-dvzs4" Oct 30 21:23:09.113961 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:23:09.113782 5378 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d848cad5-c34e-4841-a2a0-e3aab4a69029-istiod-ca-cert\") pod \"e2e-mt-d1b889bd-openshift-default-566bd56c85-dvzs4\" (UID: \"d848cad5-c34e-4841-a2a0-e3aab4a69029\") " pod="openshift-ingress/e2e-mt-d1b889bd-openshift-default-566bd56c85-dvzs4" Oct 30 21:23:09.113961 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:23:09.113809 5378 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7kxxn\" (UniqueName: \"kubernetes.io/projected/d848cad5-c34e-4841-a2a0-e3aab4a69029-kube-api-access-7kxxn\") pod \"e2e-mt-d1b889bd-openshift-default-566bd56c85-dvzs4\" (UID: \"d848cad5-c34e-4841-a2a0-e3aab4a69029\") " pod="openshift-ingress/e2e-mt-d1b889bd-openshift-default-566bd56c85-dvzs4" Oct 30 21:23:09.113961 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:23:09.113921 5378 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d848cad5-c34e-4841-a2a0-e3aab4a69029-istio-data\") pod \"e2e-mt-d1b889bd-openshift-default-566bd56c85-dvzs4\" (UID: \"d848cad5-c34e-4841-a2a0-e3aab4a69029\") " pod="openshift-ingress/e2e-mt-d1b889bd-openshift-default-566bd56c85-dvzs4" Oct 30 21:23:09.113961 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:23:09.113949 5378 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d848cad5-c34e-4841-a2a0-e3aab4a69029-istio-podinfo\") pod \"e2e-mt-d1b889bd-openshift-default-566bd56c85-dvzs4\" (UID: \"d848cad5-c34e-4841-a2a0-e3aab4a69029\") " pod="openshift-ingress/e2e-mt-d1b889bd-openshift-default-566bd56c85-dvzs4" Oct 30 21:23:09.114222 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:23:09.113978 5378 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d848cad5-c34e-4841-a2a0-e3aab4a69029-workload-certs\") pod \"e2e-mt-d1b889bd-openshift-default-566bd56c85-dvzs4\" (UID: \"d848cad5-c34e-4841-a2a0-e3aab4a69029\") " pod="openshift-ingress/e2e-mt-d1b889bd-openshift-default-566bd56c85-dvzs4" Oct 30 21:23:09.114222 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:23:09.114132 5378 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d848cad5-c34e-4841-a2a0-e3aab4a69029-credential-socket\") pod \"e2e-mt-d1b889bd-openshift-default-566bd56c85-dvzs4\" (UID: \"d848cad5-c34e-4841-a2a0-e3aab4a69029\") " pod="openshift-ingress/e2e-mt-d1b889bd-openshift-default-566bd56c85-dvzs4" Oct 30 21:23:09.114222 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:23:09.114167 5378 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d848cad5-c34e-4841-a2a0-e3aab4a69029-istio-envoy\") pod \"e2e-mt-d1b889bd-openshift-default-566bd56c85-dvzs4\" (UID: \"d848cad5-c34e-4841-a2a0-e3aab4a69029\") " pod="openshift-ingress/e2e-mt-d1b889bd-openshift-default-566bd56c85-dvzs4" Oct 30 21:23:09.114399 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:23:09.114256 5378 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d848cad5-c34e-4841-a2a0-e3aab4a69029-istio-token\") pod \"e2e-mt-d1b889bd-openshift-default-566bd56c85-dvzs4\" (UID: \"d848cad5-c34e-4841-a2a0-e3aab4a69029\") " pod="openshift-ingress/e2e-mt-d1b889bd-openshift-default-566bd56c85-dvzs4" Oct 30 21:23:09.215746 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:23:09.215715 5378 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d848cad5-c34e-4841-a2a0-e3aab4a69029-istio-token\") pod \"e2e-mt-d1b889bd-openshift-default-566bd56c85-dvzs4\" (UID: \"d848cad5-c34e-4841-a2a0-e3aab4a69029\") " pod="openshift-ingress/e2e-mt-d1b889bd-openshift-default-566bd56c85-dvzs4" Oct 30 21:23:09.215947 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:23:09.215794 5378 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d848cad5-c34e-4841-a2a0-e3aab4a69029-workload-socket\") pod \"e2e-mt-d1b889bd-openshift-default-566bd56c85-dvzs4\" (UID: \"d848cad5-c34e-4841-a2a0-e3aab4a69029\") " pod="openshift-ingress/e2e-mt-d1b889bd-openshift-default-566bd56c85-dvzs4" Oct 30 21:23:09.215947 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:23:09.215833 5378 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d848cad5-c34e-4841-a2a0-e3aab4a69029-istiod-ca-cert\") pod \"e2e-mt-d1b889bd-openshift-default-566bd56c85-dvzs4\" (UID: \"d848cad5-c34e-4841-a2a0-e3aab4a69029\") " pod="openshift-ingress/e2e-mt-d1b889bd-openshift-default-566bd56c85-dvzs4" Oct 30 21:23:09.215947 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:23:09.215861 5378 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7kxxn\" (UniqueName: \"kubernetes.io/projected/d848cad5-c34e-4841-a2a0-e3aab4a69029-kube-api-access-7kxxn\") pod \"e2e-mt-d1b889bd-openshift-default-566bd56c85-dvzs4\" (UID: \"d848cad5-c34e-4841-a2a0-e3aab4a69029\") " pod="openshift-ingress/e2e-mt-d1b889bd-openshift-default-566bd56c85-dvzs4" Oct 30 21:23:09.215947 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:23:09.215894 5378 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d848cad5-c34e-4841-a2a0-e3aab4a69029-istio-data\") pod \"e2e-mt-d1b889bd-openshift-default-566bd56c85-dvzs4\" (UID: \"d848cad5-c34e-4841-a2a0-e3aab4a69029\") " pod="openshift-ingress/e2e-mt-d1b889bd-openshift-default-566bd56c85-dvzs4" Oct 30 21:23:09.215947 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:23:09.215943 5378 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d848cad5-c34e-4841-a2a0-e3aab4a69029-istio-podinfo\") pod \"e2e-mt-d1b889bd-openshift-default-566bd56c85-dvzs4\" (UID: \"d848cad5-c34e-4841-a2a0-e3aab4a69029\") " pod="openshift-ingress/e2e-mt-d1b889bd-openshift-default-566bd56c85-dvzs4" Oct 30 21:23:09.216238 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:23:09.215973 5378 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d848cad5-c34e-4841-a2a0-e3aab4a69029-workload-certs\") pod \"e2e-mt-d1b889bd-openshift-default-566bd56c85-dvzs4\" (UID: \"d848cad5-c34e-4841-a2a0-e3aab4a69029\") " pod="openshift-ingress/e2e-mt-d1b889bd-openshift-default-566bd56c85-dvzs4" Oct 30 21:23:09.216238 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:23:09.216093 5378 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d848cad5-c34e-4841-a2a0-e3aab4a69029-credential-socket\") pod \"e2e-mt-d1b889bd-openshift-default-566bd56c85-dvzs4\" (UID: \"d848cad5-c34e-4841-a2a0-e3aab4a69029\") " pod="openshift-ingress/e2e-mt-d1b889bd-openshift-default-566bd56c85-dvzs4" Oct 30 21:23:09.216238 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:23:09.216130 5378 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d848cad5-c34e-4841-a2a0-e3aab4a69029-istio-envoy\") pod \"e2e-mt-d1b889bd-openshift-default-566bd56c85-dvzs4\" (UID: \"d848cad5-c34e-4841-a2a0-e3aab4a69029\") " pod="openshift-ingress/e2e-mt-d1b889bd-openshift-default-566bd56c85-dvzs4" Oct 30 21:23:09.216404 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:23:09.216317 5378 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d848cad5-c34e-4841-a2a0-e3aab4a69029-istio-data\") pod \"e2e-mt-d1b889bd-openshift-default-566bd56c85-dvzs4\" (UID: \"d848cad5-c34e-4841-a2a0-e3aab4a69029\") " pod="openshift-ingress/e2e-mt-d1b889bd-openshift-default-566bd56c85-dvzs4" Oct 30 21:23:09.216404 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:23:09.216363 5378 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d848cad5-c34e-4841-a2a0-e3aab4a69029-workload-socket\") pod \"e2e-mt-d1b889bd-openshift-default-566bd56c85-dvzs4\" (UID: \"d848cad5-c34e-4841-a2a0-e3aab4a69029\") " pod="openshift-ingress/e2e-mt-d1b889bd-openshift-default-566bd56c85-dvzs4" Oct 30 21:23:09.216404 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:23:09.216389 5378 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d848cad5-c34e-4841-a2a0-e3aab4a69029-workload-certs\") pod \"e2e-mt-d1b889bd-openshift-default-566bd56c85-dvzs4\" (UID: \"d848cad5-c34e-4841-a2a0-e3aab4a69029\") " pod="openshift-ingress/e2e-mt-d1b889bd-openshift-default-566bd56c85-dvzs4" Oct 30 21:23:09.216632 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:23:09.216607 5378 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d848cad5-c34e-4841-a2a0-e3aab4a69029-credential-socket\") pod \"e2e-mt-d1b889bd-openshift-default-566bd56c85-dvzs4\" (UID: \"d848cad5-c34e-4841-a2a0-e3aab4a69029\") " pod="openshift-ingress/e2e-mt-d1b889bd-openshift-default-566bd56c85-dvzs4" Oct 30 21:23:09.217178 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:23:09.217153 5378 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d848cad5-c34e-4841-a2a0-e3aab4a69029-istiod-ca-cert\") pod \"e2e-mt-d1b889bd-openshift-default-566bd56c85-dvzs4\" (UID: \"d848cad5-c34e-4841-a2a0-e3aab4a69029\") " pod="openshift-ingress/e2e-mt-d1b889bd-openshift-default-566bd56c85-dvzs4" Oct 30 21:23:09.218507 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:23:09.218490 5378 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d848cad5-c34e-4841-a2a0-e3aab4a69029-istio-envoy\") pod \"e2e-mt-d1b889bd-openshift-default-566bd56c85-dvzs4\" (UID: \"d848cad5-c34e-4841-a2a0-e3aab4a69029\") " pod="openshift-ingress/e2e-mt-d1b889bd-openshift-default-566bd56c85-dvzs4" Oct 30 21:23:09.218787 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:23:09.218765 5378 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d848cad5-c34e-4841-a2a0-e3aab4a69029-istio-podinfo\") pod \"e2e-mt-d1b889bd-openshift-default-566bd56c85-dvzs4\" (UID: \"d848cad5-c34e-4841-a2a0-e3aab4a69029\") " pod="openshift-ingress/e2e-mt-d1b889bd-openshift-default-566bd56c85-dvzs4" Oct 30 21:23:09.272862 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:23:09.272796 5378 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d848cad5-c34e-4841-a2a0-e3aab4a69029-istio-token\") pod \"e2e-mt-d1b889bd-openshift-default-566bd56c85-dvzs4\" (UID: \"d848cad5-c34e-4841-a2a0-e3aab4a69029\") " pod="openshift-ingress/e2e-mt-d1b889bd-openshift-default-566bd56c85-dvzs4" Oct 30 21:23:09.273130 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:23:09.273108 5378 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7kxxn\" (UniqueName: \"kubernetes.io/projected/d848cad5-c34e-4841-a2a0-e3aab4a69029-kube-api-access-7kxxn\") pod \"e2e-mt-d1b889bd-openshift-default-566bd56c85-dvzs4\" (UID: \"d848cad5-c34e-4841-a2a0-e3aab4a69029\") " pod="openshift-ingress/e2e-mt-d1b889bd-openshift-default-566bd56c85-dvzs4" Oct 30 21:23:09.304098 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:23:09.304064 5378 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-d1b889bd-openshift-default-566bd56c85-dvzs4" Oct 30 21:23:09.755258 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:23:09.755232 5378 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-d1b889bd-openshift-default-566bd56c85-dvzs4"] Oct 30 21:23:09.756038 ip-10-0-138-149 kubenswrapper[5378]: W1030 21:23:09.756005 5378 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd848cad5_c34e_4841_a2a0_e3aab4a69029.slice/crio-80c59336f265a4cd0b263f497f12d3c9922633be768e72ff21d90f0c61fea07a WatchSource:0}: Error finding container 80c59336f265a4cd0b263f497f12d3c9922633be768e72ff21d90f0c61fea07a: Status 404 returned error can't find the container with id 80c59336f265a4cd0b263f497f12d3c9922633be768e72ff21d90f0c61fea07a Oct 30 21:23:09.759004 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:23:09.758969 5378 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 21:23:09.759088 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:23:09.759047 5378 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 21:23:09.759133 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:23:09.759086 5378 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 21:23:10.102709 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:23:10.102667 5378 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-d1b889bd-openshift-default-566bd56c85-dvzs4" event={"ID":"d848cad5-c34e-4841-a2a0-e3aab4a69029","Type":"ContainerStarted","Data":"3b9304456d83f1af5b7b823de73c14a63e17a08a5504aabb0d954f67c4de92ba"} Oct 30 21:23:10.103204 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:23:10.102741 5378 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-d1b889bd-openshift-default-566bd56c85-dvzs4" event={"ID":"d848cad5-c34e-4841-a2a0-e3aab4a69029","Type":"ContainerStarted","Data":"80c59336f265a4cd0b263f497f12d3c9922633be768e72ff21d90f0c61fea07a"} Oct 30 21:23:10.225202 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:23:10.224601 5378 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-d1b889bd-openshift-default-566bd56c85-dvzs4" podStartSLOduration=2.2245482069999998 podStartE2EDuration="2.224548207s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:23:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:23:10.223693129 +0000 UTC m=+2680.651807048" watchObservedRunningTime="2026-10-30 21:23:10.224548207 +0000 UTC m=+2680.652662126" Oct 30 21:23:10.305067 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:23:10.305033 5378 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-d1b889bd-openshift-default-566bd56c85-dvzs4" Oct 30 21:23:10.306498 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:23:10.306469 5378 patch_prober.go:29] interesting pod/e2e-mt-d1b889bd-openshift-default-566bd56c85-dvzs4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.165:15021/healthz/ready\": dial tcp 10.133.0.165:15021: connect: connection refused" start-of-body= Oct 30 21:23:10.306675 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:23:10.306527 5378 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-d1b889bd-openshift-default-566bd56c85-dvzs4" podUID="d848cad5-c34e-4841-a2a0-e3aab4a69029" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.165:15021/healthz/ready\": dial tcp 10.133.0.165:15021: connect: connection refused" Oct 30 21:23:10.740128 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:23:10.740098 5378 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-r9g59"] Oct 30 21:23:11.316590 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:23:11.314318 5378 patch_prober.go:29] interesting pod/e2e-mt-d1b889bd-openshift-default-566bd56c85-dvzs4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.165:15021/healthz/ready\": dial tcp 10.133.0.165:15021: connect: connection refused" start-of-body= Oct 30 21:23:11.316590 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:23:11.314391 5378 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-d1b889bd-openshift-default-566bd56c85-dvzs4" podUID="d848cad5-c34e-4841-a2a0-e3aab4a69029" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.165:15021/healthz/ready\": dial tcp 10.133.0.165:15021: connect: connection refused" Oct 30 21:23:12.304445 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:23:12.304413 5378 patch_prober.go:29] interesting pod/e2e-mt-d1b889bd-openshift-default-566bd56c85-dvzs4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.165:15021/healthz/ready\": dial tcp 10.133.0.165:15021: connect: connection refused" start-of-body= Oct 30 21:23:12.304653 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:23:12.304464 5378 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-d1b889bd-openshift-default-566bd56c85-dvzs4" podUID="d848cad5-c34e-4841-a2a0-e3aab4a69029" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.165:15021/healthz/ready\": dial tcp 10.133.0.165:15021: connect: connection refused" Oct 30 21:23:12.780272 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:23:12.780235 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:23:12.917388 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:23:12.917354 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:23:13.304927 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:23:13.304888 5378 patch_prober.go:29] interesting pod/e2e-mt-d1b889bd-openshift-default-566bd56c85-dvzs4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.165:15021/healthz/ready\": dial tcp 10.133.0.165:15021: connect: connection refused" start-of-body= Oct 30 21:23:13.305115 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:23:13.304945 5378 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-d1b889bd-openshift-default-566bd56c85-dvzs4" podUID="d848cad5-c34e-4841-a2a0-e3aab4a69029" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.165:15021/healthz/ready\": dial tcp 10.133.0.165:15021: connect: connection refused" Oct 30 21:23:13.899511 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:23:13.899478 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:23:14.307295 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:23:14.307202 5378 patch_prober.go:29] interesting pod/e2e-mt-d1b889bd-openshift-default-566bd56c85-dvzs4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.165:15021/healthz/ready\": dial tcp 10.133.0.165:15021: connect: connection refused" start-of-body= Oct 30 21:23:14.307720 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:23:14.307677 5378 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-d1b889bd-openshift-default-566bd56c85-dvzs4" podUID="d848cad5-c34e-4841-a2a0-e3aab4a69029" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.165:15021/healthz/ready\": dial tcp 10.133.0.165:15021: connect: connection refused" Oct 30 21:23:15.304936 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:23:15.304905 5378 patch_prober.go:29] interesting pod/e2e-mt-d1b889bd-openshift-default-566bd56c85-dvzs4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.165:15021/healthz/ready\": dial tcp 10.133.0.165:15021: connect: connection refused" start-of-body= Oct 30 21:23:15.305400 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:23:15.304963 5378 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-d1b889bd-openshift-default-566bd56c85-dvzs4" podUID="d848cad5-c34e-4841-a2a0-e3aab4a69029" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.165:15021/healthz/ready\": dial tcp 10.133.0.165:15021: connect: connection refused" Oct 30 21:23:16.310253 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:23:16.310219 5378 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-d1b889bd-openshift-default-566bd56c85-dvzs4" Oct 30 21:23:16.310823 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:23:16.310678 5378 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-d1b889bd-openshift-default-566bd56c85-dvzs4" Oct 30 21:23:16.312096 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:23:16.312067 5378 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-d1b889bd-openshift-default-566bd56c85-dvzs4" Oct 30 21:23:18.837931 ip-10-0-138-149 kubenswrapper[5378]: E1030 21:23:18.837894 5378 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q6mx7" containerName="extractor" Oct 30 21:23:18.858596 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:23:18.858553 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:23:23.667650 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:23:23.667613 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:23:27.914714 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:23:27.914679 5378 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-2f9eff-8594f457db-5m52x"] Oct 30 21:23:27.917328 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:23:27.917292 5378 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-2f9eff-8594f457db-5m52x" podUID="fd6d4723-fe0b-4eae-94fb-c7b61de829c5" containerName="maas-api" containerID="cri-o://c893ee8fd320efc00243b332c70b50f6222eb24333264493df8692a72fc3f461" gracePeriod=30 Oct 30 21:23:28.025303 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:23:28.022015 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:23:28.262307 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:23:28.262267 5378 generic.go:356] "Generic (PLEG): container finished" podID="fd6d4723-fe0b-4eae-94fb-c7b61de829c5" containerID="c893ee8fd320efc00243b332c70b50f6222eb24333264493df8692a72fc3f461" exitCode=0 Oct 30 21:23:28.262538 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:23:28.262353 5378 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-2f9eff-8594f457db-5m52x" event={"ID":"fd6d4723-fe0b-4eae-94fb-c7b61de829c5","Type":"ContainerDied","Data":"c893ee8fd320efc00243b332c70b50f6222eb24333264493df8692a72fc3f461"} Oct 30 21:23:28.365252 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:23:28.364851 5378 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-2f9eff-8594f457db-5m52x" Oct 30 21:23:28.455105 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:23:28.455011 5378 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fd6d4723-fe0b-4eae-94fb-c7b61de829c5-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fd6d4723-fe0b-4eae-94fb-c7b61de829c5-maas-api-tls\") pod \"fd6d4723-fe0b-4eae-94fb-c7b61de829c5\" (UID: \"fd6d4723-fe0b-4eae-94fb-c7b61de829c5\") " Oct 30 21:23:28.455409 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:23:28.455385 5378 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fd6d4723-fe0b-4eae-94fb-c7b61de829c5-kube-api-access-g8bqr\" (UniqueName: \"kubernetes.io/projected/fd6d4723-fe0b-4eae-94fb-c7b61de829c5-kube-api-access-g8bqr\") pod \"fd6d4723-fe0b-4eae-94fb-c7b61de829c5\" (UID: \"fd6d4723-fe0b-4eae-94fb-c7b61de829c5\") " Oct 30 21:23:28.455614 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:23:28.455598 5378 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fd6d4723-fe0b-4eae-94fb-c7b61de829c5-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fd6d4723-fe0b-4eae-94fb-c7b61de829c5-openshift-service-ca\") pod \"fd6d4723-fe0b-4eae-94fb-c7b61de829c5\" (UID: \"fd6d4723-fe0b-4eae-94fb-c7b61de829c5\") " Oct 30 21:23:28.455841 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:23:28.455824 5378 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fd6d4723-fe0b-4eae-94fb-c7b61de829c5-metrics-certs\" (UniqueName: \"kubernetes.io/secret/fd6d4723-fe0b-4eae-94fb-c7b61de829c5-metrics-certs\") pod \"fd6d4723-fe0b-4eae-94fb-c7b61de829c5\" (UID: \"fd6d4723-fe0b-4eae-94fb-c7b61de829c5\") " Oct 30 21:23:28.457738 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:23:28.457677 5378 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fd6d4723-fe0b-4eae-94fb-c7b61de829c5-openshift-service-ca" pod "fd6d4723-fe0b-4eae-94fb-c7b61de829c5" (UID: "fd6d4723-fe0b-4eae-94fb-c7b61de829c5"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:23:28.497829 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:23:28.464355 5378 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fd6d4723-fe0b-4eae-94fb-c7b61de829c5-metrics-certs" pod "fd6d4723-fe0b-4eae-94fb-c7b61de829c5" (UID: "fd6d4723-fe0b-4eae-94fb-c7b61de829c5"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:23:28.497829 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:23:28.464508 5378 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fd6d4723-fe0b-4eae-94fb-c7b61de829c5-kube-api-access-g8bqr" pod "fd6d4723-fe0b-4eae-94fb-c7b61de829c5" (UID: "fd6d4723-fe0b-4eae-94fb-c7b61de829c5"). InnerVolumeSpecName "kube-api-access-g8bqr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:23:28.499064 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:23:28.499032 5378 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fd6d4723-fe0b-4eae-94fb-c7b61de829c5-maas-api-tls" pod "fd6d4723-fe0b-4eae-94fb-c7b61de829c5" (UID: "fd6d4723-fe0b-4eae-94fb-c7b61de829c5"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:23:28.573892 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:23:28.573846 5378 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fd6d4723-fe0b-4eae-94fb-c7b61de829c5-metrics-certs\") on node \"ip-10-0-138-149.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:23:28.573892 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:23:28.573889 5378 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fd6d4723-fe0b-4eae-94fb-c7b61de829c5-maas-api-tls\") on node \"ip-10-0-138-149.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:23:28.574172 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:23:28.573907 5378 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g8bqr\" (UniqueName: \"kubernetes.io/projected/fd6d4723-fe0b-4eae-94fb-c7b61de829c5-kube-api-access-g8bqr\") on node \"ip-10-0-138-149.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:23:28.574172 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:23:28.573926 5378 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fd6d4723-fe0b-4eae-94fb-c7b61de829c5-openshift-service-ca\") on node \"ip-10-0-138-149.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:23:29.270151 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:23:29.270119 5378 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-2f9eff-8594f457db-5m52x" Oct 30 21:23:29.270702 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:23:29.270121 5378 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-2f9eff-8594f457db-5m52x" event={"ID":"fd6d4723-fe0b-4eae-94fb-c7b61de829c5","Type":"ContainerDied","Data":"5e62c404cc16fba8ae0c69eaa518abdee924fda4c327c7656c1d7bf10442a113"} Oct 30 21:23:29.270702 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:23:29.270267 5378 scope.go:122] "RemoveContainer" containerID="c893ee8fd320efc00243b332c70b50f6222eb24333264493df8692a72fc3f461" Oct 30 21:23:29.436097 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:23:29.436064 5378 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-2f9eff-8594f457db-5m52x"] Oct 30 21:23:29.486439 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:23:29.486406 5378 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-2f9eff-8594f457db-5m52x"] Oct 30 21:23:29.653819 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:23:29.653788 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:23:29.741182 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:23:29.741150 5378 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fd6d4723-fe0b-4eae-94fb-c7b61de829c5" path="/var/lib/kubelet/pods/fd6d4723-fe0b-4eae-94fb-c7b61de829c5/volumes" Oct 30 21:23:29.971515 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:23:29.971437 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:23:30.112284 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:23:30.112253 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:23:30.134152 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:23:30.134128 5378 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-7x9jg_c2ff359c-cf92-4a5b-8610-f6eb41acbe73/ovn-acl-logging/0.log" Oct 30 21:23:30.134625 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:23:30.134606 5378 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-7x9jg_c2ff359c-cf92-4a5b-8610-f6eb41acbe73/ovn-acl-logging/0.log" Oct 30 21:23:30.753603 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:23:30.753555 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:23:35.151958 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:23:35.151928 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:23:36.818162 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:23:36.818127 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:23:36.975039 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:23:36.975011 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:23:37.201681 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:23:37.201646 5378 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-2f9eff-openshift-default-76f4d54d8c-mlhvh"] Oct 30 21:23:37.202124 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:23:37.202092 5378 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-2f9eff-openshift-default-76f4d54d8c-mlhvh" podUID="bebe8fda-f6a0-4527-9798-2cb425757af6" containerName="istio-proxy" containerID="cri-o://d909f2351ef68294519a050233e5adfed15786ac68ccf9d69a2d16d465dac571" gracePeriod=30 Oct 30 21:23:37.211594 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:23:37.211542 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:23:38.276657 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:23:38.276619 5378 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-769df7846c-b77jh"] Oct 30 21:23:38.277318 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:23:38.277258 5378 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-b77jh" podUID="7bfe56ea-dd63-441e-a5c1-9b116f068419" containerName="istio-proxy" containerID="cri-o://3dd624ef0bf385fcbf1445ed0e49b444edc0d889d3f0d9b3dcf8a159308c4ecc" gracePeriod=30 Oct 30 21:23:38.326866 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:23:38.326836 5378 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-769df7846c-r5j2r"] Oct 30 21:23:38.327085 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:23:38.327062 5378 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-r5j2r" podUID="04d5e742-36a9-41a8-9bb7-c9c8808d74cc" containerName="istio-proxy" containerID="cri-o://7bdaccbcc89ad3f46ed294c698f27162cbb4a16505169bcb29858637febdb929" gracePeriod=30 Oct 30 21:23:39.017626 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:23:39.017590 5378 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-769df7846c-b77jh container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 21:23:39.017827 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:23:39.017645 5378 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-b77jh" podUID="7bfe56ea-dd63-441e-a5c1-9b116f068419" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 21:23:39.976554 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:23:39.976519 5378 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-r9g59"] Oct 30 21:23:40.412218 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:23:40.412183 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:23:42.374712 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:23:42.374683 5378 generic.go:356] "Generic (PLEG): container finished" podID="bebe8fda-f6a0-4527-9798-2cb425757af6" containerID="d909f2351ef68294519a050233e5adfed15786ac68ccf9d69a2d16d465dac571" exitCode=0 Oct 30 21:23:42.375109 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:23:42.374764 5378 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-2f9eff-openshift-default-76f4d54d8c-mlhvh" event={"ID":"bebe8fda-f6a0-4527-9798-2cb425757af6","Type":"ContainerDied","Data":"d909f2351ef68294519a050233e5adfed15786ac68ccf9d69a2d16d465dac571"} Oct 30 21:23:42.470587 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:23:42.470519 5378 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-2f9eff-openshift-default-76f4d54d8c-mlhvh" Oct 30 21:23:42.643820 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:23:42.643762 5378 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bebe8fda-f6a0-4527-9798-2cb425757af6-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bebe8fda-f6a0-4527-9798-2cb425757af6-istio-data\") pod \"bebe8fda-f6a0-4527-9798-2cb425757af6\" (UID: \"bebe8fda-f6a0-4527-9798-2cb425757af6\") " Oct 30 21:23:42.644113 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:23:42.643840 5378 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bebe8fda-f6a0-4527-9798-2cb425757af6-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bebe8fda-f6a0-4527-9798-2cb425757af6-workload-certs\") pod \"bebe8fda-f6a0-4527-9798-2cb425757af6\" (UID: \"bebe8fda-f6a0-4527-9798-2cb425757af6\") " Oct 30 21:23:42.644113 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:23:42.643890 5378 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bebe8fda-f6a0-4527-9798-2cb425757af6-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bebe8fda-f6a0-4527-9798-2cb425757af6-credential-socket\") pod \"bebe8fda-f6a0-4527-9798-2cb425757af6\" (UID: \"bebe8fda-f6a0-4527-9798-2cb425757af6\") " Oct 30 21:23:42.644113 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:23:42.643938 5378 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bebe8fda-f6a0-4527-9798-2cb425757af6-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bebe8fda-f6a0-4527-9798-2cb425757af6-istio-envoy\") pod \"bebe8fda-f6a0-4527-9798-2cb425757af6\" (UID: \"bebe8fda-f6a0-4527-9798-2cb425757af6\") " Oct 30 21:23:42.644113 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:23:42.644004 5378 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bebe8fda-f6a0-4527-9798-2cb425757af6-istio-token\" (UniqueName: \"kubernetes.io/projected/bebe8fda-f6a0-4527-9798-2cb425757af6-istio-token\") pod \"bebe8fda-f6a0-4527-9798-2cb425757af6\" (UID: \"bebe8fda-f6a0-4527-9798-2cb425757af6\") " Oct 30 21:23:42.644113 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:23:42.644052 5378 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bebe8fda-f6a0-4527-9798-2cb425757af6-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bebe8fda-f6a0-4527-9798-2cb425757af6-workload-socket\") pod \"bebe8fda-f6a0-4527-9798-2cb425757af6\" (UID: \"bebe8fda-f6a0-4527-9798-2cb425757af6\") " Oct 30 21:23:42.644395 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:23:42.644150 5378 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/bebe8fda-f6a0-4527-9798-2cb425757af6-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bebe8fda-f6a0-4527-9798-2cb425757af6-istio-podinfo\") pod \"bebe8fda-f6a0-4527-9798-2cb425757af6\" (UID: \"bebe8fda-f6a0-4527-9798-2cb425757af6\") " Oct 30 21:23:42.644395 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:23:42.644205 5378 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bebe8fda-f6a0-4527-9798-2cb425757af6-kube-api-access-qllvm\" (UniqueName: \"kubernetes.io/projected/bebe8fda-f6a0-4527-9798-2cb425757af6-kube-api-access-qllvm\") pod \"bebe8fda-f6a0-4527-9798-2cb425757af6\" (UID: \"bebe8fda-f6a0-4527-9798-2cb425757af6\") " Oct 30 21:23:42.644395 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:23:42.644216 5378 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bebe8fda-f6a0-4527-9798-2cb425757af6-workload-certs" pod "bebe8fda-f6a0-4527-9798-2cb425757af6" (UID: "bebe8fda-f6a0-4527-9798-2cb425757af6"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:23:42.644395 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:23:42.644233 5378 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bebe8fda-f6a0-4527-9798-2cb425757af6-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bebe8fda-f6a0-4527-9798-2cb425757af6-istiod-ca-cert\") pod \"bebe8fda-f6a0-4527-9798-2cb425757af6\" (UID: \"bebe8fda-f6a0-4527-9798-2cb425757af6\") " Oct 30 21:23:42.644624 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:23:42.644412 5378 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bebe8fda-f6a0-4527-9798-2cb425757af6-istio-data" pod "bebe8fda-f6a0-4527-9798-2cb425757af6" (UID: "bebe8fda-f6a0-4527-9798-2cb425757af6"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:23:42.644786 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:23:42.644768 5378 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bebe8fda-f6a0-4527-9798-2cb425757af6-istio-data\") on node \"ip-10-0-138-149.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:23:42.644849 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:23:42.644787 5378 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bebe8fda-f6a0-4527-9798-2cb425757af6-workload-certs\") on node \"ip-10-0-138-149.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:23:42.644849 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:23:42.644789 5378 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bebe8fda-f6a0-4527-9798-2cb425757af6-istiod-ca-cert" pod "bebe8fda-f6a0-4527-9798-2cb425757af6" (UID: "bebe8fda-f6a0-4527-9798-2cb425757af6"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:23:42.645340 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:23:42.645054 5378 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bebe8fda-f6a0-4527-9798-2cb425757af6-workload-socket" pod "bebe8fda-f6a0-4527-9798-2cb425757af6" (UID: "bebe8fda-f6a0-4527-9798-2cb425757af6"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:23:42.646068 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:23:42.646037 5378 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bebe8fda-f6a0-4527-9798-2cb425757af6-credential-socket" pod "bebe8fda-f6a0-4527-9798-2cb425757af6" (UID: "bebe8fda-f6a0-4527-9798-2cb425757af6"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:23:42.647931 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:23:42.647905 5378 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bebe8fda-f6a0-4527-9798-2cb425757af6-istio-token" pod "bebe8fda-f6a0-4527-9798-2cb425757af6" (UID: "bebe8fda-f6a0-4527-9798-2cb425757af6"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:23:42.648920 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:23:42.648872 5378 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bebe8fda-f6a0-4527-9798-2cb425757af6-istio-envoy" pod "bebe8fda-f6a0-4527-9798-2cb425757af6" (UID: "bebe8fda-f6a0-4527-9798-2cb425757af6"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:23:42.651030 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:23:42.651009 5378 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bebe8fda-f6a0-4527-9798-2cb425757af6-kube-api-access-qllvm" pod "bebe8fda-f6a0-4527-9798-2cb425757af6" (UID: "bebe8fda-f6a0-4527-9798-2cb425757af6"). InnerVolumeSpecName "kube-api-access-qllvm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:23:42.655968 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:23:42.655941 5378 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/bebe8fda-f6a0-4527-9798-2cb425757af6-istio-podinfo" pod "bebe8fda-f6a0-4527-9798-2cb425757af6" (UID: "bebe8fda-f6a0-4527-9798-2cb425757af6"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 21:23:42.721816 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:23:42.721744 5378 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-1fdaa7-openshift-default-84586db7d5-sbpk8"] Oct 30 21:23:42.721972 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:23:42.721952 5378 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-1fdaa7-openshift-default-84586db7d5-sbpk8" podUID="2391a719-0c9b-4c56-b229-8f286a9aac96" containerName="istio-proxy" containerID="cri-o://92f7f85bab6bc2319d456ad69b2778e842c9505a32745d738d82a63bfbb854b5" gracePeriod=30 Oct 30 21:23:42.746501 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:23:42.746462 5378 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bebe8fda-f6a0-4527-9798-2cb425757af6-credential-socket\") on node \"ip-10-0-138-149.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:23:42.746501 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:23:42.746501 5378 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bebe8fda-f6a0-4527-9798-2cb425757af6-istio-envoy\") on node \"ip-10-0-138-149.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:23:42.746750 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:23:42.746516 5378 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bebe8fda-f6a0-4527-9798-2cb425757af6-istio-token\") on node \"ip-10-0-138-149.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:23:42.746750 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:23:42.746529 5378 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bebe8fda-f6a0-4527-9798-2cb425757af6-workload-socket\") on node \"ip-10-0-138-149.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:23:42.746750 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:23:42.746542 5378 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bebe8fda-f6a0-4527-9798-2cb425757af6-istio-podinfo\") on node \"ip-10-0-138-149.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:23:42.746750 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:23:42.746555 5378 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qllvm\" (UniqueName: \"kubernetes.io/projected/bebe8fda-f6a0-4527-9798-2cb425757af6-kube-api-access-qllvm\") on node \"ip-10-0-138-149.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:23:42.746750 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:23:42.746585 5378 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bebe8fda-f6a0-4527-9798-2cb425757af6-istiod-ca-cert\") on node \"ip-10-0-138-149.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:23:43.382407 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:23:43.382385 5378 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-2f9eff-openshift-default-76f4d54d8c-mlhvh" Oct 30 21:23:43.382833 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:23:43.382385 5378 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-2f9eff-openshift-default-76f4d54d8c-mlhvh" event={"ID":"bebe8fda-f6a0-4527-9798-2cb425757af6","Type":"ContainerDied","Data":"90ef0fda3cc250bcc1c4712b68d2e1d719bbedc548f1a6af4f8e9a48b18835b9"} Oct 30 21:23:43.382833 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:23:43.382527 5378 scope.go:122] "RemoveContainer" containerID="d909f2351ef68294519a050233e5adfed15786ac68ccf9d69a2d16d465dac571" Oct 30 21:23:43.565464 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:23:43.565430 5378 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-2f9eff-openshift-default-76f4d54d8c-mlhvh"] Oct 30 21:23:43.600682 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:23:43.600660 5378 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-b77jh" Oct 30 21:23:43.616853 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:23:43.616827 5378 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-2f9eff-openshift-default-76f4d54d8c-mlhvh"] Oct 30 21:23:43.635743 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:23:43.635721 5378 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-r5j2r" Oct 30 21:23:43.670189 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:23:43.670163 5378 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7bfe56ea-dd63-441e-a5c1-9b116f068419-kube-api-access-2rjhh\" (UniqueName: \"kubernetes.io/projected/7bfe56ea-dd63-441e-a5c1-9b116f068419-kube-api-access-2rjhh\") pod \"7bfe56ea-dd63-441e-a5c1-9b116f068419\" (UID: \"7bfe56ea-dd63-441e-a5c1-9b116f068419\") " Oct 30 21:23:43.670286 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:23:43.670193 5378 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/04d5e742-36a9-41a8-9bb7-c9c8808d74cc-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/04d5e742-36a9-41a8-9bb7-c9c8808d74cc-istio-podinfo\") pod \"04d5e742-36a9-41a8-9bb7-c9c8808d74cc\" (UID: \"04d5e742-36a9-41a8-9bb7-c9c8808d74cc\") " Oct 30 21:23:43.670286 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:23:43.670221 5378 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/04d5e742-36a9-41a8-9bb7-c9c8808d74cc-istio-token\" (UniqueName: \"kubernetes.io/projected/04d5e742-36a9-41a8-9bb7-c9c8808d74cc-istio-token\") pod \"04d5e742-36a9-41a8-9bb7-c9c8808d74cc\" (UID: \"04d5e742-36a9-41a8-9bb7-c9c8808d74cc\") " Oct 30 21:23:43.670286 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:23:43.670243 5378 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/04d5e742-36a9-41a8-9bb7-c9c8808d74cc-kube-api-access-lp65v\" (UniqueName: \"kubernetes.io/projected/04d5e742-36a9-41a8-9bb7-c9c8808d74cc-kube-api-access-lp65v\") pod \"04d5e742-36a9-41a8-9bb7-c9c8808d74cc\" (UID: \"04d5e742-36a9-41a8-9bb7-c9c8808d74cc\") " Oct 30 21:23:43.670478 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:23:43.670347 5378 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7bfe56ea-dd63-441e-a5c1-9b116f068419-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7bfe56ea-dd63-441e-a5c1-9b116f068419-istiod-ca-cert\") pod \"7bfe56ea-dd63-441e-a5c1-9b116f068419\" (UID: \"7bfe56ea-dd63-441e-a5c1-9b116f068419\") " Oct 30 21:23:43.670478 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:23:43.670410 5378 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/04d5e742-36a9-41a8-9bb7-c9c8808d74cc-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/04d5e742-36a9-41a8-9bb7-c9c8808d74cc-istiod-ca-cert\") pod \"04d5e742-36a9-41a8-9bb7-c9c8808d74cc\" (UID: \"04d5e742-36a9-41a8-9bb7-c9c8808d74cc\") " Oct 30 21:23:43.670478 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:23:43.670443 5378 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7bfe56ea-dd63-441e-a5c1-9b116f068419-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7bfe56ea-dd63-441e-a5c1-9b116f068419-credential-socket\") pod \"7bfe56ea-dd63-441e-a5c1-9b116f068419\" (UID: \"7bfe56ea-dd63-441e-a5c1-9b116f068419\") " Oct 30 21:23:43.670669 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:23:43.670479 5378 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/04d5e742-36a9-41a8-9bb7-c9c8808d74cc-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/04d5e742-36a9-41a8-9bb7-c9c8808d74cc-credential-socket\") pod \"04d5e742-36a9-41a8-9bb7-c9c8808d74cc\" (UID: \"04d5e742-36a9-41a8-9bb7-c9c8808d74cc\") " Oct 30 21:23:43.670669 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:23:43.670529 5378 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/04d5e742-36a9-41a8-9bb7-c9c8808d74cc-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/04d5e742-36a9-41a8-9bb7-c9c8808d74cc-istio-data\") pod \"04d5e742-36a9-41a8-9bb7-c9c8808d74cc\" (UID: \"04d5e742-36a9-41a8-9bb7-c9c8808d74cc\") " Oct 30 21:23:43.670800 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:23:43.670556 5378 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7bfe56ea-dd63-441e-a5c1-9b116f068419-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7bfe56ea-dd63-441e-a5c1-9b116f068419-workload-certs\") pod \"7bfe56ea-dd63-441e-a5c1-9b116f068419\" (UID: \"7bfe56ea-dd63-441e-a5c1-9b116f068419\") " Oct 30 21:23:43.671017 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:23:43.670996 5378 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/04d5e742-36a9-41a8-9bb7-c9c8808d74cc-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/04d5e742-36a9-41a8-9bb7-c9c8808d74cc-workload-socket\") pod \"04d5e742-36a9-41a8-9bb7-c9c8808d74cc\" (UID: \"04d5e742-36a9-41a8-9bb7-c9c8808d74cc\") " Oct 30 21:23:43.671152 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:23:43.670832 5378 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/04d5e742-36a9-41a8-9bb7-c9c8808d74cc-credential-socket" pod "04d5e742-36a9-41a8-9bb7-c9c8808d74cc" (UID: "04d5e742-36a9-41a8-9bb7-c9c8808d74cc"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:23:43.671225 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:23:43.670830 5378 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/04d5e742-36a9-41a8-9bb7-c9c8808d74cc-istiod-ca-cert" pod "04d5e742-36a9-41a8-9bb7-c9c8808d74cc" (UID: "04d5e742-36a9-41a8-9bb7-c9c8808d74cc"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:23:43.671225 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:23:43.671077 5378 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/04d5e742-36a9-41a8-9bb7-c9c8808d74cc-istio-data" pod "04d5e742-36a9-41a8-9bb7-c9c8808d74cc" (UID: "04d5e742-36a9-41a8-9bb7-c9c8808d74cc"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:23:43.671225 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:23:43.671119 5378 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7bfe56ea-dd63-441e-a5c1-9b116f068419-istiod-ca-cert" pod "7bfe56ea-dd63-441e-a5c1-9b116f068419" (UID: "7bfe56ea-dd63-441e-a5c1-9b116f068419"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:23:43.671225 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:23:43.671137 5378 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/7bfe56ea-dd63-441e-a5c1-9b116f068419-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7bfe56ea-dd63-441e-a5c1-9b116f068419-istio-podinfo\") pod \"7bfe56ea-dd63-441e-a5c1-9b116f068419\" (UID: \"7bfe56ea-dd63-441e-a5c1-9b116f068419\") " Oct 30 21:23:43.671428 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:23:43.671273 5378 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7bfe56ea-dd63-441e-a5c1-9b116f068419-workload-certs" pod "7bfe56ea-dd63-441e-a5c1-9b116f068419" (UID: "7bfe56ea-dd63-441e-a5c1-9b116f068419"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:23:43.671428 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:23:43.671296 5378 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7bfe56ea-dd63-441e-a5c1-9b116f068419-credential-socket" pod "7bfe56ea-dd63-441e-a5c1-9b116f068419" (UID: "7bfe56ea-dd63-441e-a5c1-9b116f068419"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:23:43.671428 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:23:43.671310 5378 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/04d5e742-36a9-41a8-9bb7-c9c8808d74cc-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/04d5e742-36a9-41a8-9bb7-c9c8808d74cc-workload-certs\") pod \"04d5e742-36a9-41a8-9bb7-c9c8808d74cc\" (UID: \"04d5e742-36a9-41a8-9bb7-c9c8808d74cc\") " Oct 30 21:23:43.671428 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:23:43.671351 5378 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7bfe56ea-dd63-441e-a5c1-9b116f068419-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7bfe56ea-dd63-441e-a5c1-9b116f068419-workload-socket\") pod \"7bfe56ea-dd63-441e-a5c1-9b116f068419\" (UID: \"7bfe56ea-dd63-441e-a5c1-9b116f068419\") " Oct 30 21:23:43.671428 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:23:43.671394 5378 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7bfe56ea-dd63-441e-a5c1-9b116f068419-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7bfe56ea-dd63-441e-a5c1-9b116f068419-istio-data\") pod \"7bfe56ea-dd63-441e-a5c1-9b116f068419\" (UID: \"7bfe56ea-dd63-441e-a5c1-9b116f068419\") " Oct 30 21:23:43.671428 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:23:43.671426 5378 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7bfe56ea-dd63-441e-a5c1-9b116f068419-istio-token\" (UniqueName: \"kubernetes.io/projected/7bfe56ea-dd63-441e-a5c1-9b116f068419-istio-token\") pod \"7bfe56ea-dd63-441e-a5c1-9b116f068419\" (UID: \"7bfe56ea-dd63-441e-a5c1-9b116f068419\") " Oct 30 21:23:43.671757 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:23:43.671470 5378 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/04d5e742-36a9-41a8-9bb7-c9c8808d74cc-workload-socket" pod "04d5e742-36a9-41a8-9bb7-c9c8808d74cc" (UID: "04d5e742-36a9-41a8-9bb7-c9c8808d74cc"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:23:43.671757 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:23:43.671516 5378 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7bfe56ea-dd63-441e-a5c1-9b116f068419-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7bfe56ea-dd63-441e-a5c1-9b116f068419-istio-envoy\") pod \"7bfe56ea-dd63-441e-a5c1-9b116f068419\" (UID: \"7bfe56ea-dd63-441e-a5c1-9b116f068419\") " Oct 30 21:23:43.671757 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:23:43.671586 5378 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/04d5e742-36a9-41a8-9bb7-c9c8808d74cc-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/04d5e742-36a9-41a8-9bb7-c9c8808d74cc-istio-envoy\") pod \"04d5e742-36a9-41a8-9bb7-c9c8808d74cc\" (UID: \"04d5e742-36a9-41a8-9bb7-c9c8808d74cc\") " Oct 30 21:23:43.671757 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:23:43.671688 5378 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7bfe56ea-dd63-441e-a5c1-9b116f068419-workload-socket" pod "7bfe56ea-dd63-441e-a5c1-9b116f068419" (UID: "7bfe56ea-dd63-441e-a5c1-9b116f068419"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:23:43.671944 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:23:43.671847 5378 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/04d5e742-36a9-41a8-9bb7-c9c8808d74cc-workload-certs" pod "04d5e742-36a9-41a8-9bb7-c9c8808d74cc" (UID: "04d5e742-36a9-41a8-9bb7-c9c8808d74cc"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:23:43.672278 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:23:43.672173 5378 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7bfe56ea-dd63-441e-a5c1-9b116f068419-istiod-ca-cert\") on node \"ip-10-0-138-149.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:23:43.672278 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:23:43.672200 5378 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/04d5e742-36a9-41a8-9bb7-c9c8808d74cc-istiod-ca-cert\") on node \"ip-10-0-138-149.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:23:43.672278 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:23:43.672213 5378 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7bfe56ea-dd63-441e-a5c1-9b116f068419-credential-socket\") on node \"ip-10-0-138-149.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:23:43.672278 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:23:43.672228 5378 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/04d5e742-36a9-41a8-9bb7-c9c8808d74cc-credential-socket\") on node \"ip-10-0-138-149.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:23:43.672278 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:23:43.672241 5378 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/04d5e742-36a9-41a8-9bb7-c9c8808d74cc-istio-data\") on node \"ip-10-0-138-149.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:23:43.672278 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:23:43.672254 5378 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7bfe56ea-dd63-441e-a5c1-9b116f068419-workload-certs\") on node \"ip-10-0-138-149.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:23:43.672278 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:23:43.672268 5378 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/04d5e742-36a9-41a8-9bb7-c9c8808d74cc-workload-socket\") on node \"ip-10-0-138-149.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:23:43.672672 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:23:43.672282 5378 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/04d5e742-36a9-41a8-9bb7-c9c8808d74cc-workload-certs\") on node \"ip-10-0-138-149.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:23:43.672672 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:23:43.672296 5378 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7bfe56ea-dd63-441e-a5c1-9b116f068419-workload-socket\") on node \"ip-10-0-138-149.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:23:43.672672 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:23:43.672363 5378 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7bfe56ea-dd63-441e-a5c1-9b116f068419-istio-data" pod "7bfe56ea-dd63-441e-a5c1-9b116f068419" (UID: "7bfe56ea-dd63-441e-a5c1-9b116f068419"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:23:43.673086 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:23:43.673058 5378 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/04d5e742-36a9-41a8-9bb7-c9c8808d74cc-istio-token" pod "04d5e742-36a9-41a8-9bb7-c9c8808d74cc" (UID: "04d5e742-36a9-41a8-9bb7-c9c8808d74cc"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:23:43.673189 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:23:43.673109 5378 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7bfe56ea-dd63-441e-a5c1-9b116f068419-kube-api-access-2rjhh" pod "7bfe56ea-dd63-441e-a5c1-9b116f068419" (UID: "7bfe56ea-dd63-441e-a5c1-9b116f068419"). InnerVolumeSpecName "kube-api-access-2rjhh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:23:43.673396 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:23:43.673368 5378 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/04d5e742-36a9-41a8-9bb7-c9c8808d74cc-kube-api-access-lp65v" pod "04d5e742-36a9-41a8-9bb7-c9c8808d74cc" (UID: "04d5e742-36a9-41a8-9bb7-c9c8808d74cc"). InnerVolumeSpecName "kube-api-access-lp65v". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:23:43.674030 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:23:43.674004 5378 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7bfe56ea-dd63-441e-a5c1-9b116f068419-istio-token" pod "7bfe56ea-dd63-441e-a5c1-9b116f068419" (UID: "7bfe56ea-dd63-441e-a5c1-9b116f068419"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:23:43.674127 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:23:43.674063 5378 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/04d5e742-36a9-41a8-9bb7-c9c8808d74cc-istio-podinfo" pod "04d5e742-36a9-41a8-9bb7-c9c8808d74cc" (UID: "04d5e742-36a9-41a8-9bb7-c9c8808d74cc"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 21:23:43.674127 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:23:43.674063 5378 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/04d5e742-36a9-41a8-9bb7-c9c8808d74cc-istio-envoy" pod "04d5e742-36a9-41a8-9bb7-c9c8808d74cc" (UID: "04d5e742-36a9-41a8-9bb7-c9c8808d74cc"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:23:43.674465 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:23:43.674448 5378 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/7bfe56ea-dd63-441e-a5c1-9b116f068419-istio-podinfo" pod "7bfe56ea-dd63-441e-a5c1-9b116f068419" (UID: "7bfe56ea-dd63-441e-a5c1-9b116f068419"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 21:23:43.675165 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:23:43.675149 5378 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7bfe56ea-dd63-441e-a5c1-9b116f068419-istio-envoy" pod "7bfe56ea-dd63-441e-a5c1-9b116f068419" (UID: "7bfe56ea-dd63-441e-a5c1-9b116f068419"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:23:43.745766 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:23:43.745728 5378 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bebe8fda-f6a0-4527-9798-2cb425757af6" path="/var/lib/kubelet/pods/bebe8fda-f6a0-4527-9798-2cb425757af6/volumes" Oct 30 21:23:43.773646 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:23:43.773622 5378 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7bfe56ea-dd63-441e-a5c1-9b116f068419-istio-envoy\") on node \"ip-10-0-138-149.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:23:43.773646 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:23:43.773646 5378 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/04d5e742-36a9-41a8-9bb7-c9c8808d74cc-istio-envoy\") on node \"ip-10-0-138-149.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:23:43.773810 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:23:43.773656 5378 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2rjhh\" (UniqueName: \"kubernetes.io/projected/7bfe56ea-dd63-441e-a5c1-9b116f068419-kube-api-access-2rjhh\") on node \"ip-10-0-138-149.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:23:43.773810 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:23:43.773665 5378 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/04d5e742-36a9-41a8-9bb7-c9c8808d74cc-istio-podinfo\") on node \"ip-10-0-138-149.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:23:43.773810 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:23:43.773676 5378 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/04d5e742-36a9-41a8-9bb7-c9c8808d74cc-istio-token\") on node \"ip-10-0-138-149.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:23:43.773810 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:23:43.773685 5378 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lp65v\" (UniqueName: \"kubernetes.io/projected/04d5e742-36a9-41a8-9bb7-c9c8808d74cc-kube-api-access-lp65v\") on node \"ip-10-0-138-149.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:23:43.773810 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:23:43.773694 5378 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7bfe56ea-dd63-441e-a5c1-9b116f068419-istio-podinfo\") on node \"ip-10-0-138-149.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:23:43.773810 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:23:43.773702 5378 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7bfe56ea-dd63-441e-a5c1-9b116f068419-istio-data\") on node \"ip-10-0-138-149.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:23:43.773810 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:23:43.773711 5378 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7bfe56ea-dd63-441e-a5c1-9b116f068419-istio-token\") on node \"ip-10-0-138-149.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:23:44.389525 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:23:44.389489 5378 generic.go:356] "Generic (PLEG): container finished" podID="7bfe56ea-dd63-441e-a5c1-9b116f068419" containerID="3dd624ef0bf385fcbf1445ed0e49b444edc0d889d3f0d9b3dcf8a159308c4ecc" exitCode=0 Oct 30 21:23:44.389982 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:23:44.389541 5378 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-b77jh" Oct 30 21:23:44.389982 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:23:44.389596 5378 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-b77jh" event={"ID":"7bfe56ea-dd63-441e-a5c1-9b116f068419","Type":"ContainerDied","Data":"3dd624ef0bf385fcbf1445ed0e49b444edc0d889d3f0d9b3dcf8a159308c4ecc"} Oct 30 21:23:44.389982 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:23:44.389653 5378 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-b77jh" event={"ID":"7bfe56ea-dd63-441e-a5c1-9b116f068419","Type":"ContainerDied","Data":"f7e9192de7bbe4569bf25aa3ed298ce5c4a9d22bbd225dbfd7ea25c87780a3f6"} Oct 30 21:23:44.389982 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:23:44.389683 5378 scope.go:122] "RemoveContainer" containerID="3dd624ef0bf385fcbf1445ed0e49b444edc0d889d3f0d9b3dcf8a159308c4ecc" Oct 30 21:23:44.391266 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:23:44.391244 5378 generic.go:356] "Generic (PLEG): container finished" podID="04d5e742-36a9-41a8-9bb7-c9c8808d74cc" containerID="7bdaccbcc89ad3f46ed294c698f27162cbb4a16505169bcb29858637febdb929" exitCode=0 Oct 30 21:23:44.391392 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:23:44.391289 5378 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-r5j2r" Oct 30 21:23:44.391392 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:23:44.391324 5378 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-r5j2r" event={"ID":"04d5e742-36a9-41a8-9bb7-c9c8808d74cc","Type":"ContainerDied","Data":"7bdaccbcc89ad3f46ed294c698f27162cbb4a16505169bcb29858637febdb929"} Oct 30 21:23:44.391392 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:23:44.391368 5378 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-r5j2r" event={"ID":"04d5e742-36a9-41a8-9bb7-c9c8808d74cc","Type":"ContainerDied","Data":"94734d844da343f0b612d6fc696ffa29b3a3cea6c636cc6b5e0b9ba48ff23f1b"} Oct 30 21:23:44.399382 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:23:44.399367 5378 scope.go:122] "RemoveContainer" containerID="3dd624ef0bf385fcbf1445ed0e49b444edc0d889d3f0d9b3dcf8a159308c4ecc" Oct 30 21:23:44.399641 ip-10-0-138-149 kubenswrapper[5378]: E1030 21:23:44.399622 5378 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3dd624ef0bf385fcbf1445ed0e49b444edc0d889d3f0d9b3dcf8a159308c4ecc\": container with ID starting with 3dd624ef0bf385fcbf1445ed0e49b444edc0d889d3f0d9b3dcf8a159308c4ecc not found: ID does not exist" containerID="3dd624ef0bf385fcbf1445ed0e49b444edc0d889d3f0d9b3dcf8a159308c4ecc" Oct 30 21:23:44.399692 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:23:44.399648 5378 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3dd624ef0bf385fcbf1445ed0e49b444edc0d889d3f0d9b3dcf8a159308c4ecc"} err="failed to get container status \"3dd624ef0bf385fcbf1445ed0e49b444edc0d889d3f0d9b3dcf8a159308c4ecc\": rpc error: code = NotFound desc = could not find container \"3dd624ef0bf385fcbf1445ed0e49b444edc0d889d3f0d9b3dcf8a159308c4ecc\": container with ID starting with 3dd624ef0bf385fcbf1445ed0e49b444edc0d889d3f0d9b3dcf8a159308c4ecc not found: ID does not exist" Oct 30 21:23:44.399692 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:23:44.399662 5378 scope.go:122] "RemoveContainer" containerID="7bdaccbcc89ad3f46ed294c698f27162cbb4a16505169bcb29858637febdb929" Oct 30 21:23:44.408333 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:23:44.408319 5378 scope.go:122] "RemoveContainer" containerID="7bdaccbcc89ad3f46ed294c698f27162cbb4a16505169bcb29858637febdb929" Oct 30 21:23:44.408625 ip-10-0-138-149 kubenswrapper[5378]: E1030 21:23:44.408602 5378 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7bdaccbcc89ad3f46ed294c698f27162cbb4a16505169bcb29858637febdb929\": container with ID starting with 7bdaccbcc89ad3f46ed294c698f27162cbb4a16505169bcb29858637febdb929 not found: ID does not exist" containerID="7bdaccbcc89ad3f46ed294c698f27162cbb4a16505169bcb29858637febdb929" Oct 30 21:23:44.408759 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:23:44.408631 5378 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7bdaccbcc89ad3f46ed294c698f27162cbb4a16505169bcb29858637febdb929"} err="failed to get container status \"7bdaccbcc89ad3f46ed294c698f27162cbb4a16505169bcb29858637febdb929\": rpc error: code = NotFound desc = could not find container \"7bdaccbcc89ad3f46ed294c698f27162cbb4a16505169bcb29858637febdb929\": container with ID starting with 7bdaccbcc89ad3f46ed294c698f27162cbb4a16505169bcb29858637febdb929 not found: ID does not exist" Oct 30 21:23:44.556179 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:23:44.556135 5378 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-769df7846c-b77jh"] Oct 30 21:23:44.606614 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:23:44.606559 5378 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-769df7846c-b77jh"] Oct 30 21:23:44.718067 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:23:44.718038 5378 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-769df7846c-r5j2r"] Oct 30 21:23:44.723717 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:23:44.723691 5378 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-769df7846c-r5j2r"] Oct 30 21:23:45.081479 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:23:45.081396 5378 patch_prober.go:29] interesting pod/e2e-worker-w4-1fdaa7-openshift-default-84586db7d5-sbpk8 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 21:23:45.081479 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:23:45.081451 5378 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-1fdaa7-openshift-default-84586db7d5-sbpk8" podUID="2391a719-0c9b-4c56-b229-8f286a9aac96" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 21:23:45.691454 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:23:45.691424 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:23:45.763205 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:23:45.763156 5378 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="04d5e742-36a9-41a8-9bb7-c9c8808d74cc" path="/var/lib/kubelet/pods/04d5e742-36a9-41a8-9bb7-c9c8808d74cc/volumes" Oct 30 21:23:45.763849 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:23:45.763828 5378 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7bfe56ea-dd63-441e-a5c1-9b116f068419" path="/var/lib/kubelet/pods/7bfe56ea-dd63-441e-a5c1-9b116f068419/volumes" Oct 30 21:23:47.954990 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:23:47.954958 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:23:47.999008 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:23:47.998951 5378 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-1fdaa7-openshift-default-84586db7d5-sbpk8" Oct 30 21:23:48.026997 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:23:48.026066 5378 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2391a719-0c9b-4c56-b229-8f286a9aac96-kube-api-access-6fq99\" (UniqueName: \"kubernetes.io/projected/2391a719-0c9b-4c56-b229-8f286a9aac96-kube-api-access-6fq99\") pod \"2391a719-0c9b-4c56-b229-8f286a9aac96\" (UID: \"2391a719-0c9b-4c56-b229-8f286a9aac96\") " Oct 30 21:23:48.026997 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:23:48.026159 5378 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2391a719-0c9b-4c56-b229-8f286a9aac96-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2391a719-0c9b-4c56-b229-8f286a9aac96-istio-envoy\") pod \"2391a719-0c9b-4c56-b229-8f286a9aac96\" (UID: \"2391a719-0c9b-4c56-b229-8f286a9aac96\") " Oct 30 21:23:48.026997 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:23:48.026223 5378 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2391a719-0c9b-4c56-b229-8f286a9aac96-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2391a719-0c9b-4c56-b229-8f286a9aac96-credential-socket\") pod \"2391a719-0c9b-4c56-b229-8f286a9aac96\" (UID: \"2391a719-0c9b-4c56-b229-8f286a9aac96\") " Oct 30 21:23:48.026997 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:23:48.026252 5378 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2391a719-0c9b-4c56-b229-8f286a9aac96-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2391a719-0c9b-4c56-b229-8f286a9aac96-workload-certs\") pod \"2391a719-0c9b-4c56-b229-8f286a9aac96\" (UID: \"2391a719-0c9b-4c56-b229-8f286a9aac96\") " Oct 30 21:23:48.026997 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:23:48.026302 5378 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2391a719-0c9b-4c56-b229-8f286a9aac96-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2391a719-0c9b-4c56-b229-8f286a9aac96-istio-data\") pod \"2391a719-0c9b-4c56-b229-8f286a9aac96\" (UID: \"2391a719-0c9b-4c56-b229-8f286a9aac96\") " Oct 30 21:23:48.026997 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:23:48.026329 5378 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2391a719-0c9b-4c56-b229-8f286a9aac96-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2391a719-0c9b-4c56-b229-8f286a9aac96-istiod-ca-cert\") pod \"2391a719-0c9b-4c56-b229-8f286a9aac96\" (UID: \"2391a719-0c9b-4c56-b229-8f286a9aac96\") " Oct 30 21:23:48.026997 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:23:48.026395 5378 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2391a719-0c9b-4c56-b229-8f286a9aac96-istio-token\" (UniqueName: \"kubernetes.io/projected/2391a719-0c9b-4c56-b229-8f286a9aac96-istio-token\") pod \"2391a719-0c9b-4c56-b229-8f286a9aac96\" (UID: \"2391a719-0c9b-4c56-b229-8f286a9aac96\") " Oct 30 21:23:48.026997 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:23:48.026442 5378 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2391a719-0c9b-4c56-b229-8f286a9aac96-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2391a719-0c9b-4c56-b229-8f286a9aac96-workload-socket\") pod \"2391a719-0c9b-4c56-b229-8f286a9aac96\" (UID: \"2391a719-0c9b-4c56-b229-8f286a9aac96\") " Oct 30 21:23:48.026997 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:23:48.026489 5378 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/2391a719-0c9b-4c56-b229-8f286a9aac96-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2391a719-0c9b-4c56-b229-8f286a9aac96-istio-podinfo\") pod \"2391a719-0c9b-4c56-b229-8f286a9aac96\" (UID: \"2391a719-0c9b-4c56-b229-8f286a9aac96\") " Oct 30 21:23:48.028502 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:23:48.028268 5378 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2391a719-0c9b-4c56-b229-8f286a9aac96-istiod-ca-cert" pod "2391a719-0c9b-4c56-b229-8f286a9aac96" (UID: "2391a719-0c9b-4c56-b229-8f286a9aac96"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:23:48.028876 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:23:48.028691 5378 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2391a719-0c9b-4c56-b229-8f286a9aac96-workload-certs" pod "2391a719-0c9b-4c56-b229-8f286a9aac96" (UID: "2391a719-0c9b-4c56-b229-8f286a9aac96"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:23:48.029014 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:23:48.028988 5378 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2391a719-0c9b-4c56-b229-8f286a9aac96-istio-data" pod "2391a719-0c9b-4c56-b229-8f286a9aac96" (UID: "2391a719-0c9b-4c56-b229-8f286a9aac96"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:23:48.029738 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:23:48.029644 5378 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2391a719-0c9b-4c56-b229-8f286a9aac96-workload-socket" pod "2391a719-0c9b-4c56-b229-8f286a9aac96" (UID: "2391a719-0c9b-4c56-b229-8f286a9aac96"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:23:48.038620 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:23:48.035001 5378 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2391a719-0c9b-4c56-b229-8f286a9aac96-workload-certs\") on node \"ip-10-0-138-149.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:23:48.038620 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:23:48.035031 5378 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2391a719-0c9b-4c56-b229-8f286a9aac96-istio-data\") on node \"ip-10-0-138-149.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:23:48.038620 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:23:48.035046 5378 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2391a719-0c9b-4c56-b229-8f286a9aac96-istiod-ca-cert\") on node \"ip-10-0-138-149.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:23:48.038620 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:23:48.035059 5378 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2391a719-0c9b-4c56-b229-8f286a9aac96-workload-socket\") on node \"ip-10-0-138-149.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:23:48.069630 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:23:48.054671 5378 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2391a719-0c9b-4c56-b229-8f286a9aac96-credential-socket" pod "2391a719-0c9b-4c56-b229-8f286a9aac96" (UID: "2391a719-0c9b-4c56-b229-8f286a9aac96"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:23:48.072396 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:23:48.072341 5378 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/2391a719-0c9b-4c56-b229-8f286a9aac96-istio-podinfo" pod "2391a719-0c9b-4c56-b229-8f286a9aac96" (UID: "2391a719-0c9b-4c56-b229-8f286a9aac96"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 21:23:48.074660 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:23:48.073696 5378 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2391a719-0c9b-4c56-b229-8f286a9aac96-kube-api-access-6fq99" pod "2391a719-0c9b-4c56-b229-8f286a9aac96" (UID: "2391a719-0c9b-4c56-b229-8f286a9aac96"). InnerVolumeSpecName "kube-api-access-6fq99". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:23:48.074660 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:23:48.074164 5378 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2391a719-0c9b-4c56-b229-8f286a9aac96-istio-envoy" pod "2391a719-0c9b-4c56-b229-8f286a9aac96" (UID: "2391a719-0c9b-4c56-b229-8f286a9aac96"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:23:48.076342 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:23:48.076315 5378 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2391a719-0c9b-4c56-b229-8f286a9aac96-istio-token" pod "2391a719-0c9b-4c56-b229-8f286a9aac96" (UID: "2391a719-0c9b-4c56-b229-8f286a9aac96"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:23:48.136523 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:23:48.136481 5378 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6fq99\" (UniqueName: \"kubernetes.io/projected/2391a719-0c9b-4c56-b229-8f286a9aac96-kube-api-access-6fq99\") on node \"ip-10-0-138-149.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:23:48.136523 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:23:48.136512 5378 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2391a719-0c9b-4c56-b229-8f286a9aac96-istio-envoy\") on node \"ip-10-0-138-149.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:23:48.136523 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:23:48.136523 5378 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2391a719-0c9b-4c56-b229-8f286a9aac96-credential-socket\") on node \"ip-10-0-138-149.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:23:48.136523 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:23:48.136534 5378 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2391a719-0c9b-4c56-b229-8f286a9aac96-istio-token\") on node \"ip-10-0-138-149.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:23:48.136923 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:23:48.136547 5378 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2391a719-0c9b-4c56-b229-8f286a9aac96-istio-podinfo\") on node \"ip-10-0-138-149.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:23:48.420381 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:23:48.420349 5378 generic.go:356] "Generic (PLEG): container finished" podID="2391a719-0c9b-4c56-b229-8f286a9aac96" containerID="92f7f85bab6bc2319d456ad69b2778e842c9505a32745d738d82a63bfbb854b5" exitCode=0 Oct 30 21:23:48.420550 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:23:48.420407 5378 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-1fdaa7-openshift-default-84586db7d5-sbpk8" Oct 30 21:23:48.420550 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:23:48.420429 5378 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-1fdaa7-openshift-default-84586db7d5-sbpk8" event={"ID":"2391a719-0c9b-4c56-b229-8f286a9aac96","Type":"ContainerDied","Data":"92f7f85bab6bc2319d456ad69b2778e842c9505a32745d738d82a63bfbb854b5"} Oct 30 21:23:48.420550 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:23:48.420480 5378 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-1fdaa7-openshift-default-84586db7d5-sbpk8" event={"ID":"2391a719-0c9b-4c56-b229-8f286a9aac96","Type":"ContainerDied","Data":"ff9f5edb579cceae20821e3cd210956f935a877232b79d3769f6712512de9b7e"} Oct 30 21:23:48.420550 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:23:48.420503 5378 scope.go:122] "RemoveContainer" containerID="92f7f85bab6bc2319d456ad69b2778e842c9505a32745d738d82a63bfbb854b5" Oct 30 21:23:48.431155 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:23:48.431139 5378 scope.go:122] "RemoveContainer" containerID="4c4e053ee467b0b8deb3acae00ce4858327a6294f588d162c000012246a5364b" Oct 30 21:23:48.439514 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:23:48.439500 5378 scope.go:122] "RemoveContainer" containerID="92f7f85bab6bc2319d456ad69b2778e842c9505a32745d738d82a63bfbb854b5" Oct 30 21:23:48.439790 ip-10-0-138-149 kubenswrapper[5378]: E1030 21:23:48.439772 5378 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"92f7f85bab6bc2319d456ad69b2778e842c9505a32745d738d82a63bfbb854b5\": container with ID starting with 92f7f85bab6bc2319d456ad69b2778e842c9505a32745d738d82a63bfbb854b5 not found: ID does not exist" containerID="92f7f85bab6bc2319d456ad69b2778e842c9505a32745d738d82a63bfbb854b5" Oct 30 21:23:48.439848 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:23:48.439799 5378 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"92f7f85bab6bc2319d456ad69b2778e842c9505a32745d738d82a63bfbb854b5"} err="failed to get container status \"92f7f85bab6bc2319d456ad69b2778e842c9505a32745d738d82a63bfbb854b5\": rpc error: code = NotFound desc = could not find container \"92f7f85bab6bc2319d456ad69b2778e842c9505a32745d738d82a63bfbb854b5\": container with ID starting with 92f7f85bab6bc2319d456ad69b2778e842c9505a32745d738d82a63bfbb854b5 not found: ID does not exist" Oct 30 21:23:48.439848 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:23:48.439815 5378 scope.go:122] "RemoveContainer" containerID="4c4e053ee467b0b8deb3acae00ce4858327a6294f588d162c000012246a5364b" Oct 30 21:23:48.440057 ip-10-0-138-149 kubenswrapper[5378]: E1030 21:23:48.440037 5378 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4c4e053ee467b0b8deb3acae00ce4858327a6294f588d162c000012246a5364b\": container with ID starting with 4c4e053ee467b0b8deb3acae00ce4858327a6294f588d162c000012246a5364b not found: ID does not exist" containerID="4c4e053ee467b0b8deb3acae00ce4858327a6294f588d162c000012246a5364b" Oct 30 21:23:48.440107 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:23:48.440063 5378 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4c4e053ee467b0b8deb3acae00ce4858327a6294f588d162c000012246a5364b"} err="failed to get container status \"4c4e053ee467b0b8deb3acae00ce4858327a6294f588d162c000012246a5364b\": rpc error: code = NotFound desc = could not find container \"4c4e053ee467b0b8deb3acae00ce4858327a6294f588d162c000012246a5364b\": container with ID starting with 4c4e053ee467b0b8deb3acae00ce4858327a6294f588d162c000012246a5364b not found: ID does not exist" Oct 30 21:23:48.587110 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:23:48.587077 5378 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-1fdaa7-openshift-default-84586db7d5-sbpk8"] Oct 30 21:23:48.590095 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:23:48.590066 5378 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-1fdaa7-openshift-default-84586db7d5-sbpk8"] Oct 30 21:23:48.837215 ip-10-0-138-149 kubenswrapper[5378]: E1030 21:23:48.837180 5378 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q6mx7" containerName="extractor" Oct 30 21:23:49.740837 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:23:49.740797 5378 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2391a719-0c9b-4c56-b229-8f286a9aac96" path="/var/lib/kubelet/pods/2391a719-0c9b-4c56-b229-8f286a9aac96/volumes" Oct 30 21:23:50.478740 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:23:50.478705 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:23:51.101180 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:23:51.101136 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:23:51.223601 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:23:51.223539 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:23:51.498125 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:23:51.498030 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:23:53.603028 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:23:53.602978 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:23:53.722579 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:23:53.722538 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:23:55.307427 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:23:55.307400 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:24:00.068464 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:24:00.068428 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:24:01.254515 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:24:01.254480 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:24:04.955744 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:24:04.955710 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:24:09.775615 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:24:09.775558 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:24:10.304382 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:24:10.304341 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:24:10.307647 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:24:10.307624 5378 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-639b07/test-model-639b07-kserve-b8485fd9-ph8tq"] Oct 30 21:24:11.188036 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:24:11.188000 5378 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-r9g59"] Oct 30 21:24:14.354722 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:24:14.354691 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:24:16.021507 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:24:16.021471 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:24:16.304725 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:24:16.304622 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:24:16.441113 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:24:16.441082 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:24:16.954255 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:24:16.954209 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:24:18.836313 ip-10-0-138-149 kubenswrapper[5378]: E1030 21:24:18.836276 5378 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q6mx7" containerName="extractor" Oct 30 21:24:20.734249 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:24:20.734214 5378 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-08df98-dc45f5c6-6hbng"] Oct 30 21:24:20.734725 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:24:20.734411 5378 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-08df98-dc45f5c6-6hbng" podUID="0acdf79f-8312-4818-a1fb-8e50c7948634" containerName="maas-api" containerID="cri-o://e2e1204fc7b8f93163152667ed40a0c4e55182dd1010853dfd0bf8c317eccb01" gracePeriod=30 Oct 30 21:24:20.824893 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:24:20.824844 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:24:21.039281 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:24:21.039254 5378 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-08df98-dc45f5c6-6hbng" Oct 30 21:24:21.110855 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:24:21.110823 5378 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0acdf79f-8312-4818-a1fb-8e50c7948634-metrics-certs\" (UniqueName: \"kubernetes.io/secret/0acdf79f-8312-4818-a1fb-8e50c7948634-metrics-certs\") pod \"0acdf79f-8312-4818-a1fb-8e50c7948634\" (UID: \"0acdf79f-8312-4818-a1fb-8e50c7948634\") " Oct 30 21:24:21.111037 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:24:21.110958 5378 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0acdf79f-8312-4818-a1fb-8e50c7948634-kube-api-access-zm8h5\" (UniqueName: \"kubernetes.io/projected/0acdf79f-8312-4818-a1fb-8e50c7948634-kube-api-access-zm8h5\") pod \"0acdf79f-8312-4818-a1fb-8e50c7948634\" (UID: \"0acdf79f-8312-4818-a1fb-8e50c7948634\") " Oct 30 21:24:21.111095 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:24:21.111042 5378 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0acdf79f-8312-4818-a1fb-8e50c7948634-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0acdf79f-8312-4818-a1fb-8e50c7948634-openshift-service-ca\") pod \"0acdf79f-8312-4818-a1fb-8e50c7948634\" (UID: \"0acdf79f-8312-4818-a1fb-8e50c7948634\") " Oct 30 21:24:21.111095 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:24:21.111080 5378 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0acdf79f-8312-4818-a1fb-8e50c7948634-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0acdf79f-8312-4818-a1fb-8e50c7948634-maas-api-tls\") pod \"0acdf79f-8312-4818-a1fb-8e50c7948634\" (UID: \"0acdf79f-8312-4818-a1fb-8e50c7948634\") " Oct 30 21:24:21.111406 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:24:21.111373 5378 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0acdf79f-8312-4818-a1fb-8e50c7948634-openshift-service-ca" pod "0acdf79f-8312-4818-a1fb-8e50c7948634" (UID: "0acdf79f-8312-4818-a1fb-8e50c7948634"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:24:21.112959 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:24:21.112934 5378 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0acdf79f-8312-4818-a1fb-8e50c7948634-metrics-certs" pod "0acdf79f-8312-4818-a1fb-8e50c7948634" (UID: "0acdf79f-8312-4818-a1fb-8e50c7948634"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:24:21.113313 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:24:21.113282 5378 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0acdf79f-8312-4818-a1fb-8e50c7948634-maas-api-tls" pod "0acdf79f-8312-4818-a1fb-8e50c7948634" (UID: "0acdf79f-8312-4818-a1fb-8e50c7948634"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:24:21.113409 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:24:21.113316 5378 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0acdf79f-8312-4818-a1fb-8e50c7948634-kube-api-access-zm8h5" pod "0acdf79f-8312-4818-a1fb-8e50c7948634" (UID: "0acdf79f-8312-4818-a1fb-8e50c7948634"). InnerVolumeSpecName "kube-api-access-zm8h5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:24:21.212642 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:24:21.212612 5378 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zm8h5\" (UniqueName: \"kubernetes.io/projected/0acdf79f-8312-4818-a1fb-8e50c7948634-kube-api-access-zm8h5\") on node \"ip-10-0-138-149.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:24:21.212642 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:24:21.212638 5378 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0acdf79f-8312-4818-a1fb-8e50c7948634-openshift-service-ca\") on node \"ip-10-0-138-149.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:24:21.212642 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:24:21.212650 5378 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0acdf79f-8312-4818-a1fb-8e50c7948634-maas-api-tls\") on node \"ip-10-0-138-149.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:24:21.212875 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:24:21.212659 5378 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0acdf79f-8312-4818-a1fb-8e50c7948634-metrics-certs\") on node \"ip-10-0-138-149.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:24:21.621055 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:24:21.621022 5378 generic.go:356] "Generic (PLEG): container finished" podID="0acdf79f-8312-4818-a1fb-8e50c7948634" containerID="e2e1204fc7b8f93163152667ed40a0c4e55182dd1010853dfd0bf8c317eccb01" exitCode=0 Oct 30 21:24:21.621224 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:24:21.621073 5378 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-08df98-dc45f5c6-6hbng" Oct 30 21:24:21.621224 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:24:21.621109 5378 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-08df98-dc45f5c6-6hbng" event={"ID":"0acdf79f-8312-4818-a1fb-8e50c7948634","Type":"ContainerDied","Data":"e2e1204fc7b8f93163152667ed40a0c4e55182dd1010853dfd0bf8c317eccb01"} Oct 30 21:24:21.621224 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:24:21.621164 5378 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-08df98-dc45f5c6-6hbng" event={"ID":"0acdf79f-8312-4818-a1fb-8e50c7948634","Type":"ContainerDied","Data":"f843f373ec6dc9d712644fea33c0b3deeca7ab1e3c2a37e42c0bac4ea5701754"} Oct 30 21:24:21.621224 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:24:21.621186 5378 scope.go:122] "RemoveContainer" containerID="e2e1204fc7b8f93163152667ed40a0c4e55182dd1010853dfd0bf8c317eccb01" Oct 30 21:24:21.631149 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:24:21.631126 5378 scope.go:122] "RemoveContainer" containerID="e2e1204fc7b8f93163152667ed40a0c4e55182dd1010853dfd0bf8c317eccb01" Oct 30 21:24:21.631430 ip-10-0-138-149 kubenswrapper[5378]: E1030 21:24:21.631410 5378 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e2e1204fc7b8f93163152667ed40a0c4e55182dd1010853dfd0bf8c317eccb01\": container with ID starting with e2e1204fc7b8f93163152667ed40a0c4e55182dd1010853dfd0bf8c317eccb01 not found: ID does not exist" containerID="e2e1204fc7b8f93163152667ed40a0c4e55182dd1010853dfd0bf8c317eccb01" Oct 30 21:24:21.631505 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:24:21.631437 5378 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e2e1204fc7b8f93163152667ed40a0c4e55182dd1010853dfd0bf8c317eccb01"} err="failed to get container status \"e2e1204fc7b8f93163152667ed40a0c4e55182dd1010853dfd0bf8c317eccb01\": rpc error: code = NotFound desc = could not find container \"e2e1204fc7b8f93163152667ed40a0c4e55182dd1010853dfd0bf8c317eccb01\": container with ID starting with e2e1204fc7b8f93163152667ed40a0c4e55182dd1010853dfd0bf8c317eccb01 not found: ID does not exist" Oct 30 21:24:21.791894 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:24:21.791861 5378 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-08df98-dc45f5c6-6hbng"] Oct 30 21:24:21.795340 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:24:21.795314 5378 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-08df98-dc45f5c6-6hbng"] Oct 30 21:24:22.016963 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:24:22.016883 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:24:22.361324 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:24:22.361290 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:24:22.393333 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:24:22.393307 5378 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-f0fe13-7cb54c466b-s9889"] Oct 30 21:24:22.499096 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:24:22.499065 5378 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-f0fe13-7447d4f599-smmk4"] Oct 30 21:24:22.648464 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:24:22.648363 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:24:22.781327 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:24:22.781285 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:24:22.941639 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:24:22.937231 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:24:23.324603 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:24:23.324493 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:24:23.742345 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:24:23.742312 5378 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0acdf79f-8312-4818-a1fb-8e50c7948634" path="/var/lib/kubelet/pods/0acdf79f-8312-4818-a1fb-8e50c7948634/volumes" Oct 30 21:24:25.327894 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:24:25.327858 5378 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-inf-b-639b07/test-model-639b07-kserve-b8485fd9-ph8tq" podUID="e96f3b44-86e7-4389-a90f-e0fa736cffd1" containerName="main" containerID="cri-o://625d0d1e89ac52181a02d4ed91d1426033a5a5abe58ab968521edf2f6cbc969e" gracePeriod=45 Oct 30 21:24:25.585715 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:24:25.585657 5378 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-639b07/test-model-639b07-kserve-b8485fd9-ph8tq" Oct 30 21:24:25.651505 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:24:25.651475 5378 generic.go:356] "Generic (PLEG): container finished" podID="e96f3b44-86e7-4389-a90f-e0fa736cffd1" containerID="625d0d1e89ac52181a02d4ed91d1426033a5a5abe58ab968521edf2f6cbc969e" exitCode=0 Oct 30 21:24:25.651683 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:24:25.651514 5378 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-639b07/test-model-639b07-kserve-b8485fd9-ph8tq" Oct 30 21:24:25.651683 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:24:25.651554 5378 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-639b07/test-model-639b07-kserve-b8485fd9-ph8tq" event={"ID":"e96f3b44-86e7-4389-a90f-e0fa736cffd1","Type":"ContainerDied","Data":"625d0d1e89ac52181a02d4ed91d1426033a5a5abe58ab968521edf2f6cbc969e"} Oct 30 21:24:25.651683 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:24:25.651619 5378 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-639b07/test-model-639b07-kserve-b8485fd9-ph8tq" event={"ID":"e96f3b44-86e7-4389-a90f-e0fa736cffd1","Type":"ContainerDied","Data":"1285a171d01d034023fe4b19e27a0a0bf03e34cb45b8757e7cec45de20574735"} Oct 30 21:24:25.651683 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:24:25.651638 5378 scope.go:122] "RemoveContainer" containerID="625d0d1e89ac52181a02d4ed91d1426033a5a5abe58ab968521edf2f6cbc969e" Oct 30 21:24:25.660383 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:24:25.660367 5378 scope.go:122] "RemoveContainer" containerID="625d0d1e89ac52181a02d4ed91d1426033a5a5abe58ab968521edf2f6cbc969e" Oct 30 21:24:25.660729 ip-10-0-138-149 kubenswrapper[5378]: E1030 21:24:25.660710 5378 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"625d0d1e89ac52181a02d4ed91d1426033a5a5abe58ab968521edf2f6cbc969e\": container with ID starting with 625d0d1e89ac52181a02d4ed91d1426033a5a5abe58ab968521edf2f6cbc969e not found: ID does not exist" containerID="625d0d1e89ac52181a02d4ed91d1426033a5a5abe58ab968521edf2f6cbc969e" Oct 30 21:24:25.660788 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:24:25.660738 5378 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"625d0d1e89ac52181a02d4ed91d1426033a5a5abe58ab968521edf2f6cbc969e"} err="failed to get container status \"625d0d1e89ac52181a02d4ed91d1426033a5a5abe58ab968521edf2f6cbc969e\": rpc error: code = NotFound desc = could not find container \"625d0d1e89ac52181a02d4ed91d1426033a5a5abe58ab968521edf2f6cbc969e\": container with ID starting with 625d0d1e89ac52181a02d4ed91d1426033a5a5abe58ab968521edf2f6cbc969e not found: ID does not exist" Oct 30 21:24:25.670191 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:24:25.670171 5378 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e96f3b44-86e7-4389-a90f-e0fa736cffd1-tls-certs\" (UniqueName: \"kubernetes.io/secret/e96f3b44-86e7-4389-a90f-e0fa736cffd1-tls-certs\") pod \"e96f3b44-86e7-4389-a90f-e0fa736cffd1\" (UID: \"e96f3b44-86e7-4389-a90f-e0fa736cffd1\") " Oct 30 21:24:25.670305 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:24:25.670204 5378 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e96f3b44-86e7-4389-a90f-e0fa736cffd1-kube-api-access-b9gn7\" (UniqueName: \"kubernetes.io/projected/e96f3b44-86e7-4389-a90f-e0fa736cffd1-kube-api-access-b9gn7\") pod \"e96f3b44-86e7-4389-a90f-e0fa736cffd1\" (UID: \"e96f3b44-86e7-4389-a90f-e0fa736cffd1\") " Oct 30 21:24:25.670305 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:24:25.670232 5378 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e96f3b44-86e7-4389-a90f-e0fa736cffd1-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e96f3b44-86e7-4389-a90f-e0fa736cffd1-model-cache\") pod \"e96f3b44-86e7-4389-a90f-e0fa736cffd1\" (UID: \"e96f3b44-86e7-4389-a90f-e0fa736cffd1\") " Oct 30 21:24:25.670305 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:24:25.670265 5378 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e96f3b44-86e7-4389-a90f-e0fa736cffd1-dshm\" (UniqueName: \"kubernetes.io/empty-dir/e96f3b44-86e7-4389-a90f-e0fa736cffd1-dshm\") pod \"e96f3b44-86e7-4389-a90f-e0fa736cffd1\" (UID: \"e96f3b44-86e7-4389-a90f-e0fa736cffd1\") " Oct 30 21:24:25.670305 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:24:25.670284 5378 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e96f3b44-86e7-4389-a90f-e0fa736cffd1-home\" (UniqueName: \"kubernetes.io/empty-dir/e96f3b44-86e7-4389-a90f-e0fa736cffd1-home\") pod \"e96f3b44-86e7-4389-a90f-e0fa736cffd1\" (UID: \"e96f3b44-86e7-4389-a90f-e0fa736cffd1\") " Oct 30 21:24:25.670556 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:24:25.670373 5378 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e96f3b44-86e7-4389-a90f-e0fa736cffd1-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e96f3b44-86e7-4389-a90f-e0fa736cffd1-tmp-dir\") pod \"e96f3b44-86e7-4389-a90f-e0fa736cffd1\" (UID: \"e96f3b44-86e7-4389-a90f-e0fa736cffd1\") " Oct 30 21:24:25.670639 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:24:25.670556 5378 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e96f3b44-86e7-4389-a90f-e0fa736cffd1-model-cache" pod "e96f3b44-86e7-4389-a90f-e0fa736cffd1" (UID: "e96f3b44-86e7-4389-a90f-e0fa736cffd1"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:24:25.670732 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:24:25.670701 5378 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e96f3b44-86e7-4389-a90f-e0fa736cffd1-home" pod "e96f3b44-86e7-4389-a90f-e0fa736cffd1" (UID: "e96f3b44-86e7-4389-a90f-e0fa736cffd1"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:24:25.670732 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:24:25.670720 5378 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e96f3b44-86e7-4389-a90f-e0fa736cffd1-tmp-dir" pod "e96f3b44-86e7-4389-a90f-e0fa736cffd1" (UID: "e96f3b44-86e7-4389-a90f-e0fa736cffd1"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:24:25.670921 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:24:25.670862 5378 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e96f3b44-86e7-4389-a90f-e0fa736cffd1-model-cache\") on node \"ip-10-0-138-149.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:24:25.670921 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:24:25.670880 5378 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e96f3b44-86e7-4389-a90f-e0fa736cffd1-home\") on node \"ip-10-0-138-149.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:24:25.670921 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:24:25.670889 5378 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e96f3b44-86e7-4389-a90f-e0fa736cffd1-tmp-dir\") on node \"ip-10-0-138-149.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:24:25.672601 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:24:25.672575 5378 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e96f3b44-86e7-4389-a90f-e0fa736cffd1-tls-certs" pod "e96f3b44-86e7-4389-a90f-e0fa736cffd1" (UID: "e96f3b44-86e7-4389-a90f-e0fa736cffd1"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:24:25.672601 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:24:25.672587 5378 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e96f3b44-86e7-4389-a90f-e0fa736cffd1-kube-api-access-b9gn7" pod "e96f3b44-86e7-4389-a90f-e0fa736cffd1" (UID: "e96f3b44-86e7-4389-a90f-e0fa736cffd1"). InnerVolumeSpecName "kube-api-access-b9gn7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:24:25.672734 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:24:25.672614 5378 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e96f3b44-86e7-4389-a90f-e0fa736cffd1-dshm" pod "e96f3b44-86e7-4389-a90f-e0fa736cffd1" (UID: "e96f3b44-86e7-4389-a90f-e0fa736cffd1"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:24:25.771865 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:24:25.771839 5378 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e96f3b44-86e7-4389-a90f-e0fa736cffd1-tls-certs\") on node \"ip-10-0-138-149.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:24:25.771865 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:24:25.771863 5378 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b9gn7\" (UniqueName: \"kubernetes.io/projected/e96f3b44-86e7-4389-a90f-e0fa736cffd1-kube-api-access-b9gn7\") on node \"ip-10-0-138-149.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:24:25.772039 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:24:25.771873 5378 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e96f3b44-86e7-4389-a90f-e0fa736cffd1-dshm\") on node \"ip-10-0-138-149.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:24:26.070015 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:24:26.069983 5378 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-639b07/test-model-639b07-kserve-b8485fd9-ph8tq"] Oct 30 21:24:26.077515 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:24:26.077487 5378 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-b-639b07/test-model-639b07-kserve-b8485fd9-ph8tq"] Oct 30 21:24:27.394632 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:24:27.394587 5378 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-a-f0fe13-7cb54c466b-s9889" podUID="97e8a2e3-2092-4e54-8621-c33b9404e163" containerName="payload-processing" containerID="cri-o://93857c45ffc0ff65342d61c5ea09a58d8e1479acabb10b769ea9d46ac713259f" gracePeriod=25 Oct 30 21:24:27.499793 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:24:27.499754 5378 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-f0fe13-7447d4f599-smmk4" podUID="ef431fc4-98f7-4a6f-85c1-5989dc6665f6" containerName="payload-pre-processing" containerID="cri-o://cfa15a4128ca7c145d9e98b59f50ed566c296132662c645d33961cbfe7323012" gracePeriod=25 Oct 30 21:24:27.648004 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:24:27.647951 5378 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-f0fe13-7cb54c466b-s9889" Oct 30 21:24:27.665778 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:24:27.665751 5378 generic.go:356] "Generic (PLEG): container finished" podID="97e8a2e3-2092-4e54-8621-c33b9404e163" containerID="93857c45ffc0ff65342d61c5ea09a58d8e1479acabb10b769ea9d46ac713259f" exitCode=0 Oct 30 21:24:27.665905 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:24:27.665793 5378 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-f0fe13-7cb54c466b-s9889" Oct 30 21:24:27.665905 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:24:27.665793 5378 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-f0fe13-7cb54c466b-s9889" event={"ID":"97e8a2e3-2092-4e54-8621-c33b9404e163","Type":"ContainerDied","Data":"93857c45ffc0ff65342d61c5ea09a58d8e1479acabb10b769ea9d46ac713259f"} Oct 30 21:24:27.666033 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:24:27.665947 5378 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-f0fe13-7cb54c466b-s9889" event={"ID":"97e8a2e3-2092-4e54-8621-c33b9404e163","Type":"ContainerDied","Data":"d3927d574116af81a83b53fa83770ceeb2a995fe549823f83250b66e65c765a6"} Oct 30 21:24:27.666033 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:24:27.665976 5378 scope.go:122] "RemoveContainer" containerID="93857c45ffc0ff65342d61c5ea09a58d8e1479acabb10b769ea9d46ac713259f" Oct 30 21:24:27.675079 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:24:27.675063 5378 scope.go:122] "RemoveContainer" containerID="93857c45ffc0ff65342d61c5ea09a58d8e1479acabb10b769ea9d46ac713259f" Oct 30 21:24:27.675337 ip-10-0-138-149 kubenswrapper[5378]: E1030 21:24:27.675313 5378 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"93857c45ffc0ff65342d61c5ea09a58d8e1479acabb10b769ea9d46ac713259f\": container with ID starting with 93857c45ffc0ff65342d61c5ea09a58d8e1479acabb10b769ea9d46ac713259f not found: ID does not exist" containerID="93857c45ffc0ff65342d61c5ea09a58d8e1479acabb10b769ea9d46ac713259f" Oct 30 21:24:27.675397 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:24:27.675349 5378 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"93857c45ffc0ff65342d61c5ea09a58d8e1479acabb10b769ea9d46ac713259f"} err="failed to get container status \"93857c45ffc0ff65342d61c5ea09a58d8e1479acabb10b769ea9d46ac713259f\": rpc error: code = NotFound desc = could not find container \"93857c45ffc0ff65342d61c5ea09a58d8e1479acabb10b769ea9d46ac713259f\": container with ID starting with 93857c45ffc0ff65342d61c5ea09a58d8e1479acabb10b769ea9d46ac713259f not found: ID does not exist" Oct 30 21:24:27.740795 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:24:27.740772 5378 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e96f3b44-86e7-4389-a90f-e0fa736cffd1" path="/var/lib/kubelet/pods/e96f3b44-86e7-4389-a90f-e0fa736cffd1/volumes" Oct 30 21:24:27.793863 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:24:27.793837 5378 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/97e8a2e3-2092-4e54-8621-c33b9404e163-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/97e8a2e3-2092-4e54-8621-c33b9404e163-plugins-config-volume\") pod \"97e8a2e3-2092-4e54-8621-c33b9404e163\" (UID: \"97e8a2e3-2092-4e54-8621-c33b9404e163\") " Oct 30 21:24:27.793994 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:24:27.793937 5378 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/97e8a2e3-2092-4e54-8621-c33b9404e163-kube-api-access-z5rgn\" (UniqueName: \"kubernetes.io/projected/97e8a2e3-2092-4e54-8621-c33b9404e163-kube-api-access-z5rgn\") pod \"97e8a2e3-2092-4e54-8621-c33b9404e163\" (UID: \"97e8a2e3-2092-4e54-8621-c33b9404e163\") " Oct 30 21:24:27.794660 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:24:27.794242 5378 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/97e8a2e3-2092-4e54-8621-c33b9404e163-plugins-config-volume" pod "97e8a2e3-2092-4e54-8621-c33b9404e163" (UID: "97e8a2e3-2092-4e54-8621-c33b9404e163"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:24:27.795022 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:24:27.794996 5378 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/97e8a2e3-2092-4e54-8621-c33b9404e163-plugins-config-volume\") on node \"ip-10-0-138-149.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:24:27.796716 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:24:27.796694 5378 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/97e8a2e3-2092-4e54-8621-c33b9404e163-kube-api-access-z5rgn" pod "97e8a2e3-2092-4e54-8621-c33b9404e163" (UID: "97e8a2e3-2092-4e54-8621-c33b9404e163"). InnerVolumeSpecName "kube-api-access-z5rgn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:24:27.853017 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:24:27.852991 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:24:27.896492 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:24:27.896469 5378 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z5rgn\" (UniqueName: \"kubernetes.io/projected/97e8a2e3-2092-4e54-8621-c33b9404e163-kube-api-access-z5rgn\") on node \"ip-10-0-138-149.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:24:28.087291 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:24:28.087260 5378 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-f0fe13-7cb54c466b-s9889"] Oct 30 21:24:28.091206 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:24:28.091184 5378 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-f0fe13-7cb54c466b-s9889"] Oct 30 21:24:29.405398 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:24:29.405364 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:24:29.555892 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:24:29.555861 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:24:29.741607 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:24:29.741518 5378 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="97e8a2e3-2092-4e54-8621-c33b9404e163" path="/var/lib/kubelet/pods/97e8a2e3-2092-4e54-8621-c33b9404e163/volumes" Oct 30 21:24:29.753958 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:24:29.753929 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:24:30.249594 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:24:30.249503 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:24:31.692715 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:24:31.692673 5378 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-f0fe13-7447d4f599-smmk4" podUID="ef431fc4-98f7-4a6f-85c1-5989dc6665f6" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 21:24:32.206527 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:24:32.206491 5378 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-f0fe13-588d6cb567-29spw"] Oct 30 21:24:32.206785 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:24:32.206741 5378 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-f0fe13-588d6cb567-29spw" podUID="d251ff08-e87c-4819-a8f9-4cc702bc75ac" containerName="maas-api" containerID="cri-o://824f1f814cf968c0d7a520d0c995fc6c051f513e9e35cf17103d6ff8f7fadf20" gracePeriod=30 Oct 30 21:24:32.294913 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:24:32.294880 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:24:32.477900 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:24:32.477877 5378 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-f0fe13-588d6cb567-29spw" Oct 30 21:24:32.553366 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:24:32.553337 5378 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d251ff08-e87c-4819-a8f9-4cc702bc75ac-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d251ff08-e87c-4819-a8f9-4cc702bc75ac-maas-api-tls\") pod \"d251ff08-e87c-4819-a8f9-4cc702bc75ac\" (UID: \"d251ff08-e87c-4819-a8f9-4cc702bc75ac\") " Oct 30 21:24:32.553505 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:24:32.553429 5378 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d251ff08-e87c-4819-a8f9-4cc702bc75ac-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d251ff08-e87c-4819-a8f9-4cc702bc75ac-openshift-service-ca\") pod \"d251ff08-e87c-4819-a8f9-4cc702bc75ac\" (UID: \"d251ff08-e87c-4819-a8f9-4cc702bc75ac\") " Oct 30 21:24:32.553505 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:24:32.553475 5378 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d251ff08-e87c-4819-a8f9-4cc702bc75ac-metrics-certs\" (UniqueName: \"kubernetes.io/secret/d251ff08-e87c-4819-a8f9-4cc702bc75ac-metrics-certs\") pod \"d251ff08-e87c-4819-a8f9-4cc702bc75ac\" (UID: \"d251ff08-e87c-4819-a8f9-4cc702bc75ac\") " Oct 30 21:24:32.553619 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:24:32.553528 5378 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d251ff08-e87c-4819-a8f9-4cc702bc75ac-kube-api-access-2zh4d\" (UniqueName: \"kubernetes.io/projected/d251ff08-e87c-4819-a8f9-4cc702bc75ac-kube-api-access-2zh4d\") pod \"d251ff08-e87c-4819-a8f9-4cc702bc75ac\" (UID: \"d251ff08-e87c-4819-a8f9-4cc702bc75ac\") " Oct 30 21:24:32.553822 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:24:32.553798 5378 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d251ff08-e87c-4819-a8f9-4cc702bc75ac-openshift-service-ca" pod "d251ff08-e87c-4819-a8f9-4cc702bc75ac" (UID: "d251ff08-e87c-4819-a8f9-4cc702bc75ac"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:24:32.553930 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:24:32.553915 5378 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d251ff08-e87c-4819-a8f9-4cc702bc75ac-openshift-service-ca\") on node \"ip-10-0-138-149.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:24:32.555528 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:24:32.555500 5378 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d251ff08-e87c-4819-a8f9-4cc702bc75ac-metrics-certs" pod "d251ff08-e87c-4819-a8f9-4cc702bc75ac" (UID: "d251ff08-e87c-4819-a8f9-4cc702bc75ac"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:24:32.555728 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:24:32.555613 5378 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d251ff08-e87c-4819-a8f9-4cc702bc75ac-maas-api-tls" pod "d251ff08-e87c-4819-a8f9-4cc702bc75ac" (UID: "d251ff08-e87c-4819-a8f9-4cc702bc75ac"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:24:32.555728 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:24:32.555633 5378 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d251ff08-e87c-4819-a8f9-4cc702bc75ac-kube-api-access-2zh4d" pod "d251ff08-e87c-4819-a8f9-4cc702bc75ac" (UID: "d251ff08-e87c-4819-a8f9-4cc702bc75ac"). InnerVolumeSpecName "kube-api-access-2zh4d". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:24:32.655425 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:24:32.655384 5378 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d251ff08-e87c-4819-a8f9-4cc702bc75ac-maas-api-tls\") on node \"ip-10-0-138-149.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:24:32.655425 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:24:32.655417 5378 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d251ff08-e87c-4819-a8f9-4cc702bc75ac-metrics-certs\") on node \"ip-10-0-138-149.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:24:32.655425 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:24:32.655429 5378 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2zh4d\" (UniqueName: \"kubernetes.io/projected/d251ff08-e87c-4819-a8f9-4cc702bc75ac-kube-api-access-2zh4d\") on node \"ip-10-0-138-149.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:24:32.694048 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:24:32.694013 5378 generic.go:356] "Generic (PLEG): container finished" podID="d251ff08-e87c-4819-a8f9-4cc702bc75ac" containerID="824f1f814cf968c0d7a520d0c995fc6c051f513e9e35cf17103d6ff8f7fadf20" exitCode=0 Oct 30 21:24:32.694489 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:24:32.694064 5378 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-f0fe13-588d6cb567-29spw" Oct 30 21:24:32.694489 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:24:32.694096 5378 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-f0fe13-588d6cb567-29spw" event={"ID":"d251ff08-e87c-4819-a8f9-4cc702bc75ac","Type":"ContainerDied","Data":"824f1f814cf968c0d7a520d0c995fc6c051f513e9e35cf17103d6ff8f7fadf20"} Oct 30 21:24:32.694489 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:24:32.694152 5378 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-f0fe13-588d6cb567-29spw" event={"ID":"d251ff08-e87c-4819-a8f9-4cc702bc75ac","Type":"ContainerDied","Data":"3172122db6fc756718aff5d5d5d5b43b516cbf92b0c5c5f3bc14854c3e7201d2"} Oct 30 21:24:32.694489 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:24:32.694189 5378 scope.go:122] "RemoveContainer" containerID="824f1f814cf968c0d7a520d0c995fc6c051f513e9e35cf17103d6ff8f7fadf20" Oct 30 21:24:32.703943 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:24:32.703927 5378 scope.go:122] "RemoveContainer" containerID="824f1f814cf968c0d7a520d0c995fc6c051f513e9e35cf17103d6ff8f7fadf20" Oct 30 21:24:32.704177 ip-10-0-138-149 kubenswrapper[5378]: E1030 21:24:32.704160 5378 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"824f1f814cf968c0d7a520d0c995fc6c051f513e9e35cf17103d6ff8f7fadf20\": container with ID starting with 824f1f814cf968c0d7a520d0c995fc6c051f513e9e35cf17103d6ff8f7fadf20 not found: ID does not exist" containerID="824f1f814cf968c0d7a520d0c995fc6c051f513e9e35cf17103d6ff8f7fadf20" Oct 30 21:24:32.704248 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:24:32.704182 5378 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"824f1f814cf968c0d7a520d0c995fc6c051f513e9e35cf17103d6ff8f7fadf20"} err="failed to get container status \"824f1f814cf968c0d7a520d0c995fc6c051f513e9e35cf17103d6ff8f7fadf20\": rpc error: code = NotFound desc = could not find container \"824f1f814cf968c0d7a520d0c995fc6c051f513e9e35cf17103d6ff8f7fadf20\": container with ID starting with 824f1f814cf968c0d7a520d0c995fc6c051f513e9e35cf17103d6ff8f7fadf20 not found: ID does not exist" Oct 30 21:24:32.860166 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:24:32.860136 5378 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-f0fe13-588d6cb567-29spw"] Oct 30 21:24:32.863749 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:24:32.863723 5378 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-f0fe13-588d6cb567-29spw"] Oct 30 21:24:33.741428 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:24:33.741386 5378 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d251ff08-e87c-4819-a8f9-4cc702bc75ac" path="/var/lib/kubelet/pods/d251ff08-e87c-4819-a8f9-4cc702bc75ac/volumes" Oct 30 21:24:33.914670 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:24:33.914636 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:24:34.197051 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:24:34.197010 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:24:34.337402 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:24:34.337366 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:24:34.822202 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:24:34.822131 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:24:35.358957 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:24:35.358922 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:24:38.114873 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:24:38.114835 5378 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-r9g59"] Oct 30 21:24:38.736930 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:24:38.736895 5378 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-769df7846c-swlpw"] Oct 30 21:24:38.737122 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:24:38.737101 5378 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-swlpw" podUID="c5b6e9d2-ca83-4956-819e-d7ec7997acd8" containerName="istio-proxy" containerID="cri-o://1835c8de6c7c1ff14bbea419facdc67b74fc50cb023b3e6fea23f4d4a6d80c79" gracePeriod=30 Oct 30 21:24:39.467786 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:24:39.467755 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:24:40.647746 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:24:40.647707 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:24:40.768670 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:24:40.768636 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:24:41.003602 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:24:41.003487 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:24:41.147362 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:24:41.147331 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:24:41.692833 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:24:41.692798 5378 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-f0fe13-7447d4f599-smmk4" podUID="ef431fc4-98f7-4a6f-85c1-5989dc6665f6" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 21:24:43.558250 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:24:43.558219 5378 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w3-682999-69c9476f4b-qwh92"] Oct 30 21:24:43.915768 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:24:43.915737 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:24:44.045224 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:24:44.045197 5378 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-swlpw" Oct 30 21:24:44.053988 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:24:44.053947 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:24:44.191526 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:24:44.191448 5378 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c5b6e9d2-ca83-4956-819e-d7ec7997acd8-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c5b6e9d2-ca83-4956-819e-d7ec7997acd8-workload-socket\") pod \"c5b6e9d2-ca83-4956-819e-d7ec7997acd8\" (UID: \"c5b6e9d2-ca83-4956-819e-d7ec7997acd8\") " Oct 30 21:24:44.191526 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:24:44.191492 5378 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c5b6e9d2-ca83-4956-819e-d7ec7997acd8-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c5b6e9d2-ca83-4956-819e-d7ec7997acd8-workload-certs\") pod \"c5b6e9d2-ca83-4956-819e-d7ec7997acd8\" (UID: \"c5b6e9d2-ca83-4956-819e-d7ec7997acd8\") " Oct 30 21:24:44.191794 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:24:44.191540 5378 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c5b6e9d2-ca83-4956-819e-d7ec7997acd8-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c5b6e9d2-ca83-4956-819e-d7ec7997acd8-credential-socket\") pod \"c5b6e9d2-ca83-4956-819e-d7ec7997acd8\" (UID: \"c5b6e9d2-ca83-4956-819e-d7ec7997acd8\") " Oct 30 21:24:44.191794 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:24:44.191612 5378 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c5b6e9d2-ca83-4956-819e-d7ec7997acd8-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c5b6e9d2-ca83-4956-819e-d7ec7997acd8-istio-data\") pod \"c5b6e9d2-ca83-4956-819e-d7ec7997acd8\" (UID: \"c5b6e9d2-ca83-4956-819e-d7ec7997acd8\") " Oct 30 21:24:44.191794 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:24:44.191702 5378 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/c5b6e9d2-ca83-4956-819e-d7ec7997acd8-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c5b6e9d2-ca83-4956-819e-d7ec7997acd8-istio-podinfo\") pod \"c5b6e9d2-ca83-4956-819e-d7ec7997acd8\" (UID: \"c5b6e9d2-ca83-4956-819e-d7ec7997acd8\") " Oct 30 21:24:44.191794 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:24:44.191765 5378 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c5b6e9d2-ca83-4956-819e-d7ec7997acd8-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c5b6e9d2-ca83-4956-819e-d7ec7997acd8-istio-envoy\") pod \"c5b6e9d2-ca83-4956-819e-d7ec7997acd8\" (UID: \"c5b6e9d2-ca83-4956-819e-d7ec7997acd8\") " Oct 30 21:24:44.191794 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:24:44.191776 5378 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c5b6e9d2-ca83-4956-819e-d7ec7997acd8-workload-socket" pod "c5b6e9d2-ca83-4956-819e-d7ec7997acd8" (UID: "c5b6e9d2-ca83-4956-819e-d7ec7997acd8"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:24:44.192087 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:24:44.191854 5378 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c5b6e9d2-ca83-4956-819e-d7ec7997acd8-istio-token\" (UniqueName: \"kubernetes.io/projected/c5b6e9d2-ca83-4956-819e-d7ec7997acd8-istio-token\") pod \"c5b6e9d2-ca83-4956-819e-d7ec7997acd8\" (UID: \"c5b6e9d2-ca83-4956-819e-d7ec7997acd8\") " Oct 30 21:24:44.192087 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:24:44.191887 5378 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c5b6e9d2-ca83-4956-819e-d7ec7997acd8-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c5b6e9d2-ca83-4956-819e-d7ec7997acd8-istiod-ca-cert\") pod \"c5b6e9d2-ca83-4956-819e-d7ec7997acd8\" (UID: \"c5b6e9d2-ca83-4956-819e-d7ec7997acd8\") " Oct 30 21:24:44.192087 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:24:44.191900 5378 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c5b6e9d2-ca83-4956-819e-d7ec7997acd8-workload-certs" pod "c5b6e9d2-ca83-4956-819e-d7ec7997acd8" (UID: "c5b6e9d2-ca83-4956-819e-d7ec7997acd8"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:24:44.192087 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:24:44.191924 5378 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c5b6e9d2-ca83-4956-819e-d7ec7997acd8-kube-api-access-cwpcg\" (UniqueName: \"kubernetes.io/projected/c5b6e9d2-ca83-4956-819e-d7ec7997acd8-kube-api-access-cwpcg\") pod \"c5b6e9d2-ca83-4956-819e-d7ec7997acd8\" (UID: \"c5b6e9d2-ca83-4956-819e-d7ec7997acd8\") " Oct 30 21:24:44.192087 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:24:44.191936 5378 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c5b6e9d2-ca83-4956-819e-d7ec7997acd8-credential-socket" pod "c5b6e9d2-ca83-4956-819e-d7ec7997acd8" (UID: "c5b6e9d2-ca83-4956-819e-d7ec7997acd8"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:24:44.192460 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:24:44.192123 5378 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c5b6e9d2-ca83-4956-819e-d7ec7997acd8-istio-data" pod "c5b6e9d2-ca83-4956-819e-d7ec7997acd8" (UID: "c5b6e9d2-ca83-4956-819e-d7ec7997acd8"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:24:44.192460 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:24:44.192436 5378 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c5b6e9d2-ca83-4956-819e-d7ec7997acd8-workload-socket\") on node \"ip-10-0-138-149.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:24:44.192460 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:24:44.192458 5378 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c5b6e9d2-ca83-4956-819e-d7ec7997acd8-workload-certs\") on node \"ip-10-0-138-149.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:24:44.192650 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:24:44.192472 5378 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c5b6e9d2-ca83-4956-819e-d7ec7997acd8-credential-socket\") on node \"ip-10-0-138-149.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:24:44.192650 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:24:44.192487 5378 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c5b6e9d2-ca83-4956-819e-d7ec7997acd8-istio-data\") on node \"ip-10-0-138-149.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:24:44.192650 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:24:44.192489 5378 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c5b6e9d2-ca83-4956-819e-d7ec7997acd8-istiod-ca-cert" pod "c5b6e9d2-ca83-4956-819e-d7ec7997acd8" (UID: "c5b6e9d2-ca83-4956-819e-d7ec7997acd8"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:24:44.194292 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:24:44.194261 5378 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c5b6e9d2-ca83-4956-819e-d7ec7997acd8-istio-token" pod "c5b6e9d2-ca83-4956-819e-d7ec7997acd8" (UID: "c5b6e9d2-ca83-4956-819e-d7ec7997acd8"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:24:44.194553 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:24:44.194531 5378 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c5b6e9d2-ca83-4956-819e-d7ec7997acd8-kube-api-access-cwpcg" pod "c5b6e9d2-ca83-4956-819e-d7ec7997acd8" (UID: "c5b6e9d2-ca83-4956-819e-d7ec7997acd8"). InnerVolumeSpecName "kube-api-access-cwpcg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:24:44.194695 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:24:44.194548 5378 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c5b6e9d2-ca83-4956-819e-d7ec7997acd8-istio-envoy" pod "c5b6e9d2-ca83-4956-819e-d7ec7997acd8" (UID: "c5b6e9d2-ca83-4956-819e-d7ec7997acd8"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:24:44.196885 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:24:44.196862 5378 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c5b6e9d2-ca83-4956-819e-d7ec7997acd8-istio-podinfo" pod "c5b6e9d2-ca83-4956-819e-d7ec7997acd8" (UID: "c5b6e9d2-ca83-4956-819e-d7ec7997acd8"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 21:24:44.293798 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:24:44.293763 5378 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c5b6e9d2-ca83-4956-819e-d7ec7997acd8-istio-podinfo\") on node \"ip-10-0-138-149.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:24:44.293798 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:24:44.293790 5378 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c5b6e9d2-ca83-4956-819e-d7ec7997acd8-istio-envoy\") on node \"ip-10-0-138-149.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:24:44.293798 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:24:44.293800 5378 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c5b6e9d2-ca83-4956-819e-d7ec7997acd8-istio-token\") on node \"ip-10-0-138-149.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:24:44.294038 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:24:44.293808 5378 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c5b6e9d2-ca83-4956-819e-d7ec7997acd8-istiod-ca-cert\") on node \"ip-10-0-138-149.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:24:44.294038 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:24:44.293818 5378 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cwpcg\" (UniqueName: \"kubernetes.io/projected/c5b6e9d2-ca83-4956-819e-d7ec7997acd8-kube-api-access-cwpcg\") on node \"ip-10-0-138-149.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:24:44.649460 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:24:44.649427 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:24:44.758213 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:24:44.758180 5378 generic.go:356] "Generic (PLEG): container finished" podID="c5b6e9d2-ca83-4956-819e-d7ec7997acd8" containerID="1835c8de6c7c1ff14bbea419facdc67b74fc50cb023b3e6fea23f4d4a6d80c79" exitCode=0 Oct 30 21:24:44.758405 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:24:44.758244 5378 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-swlpw" Oct 30 21:24:44.758405 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:24:44.758227 5378 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-swlpw" event={"ID":"c5b6e9d2-ca83-4956-819e-d7ec7997acd8","Type":"ContainerDied","Data":"1835c8de6c7c1ff14bbea419facdc67b74fc50cb023b3e6fea23f4d4a6d80c79"} Oct 30 21:24:44.758405 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:24:44.758388 5378 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-swlpw" event={"ID":"c5b6e9d2-ca83-4956-819e-d7ec7997acd8","Type":"ContainerDied","Data":"5acba50e417e6ac10721947d1cbcd5bf2bc0771ba58def9b3fa4112c65fa15c8"} Oct 30 21:24:44.758618 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:24:44.758419 5378 scope.go:122] "RemoveContainer" containerID="1835c8de6c7c1ff14bbea419facdc67b74fc50cb023b3e6fea23f4d4a6d80c79" Oct 30 21:24:44.769014 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:24:44.768995 5378 scope.go:122] "RemoveContainer" containerID="1835c8de6c7c1ff14bbea419facdc67b74fc50cb023b3e6fea23f4d4a6d80c79" Oct 30 21:24:44.769328 ip-10-0-138-149 kubenswrapper[5378]: E1030 21:24:44.769301 5378 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1835c8de6c7c1ff14bbea419facdc67b74fc50cb023b3e6fea23f4d4a6d80c79\": container with ID starting with 1835c8de6c7c1ff14bbea419facdc67b74fc50cb023b3e6fea23f4d4a6d80c79 not found: ID does not exist" containerID="1835c8de6c7c1ff14bbea419facdc67b74fc50cb023b3e6fea23f4d4a6d80c79" Oct 30 21:24:44.769401 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:24:44.769338 5378 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1835c8de6c7c1ff14bbea419facdc67b74fc50cb023b3e6fea23f4d4a6d80c79"} err="failed to get container status \"1835c8de6c7c1ff14bbea419facdc67b74fc50cb023b3e6fea23f4d4a6d80c79\": rpc error: code = NotFound desc = could not find container \"1835c8de6c7c1ff14bbea419facdc67b74fc50cb023b3e6fea23f4d4a6d80c79\": container with ID starting with 1835c8de6c7c1ff14bbea419facdc67b74fc50cb023b3e6fea23f4d4a6d80c79 not found: ID does not exist" Oct 30 21:24:44.926125 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:24:44.925976 5378 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-769df7846c-swlpw"] Oct 30 21:24:44.929136 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:24:44.929110 5378 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-769df7846c-swlpw"] Oct 30 21:24:45.741462 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:24:45.741429 5378 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c5b6e9d2-ca83-4956-819e-d7ec7997acd8" path="/var/lib/kubelet/pods/c5b6e9d2-ca83-4956-819e-d7ec7997acd8/volumes" Oct 30 21:24:47.762000 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:24:47.761948 5378 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-f0fe13-7447d4f599-smmk4" Oct 30 21:24:47.775810 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:24:47.775785 5378 generic.go:356] "Generic (PLEG): container finished" podID="ef431fc4-98f7-4a6f-85c1-5989dc6665f6" containerID="cfa15a4128ca7c145d9e98b59f50ed566c296132662c645d33961cbfe7323012" exitCode=0 Oct 30 21:24:47.775928 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:24:47.775819 5378 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-f0fe13-7447d4f599-smmk4" Oct 30 21:24:47.775928 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:24:47.775876 5378 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-f0fe13-7447d4f599-smmk4" event={"ID":"ef431fc4-98f7-4a6f-85c1-5989dc6665f6","Type":"ContainerDied","Data":"cfa15a4128ca7c145d9e98b59f50ed566c296132662c645d33961cbfe7323012"} Oct 30 21:24:47.775928 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:24:47.775926 5378 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-f0fe13-7447d4f599-smmk4" event={"ID":"ef431fc4-98f7-4a6f-85c1-5989dc6665f6","Type":"ContainerDied","Data":"538f498ed66d6d1a568281a092abf914f6d782248b917cd9c70841b78bdf35ff"} Oct 30 21:24:47.776041 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:24:47.775953 5378 scope.go:122] "RemoveContainer" containerID="cfa15a4128ca7c145d9e98b59f50ed566c296132662c645d33961cbfe7323012" Oct 30 21:24:47.785012 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:24:47.784983 5378 scope.go:122] "RemoveContainer" containerID="cfa15a4128ca7c145d9e98b59f50ed566c296132662c645d33961cbfe7323012" Oct 30 21:24:47.785260 ip-10-0-138-149 kubenswrapper[5378]: E1030 21:24:47.785243 5378 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cfa15a4128ca7c145d9e98b59f50ed566c296132662c645d33961cbfe7323012\": container with ID starting with cfa15a4128ca7c145d9e98b59f50ed566c296132662c645d33961cbfe7323012 not found: ID does not exist" containerID="cfa15a4128ca7c145d9e98b59f50ed566c296132662c645d33961cbfe7323012" Oct 30 21:24:47.785322 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:24:47.785268 5378 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cfa15a4128ca7c145d9e98b59f50ed566c296132662c645d33961cbfe7323012"} err="failed to get container status \"cfa15a4128ca7c145d9e98b59f50ed566c296132662c645d33961cbfe7323012\": rpc error: code = NotFound desc = could not find container \"cfa15a4128ca7c145d9e98b59f50ed566c296132662c645d33961cbfe7323012\": container with ID starting with cfa15a4128ca7c145d9e98b59f50ed566c296132662c645d33961cbfe7323012 not found: ID does not exist" Oct 30 21:24:47.934390 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:24:47.934352 5378 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ef431fc4-98f7-4a6f-85c1-5989dc6665f6-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ef431fc4-98f7-4a6f-85c1-5989dc6665f6-plugins-config-volume\") pod \"ef431fc4-98f7-4a6f-85c1-5989dc6665f6\" (UID: \"ef431fc4-98f7-4a6f-85c1-5989dc6665f6\") " Oct 30 21:24:47.934550 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:24:47.934509 5378 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ef431fc4-98f7-4a6f-85c1-5989dc6665f6-kube-api-access-dpwbr\" (UniqueName: \"kubernetes.io/projected/ef431fc4-98f7-4a6f-85c1-5989dc6665f6-kube-api-access-dpwbr\") pod \"ef431fc4-98f7-4a6f-85c1-5989dc6665f6\" (UID: \"ef431fc4-98f7-4a6f-85c1-5989dc6665f6\") " Oct 30 21:24:47.934762 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:24:47.934740 5378 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ef431fc4-98f7-4a6f-85c1-5989dc6665f6-plugins-config-volume" pod "ef431fc4-98f7-4a6f-85c1-5989dc6665f6" (UID: "ef431fc4-98f7-4a6f-85c1-5989dc6665f6"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:24:47.934868 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:24:47.934848 5378 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ef431fc4-98f7-4a6f-85c1-5989dc6665f6-plugins-config-volume\") on node \"ip-10-0-138-149.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:24:47.936555 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:24:47.936533 5378 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ef431fc4-98f7-4a6f-85c1-5989dc6665f6-kube-api-access-dpwbr" pod "ef431fc4-98f7-4a6f-85c1-5989dc6665f6" (UID: "ef431fc4-98f7-4a6f-85c1-5989dc6665f6"). InnerVolumeSpecName "kube-api-access-dpwbr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:24:48.036294 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:24:48.036204 5378 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dpwbr\" (UniqueName: \"kubernetes.io/projected/ef431fc4-98f7-4a6f-85c1-5989dc6665f6-kube-api-access-dpwbr\") on node \"ip-10-0-138-149.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:24:48.197546 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:24:48.197510 5378 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-f0fe13-7447d4f599-smmk4"] Oct 30 21:24:48.201958 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:24:48.201932 5378 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-f0fe13-7447d4f599-smmk4"] Oct 30 21:24:48.558828 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:24:48.558790 5378 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w3-682999-69c9476f4b-qwh92" podUID="6351b682-cca8-4a28-bd5f-9cb3fd38085b" containerName="payload-processing" containerID="cri-o://90865cd40bf7485e78d246a2c67dff0b0be5544b98fb6da93b2f2f0bb08c7727" gracePeriod=25 Oct 30 21:24:48.836695 ip-10-0-138-149 kubenswrapper[5378]: E1030 21:24:48.836613 5378 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q6mx7" containerName="extractor" Oct 30 21:24:49.569722 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:24:49.569689 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:24:49.741280 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:24:49.741248 5378 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ef431fc4-98f7-4a6f-85c1-5989dc6665f6" path="/var/lib/kubelet/pods/ef431fc4-98f7-4a6f-85c1-5989dc6665f6/volumes" Oct 30 21:24:51.847670 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:24:51.847639 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:24:53.528366 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:24:53.528325 5378 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w3-682999-69c9476f4b-qwh92" podUID="6351b682-cca8-4a28-bd5f-9cb3fd38085b" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 21:24:53.694633 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:24:53.694546 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:24:53.821600 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:24:53.821487 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:24:54.462974 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:24:54.462944 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:24:59.068839 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:24:59.068808 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:24:59.278520 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:24:59.278486 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:24:59.456434 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:24:59.456400 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:25:03.528544 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:25:03.528501 5378 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w3-682999-69c9476f4b-qwh92" podUID="6351b682-cca8-4a28-bd5f-9cb3fd38085b" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 21:25:04.160770 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:25:04.160744 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:25:05.297371 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:25:05.297337 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:25:05.424154 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:25:05.424119 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:25:05.664518 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:25:05.664476 5378 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-d1b889bd-openshift-default-566bd56c85-dvzs4"] Oct 30 21:25:05.664804 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:25:05.664774 5378 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-d1b889bd-openshift-default-566bd56c85-dvzs4" podUID="d848cad5-c34e-4841-a2a0-e3aab4a69029" containerName="istio-proxy" containerID="cri-o://3b9304456d83f1af5b7b823de73c14a63e17a08a5504aabb0d954f67c4de92ba" gracePeriod=30 Oct 30 21:25:05.678934 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:25:05.678910 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:25:05.679813 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:25:05.679793 5378 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-d1b889bd-openshift-default-566bd56c85-vvvjg"] Oct 30 21:25:05.679977 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:25:05.679959 5378 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-d1b889bd-openshift-default-566bd56c85-vvvjg" podUID="8fdfc4fa-d488-4267-bfb0-113448a60fc2" containerName="istio-proxy" containerID="cri-o://113d795cb55b47f8a9085116bac2ee5acc5574596f0bb731e74e78f2c6e73b21" gracePeriod=30 Oct 30 21:25:06.155560 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:25:06.155523 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:25:08.821325 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:25:08.821266 5378 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w3-682999-69c9476f4b-qwh92" Oct 30 21:25:08.882444 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:25:08.882417 5378 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6351b682-cca8-4a28-bd5f-9cb3fd38085b-kube-api-access-984sn\" (UniqueName: \"kubernetes.io/projected/6351b682-cca8-4a28-bd5f-9cb3fd38085b-kube-api-access-984sn\") pod \"6351b682-cca8-4a28-bd5f-9cb3fd38085b\" (UID: \"6351b682-cca8-4a28-bd5f-9cb3fd38085b\") " Oct 30 21:25:08.882633 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:25:08.882514 5378 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6351b682-cca8-4a28-bd5f-9cb3fd38085b-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6351b682-cca8-4a28-bd5f-9cb3fd38085b-plugins-config-volume\") pod \"6351b682-cca8-4a28-bd5f-9cb3fd38085b\" (UID: \"6351b682-cca8-4a28-bd5f-9cb3fd38085b\") " Oct 30 21:25:08.882928 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:25:08.882884 5378 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6351b682-cca8-4a28-bd5f-9cb3fd38085b-plugins-config-volume" pod "6351b682-cca8-4a28-bd5f-9cb3fd38085b" (UID: "6351b682-cca8-4a28-bd5f-9cb3fd38085b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:25:08.884818 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:25:08.884789 5378 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6351b682-cca8-4a28-bd5f-9cb3fd38085b-kube-api-access-984sn" pod "6351b682-cca8-4a28-bd5f-9cb3fd38085b" (UID: "6351b682-cca8-4a28-bd5f-9cb3fd38085b"). InnerVolumeSpecName "kube-api-access-984sn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:25:08.927627 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:25:08.927599 5378 generic.go:356] "Generic (PLEG): container finished" podID="6351b682-cca8-4a28-bd5f-9cb3fd38085b" containerID="90865cd40bf7485e78d246a2c67dff0b0be5544b98fb6da93b2f2f0bb08c7727" exitCode=0 Oct 30 21:25:08.927761 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:25:08.927665 5378 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w3-682999-69c9476f4b-qwh92" event={"ID":"6351b682-cca8-4a28-bd5f-9cb3fd38085b","Type":"ContainerDied","Data":"90865cd40bf7485e78d246a2c67dff0b0be5544b98fb6da93b2f2f0bb08c7727"} Oct 30 21:25:08.927761 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:25:08.927687 5378 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w3-682999-69c9476f4b-qwh92" Oct 30 21:25:08.927761 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:25:08.927700 5378 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w3-682999-69c9476f4b-qwh92" event={"ID":"6351b682-cca8-4a28-bd5f-9cb3fd38085b","Type":"ContainerDied","Data":"ce1aa027617321004034438c2f0b0e52ca3dc0e2b6e598bedd61bc654ff4df02"} Oct 30 21:25:08.927761 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:25:08.927728 5378 scope.go:122] "RemoveContainer" containerID="90865cd40bf7485e78d246a2c67dff0b0be5544b98fb6da93b2f2f0bb08c7727" Oct 30 21:25:08.937731 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:25:08.937701 5378 scope.go:122] "RemoveContainer" containerID="90865cd40bf7485e78d246a2c67dff0b0be5544b98fb6da93b2f2f0bb08c7727" Oct 30 21:25:08.939081 ip-10-0-138-149 kubenswrapper[5378]: E1030 21:25:08.939046 5378 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"90865cd40bf7485e78d246a2c67dff0b0be5544b98fb6da93b2f2f0bb08c7727\": container with ID starting with 90865cd40bf7485e78d246a2c67dff0b0be5544b98fb6da93b2f2f0bb08c7727 not found: ID does not exist" containerID="90865cd40bf7485e78d246a2c67dff0b0be5544b98fb6da93b2f2f0bb08c7727" Oct 30 21:25:08.939243 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:25:08.939106 5378 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"90865cd40bf7485e78d246a2c67dff0b0be5544b98fb6da93b2f2f0bb08c7727"} err="failed to get container status \"90865cd40bf7485e78d246a2c67dff0b0be5544b98fb6da93b2f2f0bb08c7727\": rpc error: code = NotFound desc = could not find container \"90865cd40bf7485e78d246a2c67dff0b0be5544b98fb6da93b2f2f0bb08c7727\": container with ID starting with 90865cd40bf7485e78d246a2c67dff0b0be5544b98fb6da93b2f2f0bb08c7727 not found: ID does not exist" Oct 30 21:25:08.983879 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:25:08.983851 5378 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-984sn\" (UniqueName: \"kubernetes.io/projected/6351b682-cca8-4a28-bd5f-9cb3fd38085b-kube-api-access-984sn\") on node \"ip-10-0-138-149.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:25:08.983879 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:25:08.983878 5378 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6351b682-cca8-4a28-bd5f-9cb3fd38085b-plugins-config-volume\") on node \"ip-10-0-138-149.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:25:09.050889 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:25:09.050864 5378 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w3-682999-69c9476f4b-qwh92"] Oct 30 21:25:09.062328 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:25:09.062296 5378 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w3-682999-69c9476f4b-qwh92"] Oct 30 21:25:09.312984 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:25:09.312932 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:25:09.741740 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:25:09.741708 5378 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6351b682-cca8-4a28-bd5f-9cb3fd38085b" path="/var/lib/kubelet/pods/6351b682-cca8-4a28-bd5f-9cb3fd38085b/volumes" Oct 30 21:25:10.937440 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:25:10.937365 5378 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-d1b889bd-openshift-default-566bd56c85-vvvjg" Oct 30 21:25:10.941961 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:25:10.941938 5378 generic.go:356] "Generic (PLEG): container finished" podID="d848cad5-c34e-4841-a2a0-e3aab4a69029" containerID="3b9304456d83f1af5b7b823de73c14a63e17a08a5504aabb0d954f67c4de92ba" exitCode=0 Oct 30 21:25:10.942089 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:25:10.942011 5378 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-d1b889bd-openshift-default-566bd56c85-dvzs4" event={"ID":"d848cad5-c34e-4841-a2a0-e3aab4a69029","Type":"ContainerDied","Data":"3b9304456d83f1af5b7b823de73c14a63e17a08a5504aabb0d954f67c4de92ba"} Oct 30 21:25:10.943368 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:25:10.943342 5378 generic.go:356] "Generic (PLEG): container finished" podID="8fdfc4fa-d488-4267-bfb0-113448a60fc2" containerID="113d795cb55b47f8a9085116bac2ee5acc5574596f0bb731e74e78f2c6e73b21" exitCode=0 Oct 30 21:25:10.943473 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:25:10.943379 5378 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-d1b889bd-openshift-default-566bd56c85-vvvjg" event={"ID":"8fdfc4fa-d488-4267-bfb0-113448a60fc2","Type":"ContainerDied","Data":"113d795cb55b47f8a9085116bac2ee5acc5574596f0bb731e74e78f2c6e73b21"} Oct 30 21:25:10.943473 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:25:10.943389 5378 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-d1b889bd-openshift-default-566bd56c85-vvvjg" Oct 30 21:25:10.943473 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:25:10.943425 5378 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-d1b889bd-openshift-default-566bd56c85-vvvjg" event={"ID":"8fdfc4fa-d488-4267-bfb0-113448a60fc2","Type":"ContainerDied","Data":"76671a323c11e92a6e51dc9c1d16d5c8dc9efb5b4fef8f9b07294e10489b5420"} Oct 30 21:25:10.943473 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:25:10.943454 5378 scope.go:122] "RemoveContainer" containerID="113d795cb55b47f8a9085116bac2ee5acc5574596f0bb731e74e78f2c6e73b21" Oct 30 21:25:10.952395 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:25:10.952378 5378 scope.go:122] "RemoveContainer" containerID="113d795cb55b47f8a9085116bac2ee5acc5574596f0bb731e74e78f2c6e73b21" Oct 30 21:25:10.952703 ip-10-0-138-149 kubenswrapper[5378]: E1030 21:25:10.952676 5378 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"113d795cb55b47f8a9085116bac2ee5acc5574596f0bb731e74e78f2c6e73b21\": container with ID starting with 113d795cb55b47f8a9085116bac2ee5acc5574596f0bb731e74e78f2c6e73b21 not found: ID does not exist" containerID="113d795cb55b47f8a9085116bac2ee5acc5574596f0bb731e74e78f2c6e73b21" Oct 30 21:25:10.952774 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:25:10.952715 5378 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"113d795cb55b47f8a9085116bac2ee5acc5574596f0bb731e74e78f2c6e73b21"} err="failed to get container status \"113d795cb55b47f8a9085116bac2ee5acc5574596f0bb731e74e78f2c6e73b21\": rpc error: code = NotFound desc = could not find container \"113d795cb55b47f8a9085116bac2ee5acc5574596f0bb731e74e78f2c6e73b21\": container with ID starting with 113d795cb55b47f8a9085116bac2ee5acc5574596f0bb731e74e78f2c6e73b21 not found: ID does not exist" Oct 30 21:25:10.966625 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:25:10.966607 5378 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-d1b889bd-openshift-default-566bd56c85-dvzs4" Oct 30 21:25:11.004829 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:25:11.004806 5378 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8fdfc4fa-d488-4267-bfb0-113448a60fc2-kube-api-access-9zsg9\" (UniqueName: \"kubernetes.io/projected/8fdfc4fa-d488-4267-bfb0-113448a60fc2-kube-api-access-9zsg9\") pod \"8fdfc4fa-d488-4267-bfb0-113448a60fc2\" (UID: \"8fdfc4fa-d488-4267-bfb0-113448a60fc2\") " Oct 30 21:25:11.004937 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:25:11.004844 5378 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8fdfc4fa-d488-4267-bfb0-113448a60fc2-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8fdfc4fa-d488-4267-bfb0-113448a60fc2-credential-socket\") pod \"8fdfc4fa-d488-4267-bfb0-113448a60fc2\" (UID: \"8fdfc4fa-d488-4267-bfb0-113448a60fc2\") " Oct 30 21:25:11.004937 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:25:11.004872 5378 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d848cad5-c34e-4841-a2a0-e3aab4a69029-kube-api-access-7kxxn\" (UniqueName: \"kubernetes.io/projected/d848cad5-c34e-4841-a2a0-e3aab4a69029-kube-api-access-7kxxn\") pod \"d848cad5-c34e-4841-a2a0-e3aab4a69029\" (UID: \"d848cad5-c34e-4841-a2a0-e3aab4a69029\") " Oct 30 21:25:11.004937 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:25:11.004900 5378 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8fdfc4fa-d488-4267-bfb0-113448a60fc2-istio-token\" (UniqueName: \"kubernetes.io/projected/8fdfc4fa-d488-4267-bfb0-113448a60fc2-istio-token\") pod \"8fdfc4fa-d488-4267-bfb0-113448a60fc2\" (UID: \"8fdfc4fa-d488-4267-bfb0-113448a60fc2\") " Oct 30 21:25:11.004937 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:25:11.004932 5378 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/8fdfc4fa-d488-4267-bfb0-113448a60fc2-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8fdfc4fa-d488-4267-bfb0-113448a60fc2-istio-podinfo\") pod \"8fdfc4fa-d488-4267-bfb0-113448a60fc2\" (UID: \"8fdfc4fa-d488-4267-bfb0-113448a60fc2\") " Oct 30 21:25:11.005183 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:25:11.004961 5378 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d848cad5-c34e-4841-a2a0-e3aab4a69029-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d848cad5-c34e-4841-a2a0-e3aab4a69029-workload-socket\") pod \"d848cad5-c34e-4841-a2a0-e3aab4a69029\" (UID: \"d848cad5-c34e-4841-a2a0-e3aab4a69029\") " Oct 30 21:25:11.005183 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:25:11.005038 5378 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8fdfc4fa-d488-4267-bfb0-113448a60fc2-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8fdfc4fa-d488-4267-bfb0-113448a60fc2-istio-data\") pod \"8fdfc4fa-d488-4267-bfb0-113448a60fc2\" (UID: \"8fdfc4fa-d488-4267-bfb0-113448a60fc2\") " Oct 30 21:25:11.005183 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:25:11.005070 5378 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8fdfc4fa-d488-4267-bfb0-113448a60fc2-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8fdfc4fa-d488-4267-bfb0-113448a60fc2-workload-certs\") pod \"8fdfc4fa-d488-4267-bfb0-113448a60fc2\" (UID: \"8fdfc4fa-d488-4267-bfb0-113448a60fc2\") " Oct 30 21:25:11.005183 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:25:11.005105 5378 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d848cad5-c34e-4841-a2a0-e3aab4a69029-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d848cad5-c34e-4841-a2a0-e3aab4a69029-istio-data\") pod \"d848cad5-c34e-4841-a2a0-e3aab4a69029\" (UID: \"d848cad5-c34e-4841-a2a0-e3aab4a69029\") " Oct 30 21:25:11.005183 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:25:11.005141 5378 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8fdfc4fa-d488-4267-bfb0-113448a60fc2-credential-socket" pod "8fdfc4fa-d488-4267-bfb0-113448a60fc2" (UID: "8fdfc4fa-d488-4267-bfb0-113448a60fc2"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:25:11.005183 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:25:11.005159 5378 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d848cad5-c34e-4841-a2a0-e3aab4a69029-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d848cad5-c34e-4841-a2a0-e3aab4a69029-workload-certs\") pod \"d848cad5-c34e-4841-a2a0-e3aab4a69029\" (UID: \"d848cad5-c34e-4841-a2a0-e3aab4a69029\") " Oct 30 21:25:11.005524 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:25:11.005255 5378 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8fdfc4fa-d488-4267-bfb0-113448a60fc2-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8fdfc4fa-d488-4267-bfb0-113448a60fc2-workload-socket\") pod \"8fdfc4fa-d488-4267-bfb0-113448a60fc2\" (UID: \"8fdfc4fa-d488-4267-bfb0-113448a60fc2\") " Oct 30 21:25:11.005524 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:25:11.005284 5378 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d848cad5-c34e-4841-a2a0-e3aab4a69029-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d848cad5-c34e-4841-a2a0-e3aab4a69029-istiod-ca-cert\") pod \"d848cad5-c34e-4841-a2a0-e3aab4a69029\" (UID: \"d848cad5-c34e-4841-a2a0-e3aab4a69029\") " Oct 30 21:25:11.005524 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:25:11.005330 5378 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8fdfc4fa-d488-4267-bfb0-113448a60fc2-istio-data" pod "8fdfc4fa-d488-4267-bfb0-113448a60fc2" (UID: "8fdfc4fa-d488-4267-bfb0-113448a60fc2"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:25:11.005524 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:25:11.005337 5378 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d848cad5-c34e-4841-a2a0-e3aab4a69029-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d848cad5-c34e-4841-a2a0-e3aab4a69029-credential-socket\") pod \"d848cad5-c34e-4841-a2a0-e3aab4a69029\" (UID: \"d848cad5-c34e-4841-a2a0-e3aab4a69029\") " Oct 30 21:25:11.005524 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:25:11.005374 5378 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d848cad5-c34e-4841-a2a0-e3aab4a69029-workload-socket" pod "d848cad5-c34e-4841-a2a0-e3aab4a69029" (UID: "d848cad5-c34e-4841-a2a0-e3aab4a69029"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:25:11.005524 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:25:11.005406 5378 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/d848cad5-c34e-4841-a2a0-e3aab4a69029-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d848cad5-c34e-4841-a2a0-e3aab4a69029-istio-podinfo\") pod \"d848cad5-c34e-4841-a2a0-e3aab4a69029\" (UID: \"d848cad5-c34e-4841-a2a0-e3aab4a69029\") " Oct 30 21:25:11.005524 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:25:11.005455 5378 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8fdfc4fa-d488-4267-bfb0-113448a60fc2-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8fdfc4fa-d488-4267-bfb0-113448a60fc2-istiod-ca-cert\") pod \"8fdfc4fa-d488-4267-bfb0-113448a60fc2\" (UID: \"8fdfc4fa-d488-4267-bfb0-113448a60fc2\") " Oct 30 21:25:11.005524 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:25:11.005481 5378 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d848cad5-c34e-4841-a2a0-e3aab4a69029-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d848cad5-c34e-4841-a2a0-e3aab4a69029-istio-envoy\") pod \"d848cad5-c34e-4841-a2a0-e3aab4a69029\" (UID: \"d848cad5-c34e-4841-a2a0-e3aab4a69029\") " Oct 30 21:25:11.005524 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:25:11.005503 5378 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d848cad5-c34e-4841-a2a0-e3aab4a69029-istio-data" pod "d848cad5-c34e-4841-a2a0-e3aab4a69029" (UID: "d848cad5-c34e-4841-a2a0-e3aab4a69029"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:25:11.006037 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:25:11.005539 5378 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8fdfc4fa-d488-4267-bfb0-113448a60fc2-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8fdfc4fa-d488-4267-bfb0-113448a60fc2-istio-envoy\") pod \"8fdfc4fa-d488-4267-bfb0-113448a60fc2\" (UID: \"8fdfc4fa-d488-4267-bfb0-113448a60fc2\") " Oct 30 21:25:11.006037 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:25:11.005548 5378 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d848cad5-c34e-4841-a2a0-e3aab4a69029-credential-socket" pod "d848cad5-c34e-4841-a2a0-e3aab4a69029" (UID: "d848cad5-c34e-4841-a2a0-e3aab4a69029"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:25:11.006037 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:25:11.005605 5378 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8fdfc4fa-d488-4267-bfb0-113448a60fc2-workload-socket" pod "8fdfc4fa-d488-4267-bfb0-113448a60fc2" (UID: "8fdfc4fa-d488-4267-bfb0-113448a60fc2"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:25:11.006037 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:25:11.005625 5378 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d848cad5-c34e-4841-a2a0-e3aab4a69029-istio-token\" (UniqueName: \"kubernetes.io/projected/d848cad5-c34e-4841-a2a0-e3aab4a69029-istio-token\") pod \"d848cad5-c34e-4841-a2a0-e3aab4a69029\" (UID: \"d848cad5-c34e-4841-a2a0-e3aab4a69029\") " Oct 30 21:25:11.006037 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:25:11.005764 5378 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8fdfc4fa-d488-4267-bfb0-113448a60fc2-workload-certs" pod "8fdfc4fa-d488-4267-bfb0-113448a60fc2" (UID: "8fdfc4fa-d488-4267-bfb0-113448a60fc2"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:25:11.006037 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:25:11.005799 5378 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d848cad5-c34e-4841-a2a0-e3aab4a69029-workload-certs" pod "d848cad5-c34e-4841-a2a0-e3aab4a69029" (UID: "d848cad5-c34e-4841-a2a0-e3aab4a69029"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:25:11.006037 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:25:11.005847 5378 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d848cad5-c34e-4841-a2a0-e3aab4a69029-istiod-ca-cert" pod "d848cad5-c34e-4841-a2a0-e3aab4a69029" (UID: "d848cad5-c34e-4841-a2a0-e3aab4a69029"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:25:11.006730 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:25:11.006699 5378 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8fdfc4fa-d488-4267-bfb0-113448a60fc2-istiod-ca-cert" pod "8fdfc4fa-d488-4267-bfb0-113448a60fc2" (UID: "8fdfc4fa-d488-4267-bfb0-113448a60fc2"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:25:11.006905 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:25:11.006881 5378 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8fdfc4fa-d488-4267-bfb0-113448a60fc2-credential-socket\") on node \"ip-10-0-138-149.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:25:11.007024 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:25:11.006908 5378 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d848cad5-c34e-4841-a2a0-e3aab4a69029-workload-socket\") on node \"ip-10-0-138-149.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:25:11.007024 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:25:11.006945 5378 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8fdfc4fa-d488-4267-bfb0-113448a60fc2-istio-data\") on node \"ip-10-0-138-149.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:25:11.007024 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:25:11.006959 5378 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8fdfc4fa-d488-4267-bfb0-113448a60fc2-workload-certs\") on node \"ip-10-0-138-149.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:25:11.007024 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:25:11.006974 5378 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d848cad5-c34e-4841-a2a0-e3aab4a69029-istio-data\") on node \"ip-10-0-138-149.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:25:11.007024 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:25:11.006988 5378 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d848cad5-c34e-4841-a2a0-e3aab4a69029-workload-certs\") on node \"ip-10-0-138-149.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:25:11.007024 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:25:11.007002 5378 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8fdfc4fa-d488-4267-bfb0-113448a60fc2-workload-socket\") on node \"ip-10-0-138-149.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:25:11.007024 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:25:11.007015 5378 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d848cad5-c34e-4841-a2a0-e3aab4a69029-istiod-ca-cert\") on node \"ip-10-0-138-149.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:25:11.007418 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:25:11.007029 5378 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d848cad5-c34e-4841-a2a0-e3aab4a69029-credential-socket\") on node \"ip-10-0-138-149.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:25:11.008190 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:25:11.008161 5378 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/8fdfc4fa-d488-4267-bfb0-113448a60fc2-istio-podinfo" pod "8fdfc4fa-d488-4267-bfb0-113448a60fc2" (UID: "8fdfc4fa-d488-4267-bfb0-113448a60fc2"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 21:25:11.008274 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:25:11.008198 5378 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d848cad5-c34e-4841-a2a0-e3aab4a69029-istio-envoy" pod "d848cad5-c34e-4841-a2a0-e3aab4a69029" (UID: "d848cad5-c34e-4841-a2a0-e3aab4a69029"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:25:11.008274 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:25:11.008243 5378 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d848cad5-c34e-4841-a2a0-e3aab4a69029-kube-api-access-7kxxn" pod "d848cad5-c34e-4841-a2a0-e3aab4a69029" (UID: "d848cad5-c34e-4841-a2a0-e3aab4a69029"). InnerVolumeSpecName "kube-api-access-7kxxn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:25:11.008525 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:25:11.008501 5378 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8fdfc4fa-d488-4267-bfb0-113448a60fc2-istio-envoy" pod "8fdfc4fa-d488-4267-bfb0-113448a60fc2" (UID: "8fdfc4fa-d488-4267-bfb0-113448a60fc2"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:25:11.008654 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:25:11.008531 5378 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d848cad5-c34e-4841-a2a0-e3aab4a69029-istio-podinfo" pod "d848cad5-c34e-4841-a2a0-e3aab4a69029" (UID: "d848cad5-c34e-4841-a2a0-e3aab4a69029"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 21:25:11.009017 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:25:11.008997 5378 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8fdfc4fa-d488-4267-bfb0-113448a60fc2-istio-token" pod "8fdfc4fa-d488-4267-bfb0-113448a60fc2" (UID: "8fdfc4fa-d488-4267-bfb0-113448a60fc2"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:25:11.009322 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:25:11.009291 5378 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d848cad5-c34e-4841-a2a0-e3aab4a69029-istio-token" pod "d848cad5-c34e-4841-a2a0-e3aab4a69029" (UID: "d848cad5-c34e-4841-a2a0-e3aab4a69029"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:25:11.009879 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:25:11.009860 5378 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8fdfc4fa-d488-4267-bfb0-113448a60fc2-kube-api-access-9zsg9" pod "8fdfc4fa-d488-4267-bfb0-113448a60fc2" (UID: "8fdfc4fa-d488-4267-bfb0-113448a60fc2"). InnerVolumeSpecName "kube-api-access-9zsg9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:25:11.108110 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:25:11.108079 5378 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d848cad5-c34e-4841-a2a0-e3aab4a69029-istio-podinfo\") on node \"ip-10-0-138-149.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:25:11.108110 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:25:11.108105 5378 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8fdfc4fa-d488-4267-bfb0-113448a60fc2-istiod-ca-cert\") on node \"ip-10-0-138-149.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:25:11.108110 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:25:11.108114 5378 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d848cad5-c34e-4841-a2a0-e3aab4a69029-istio-envoy\") on node \"ip-10-0-138-149.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:25:11.108451 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:25:11.108124 5378 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8fdfc4fa-d488-4267-bfb0-113448a60fc2-istio-envoy\") on node \"ip-10-0-138-149.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:25:11.108451 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:25:11.108133 5378 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d848cad5-c34e-4841-a2a0-e3aab4a69029-istio-token\") on node \"ip-10-0-138-149.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:25:11.108451 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:25:11.108143 5378 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9zsg9\" (UniqueName: \"kubernetes.io/projected/8fdfc4fa-d488-4267-bfb0-113448a60fc2-kube-api-access-9zsg9\") on node \"ip-10-0-138-149.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:25:11.108451 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:25:11.108152 5378 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7kxxn\" (UniqueName: \"kubernetes.io/projected/d848cad5-c34e-4841-a2a0-e3aab4a69029-kube-api-access-7kxxn\") on node \"ip-10-0-138-149.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:25:11.108451 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:25:11.108162 5378 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8fdfc4fa-d488-4267-bfb0-113448a60fc2-istio-token\") on node \"ip-10-0-138-149.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:25:11.108451 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:25:11.108176 5378 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8fdfc4fa-d488-4267-bfb0-113448a60fc2-istio-podinfo\") on node \"ip-10-0-138-149.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:25:11.161387 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:25:11.161347 5378 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w6-8fa2a7/premium-w6-8fa2a7-kserve-64bf4bf6c-dxrdr"] Oct 30 21:25:11.163349 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:25:11.163306 5378 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w6-8fa2a7/simulator-w6-8fa2a7-kserve-6cbf5bd894-lfpcq"] Oct 30 21:25:11.230221 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:25:11.230137 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:25:11.522340 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:25:11.522255 5378 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-d1b889bd-openshift-default-566bd56c85-vvvjg"] Oct 30 21:25:11.537972 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:25:11.537941 5378 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-d1b889bd-openshift-default-566bd56c85-vvvjg"] Oct 30 21:25:11.740260 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:25:11.740219 5378 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8fdfc4fa-d488-4267-bfb0-113448a60fc2" path="/var/lib/kubelet/pods/8fdfc4fa-d488-4267-bfb0-113448a60fc2/volumes" Oct 30 21:25:11.950644 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:25:11.950614 5378 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-d1b889bd-openshift-default-566bd56c85-dvzs4" Oct 30 21:25:11.950644 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:25:11.950636 5378 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-d1b889bd-openshift-default-566bd56c85-dvzs4" event={"ID":"d848cad5-c34e-4841-a2a0-e3aab4a69029","Type":"ContainerDied","Data":"80c59336f265a4cd0b263f497f12d3c9922633be768e72ff21d90f0c61fea07a"} Oct 30 21:25:11.951125 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:25:11.950702 5378 scope.go:122] "RemoveContainer" containerID="3b9304456d83f1af5b7b823de73c14a63e17a08a5504aabb0d954f67c4de92ba" Oct 30 21:25:12.117678 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:25:12.117645 5378 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-d1b889bd-openshift-default-566bd56c85-dvzs4"] Oct 30 21:25:12.124684 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:25:12.124651 5378 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-d1b889bd-openshift-default-566bd56c85-dvzs4"] Oct 30 21:25:13.001713 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:25:13.001673 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:25:13.139303 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:25:13.139264 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:25:13.255868 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:25:13.255788 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:25:13.741275 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:25:13.741241 5378 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d848cad5-c34e-4841-a2a0-e3aab4a69029" path="/var/lib/kubelet/pods/d848cad5-c34e-4841-a2a0-e3aab4a69029/volumes" Oct 30 21:25:14.070813 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:25:14.070739 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:25:15.620235 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:25:15.620204 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:25:15.865976 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:25:15.865948 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:25:16.071380 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:25:16.071307 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:25:16.577405 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:25:16.577372 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:25:18.165427 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:25:18.165394 5378 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w6-8fa2a7-7d66df94fb-clj44"] Oct 30 21:25:18.532777 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:25:18.532690 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:25:18.652838 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:25:18.652807 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:25:18.778517 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:25:18.778477 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:25:18.838166 ip-10-0-138-149 kubenswrapper[5378]: E1030 21:25:18.838073 5378 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q6mx7" containerName="extractor" Oct 30 21:25:19.256123 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:25:19.256094 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:25:21.066427 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:25:21.066393 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:25:22.486522 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:25:22.486484 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:25:22.625088 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:25:22.625047 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:25:22.850780 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:25:22.850749 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:25:22.855649 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:25:22.855623 5378 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-639b07-openshift-default-795b89858c-9q4rj"] Oct 30 21:25:22.855911 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:25:22.855883 5378 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-639b07-openshift-default-795b89858c-9q4rj" podUID="6614eb26-439d-464e-a7c2-b95744976905" containerName="istio-proxy" containerID="cri-o://0230011be2a9e2668fc4d0bf5b8c60a9168fb78cfb050d723e26b5415a110000" gracePeriod=30 Oct 30 21:25:23.165744 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:25:23.165643 5378 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w6-8fa2a7-7d66df94fb-clj44" podUID="75677fa4-95cf-4a72-bced-a11307c20ca0" containerName="payload-processing" containerID="cri-o://b480467aa93228bbf419066e938f3fbbe9902aa8bec8eaff1d83a615d4fa301a" gracePeriod=25 Oct 30 21:25:23.178503 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:25:23.178470 5378 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-r9g59"] Oct 30 21:25:23.615248 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:25:23.615211 5378 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w6-8fa2a7-7d66df94fb-clj44" podUID="75677fa4-95cf-4a72-bced-a11307c20ca0" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 21:25:23.754753 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:25:23.754715 5378 patch_prober.go:29] interesting pod/e2e-inf-b-639b07-openshift-default-795b89858c-9q4rj container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 21:25:23.754927 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:25:23.754775 5378 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-b-639b07-openshift-default-795b89858c-9q4rj" podUID="6614eb26-439d-464e-a7c2-b95744976905" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 21:25:26.177871 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:25:26.177837 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:25:26.182115 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:25:26.182085 5378 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w6-8fa2a7/premium-w6-8fa2a7-kserve-64bf4bf6c-dxrdr" podUID="484cba5f-b415-430a-b52d-50b1293be948" containerName="main" containerID="cri-o://e39bdfd69306cfe0ef86ecd80cdbece184e6a6c7740bdee328493f0ab43ce75f" gracePeriod=45 Oct 30 21:25:26.183522 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:25:26.183494 5378 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w6-8fa2a7/simulator-w6-8fa2a7-kserve-6cbf5bd894-lfpcq" podUID="f409f469-71fc-445b-bdc8-b8e84a4d1dc6" containerName="main" containerID="cri-o://ec8df931af9b57998441e9214b15235987b3a0fe13581ac2c52e3deeb7c9f85d" gracePeriod=45 Oct 30 21:25:26.438456 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:25:26.438386 5378 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-8fa2a7/premium-w6-8fa2a7-kserve-64bf4bf6c-dxrdr" Oct 30 21:25:26.467725 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:25:26.467700 5378 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-8fa2a7/simulator-w6-8fa2a7-kserve-6cbf5bd894-lfpcq" Oct 30 21:25:26.578187 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:25:26.578158 5378 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/484cba5f-b415-430a-b52d-50b1293be948-kube-api-access-78w6l\" (UniqueName: \"kubernetes.io/projected/484cba5f-b415-430a-b52d-50b1293be948-kube-api-access-78w6l\") pod \"484cba5f-b415-430a-b52d-50b1293be948\" (UID: \"484cba5f-b415-430a-b52d-50b1293be948\") " Oct 30 21:25:26.578358 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:25:26.578205 5378 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f409f469-71fc-445b-bdc8-b8e84a4d1dc6-dshm\" (UniqueName: \"kubernetes.io/empty-dir/f409f469-71fc-445b-bdc8-b8e84a4d1dc6-dshm\") pod \"f409f469-71fc-445b-bdc8-b8e84a4d1dc6\" (UID: \"f409f469-71fc-445b-bdc8-b8e84a4d1dc6\") " Oct 30 21:25:26.578358 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:25:26.578267 5378 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/484cba5f-b415-430a-b52d-50b1293be948-home\" (UniqueName: \"kubernetes.io/empty-dir/484cba5f-b415-430a-b52d-50b1293be948-home\") pod \"484cba5f-b415-430a-b52d-50b1293be948\" (UID: \"484cba5f-b415-430a-b52d-50b1293be948\") " Oct 30 21:25:26.578358 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:25:26.578291 5378 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/484cba5f-b415-430a-b52d-50b1293be948-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/484cba5f-b415-430a-b52d-50b1293be948-tmp-dir\") pod \"484cba5f-b415-430a-b52d-50b1293be948\" (UID: \"484cba5f-b415-430a-b52d-50b1293be948\") " Oct 30 21:25:26.578358 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:25:26.578347 5378 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f409f469-71fc-445b-bdc8-b8e84a4d1dc6-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f409f469-71fc-445b-bdc8-b8e84a4d1dc6-model-cache\") pod \"f409f469-71fc-445b-bdc8-b8e84a4d1dc6\" (UID: \"f409f469-71fc-445b-bdc8-b8e84a4d1dc6\") " Oct 30 21:25:26.578610 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:25:26.578389 5378 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/484cba5f-b415-430a-b52d-50b1293be948-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/484cba5f-b415-430a-b52d-50b1293be948-model-cache\") pod \"484cba5f-b415-430a-b52d-50b1293be948\" (UID: \"484cba5f-b415-430a-b52d-50b1293be948\") " Oct 30 21:25:26.578610 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:25:26.578423 5378 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/484cba5f-b415-430a-b52d-50b1293be948-tls-certs\" (UniqueName: \"kubernetes.io/secret/484cba5f-b415-430a-b52d-50b1293be948-tls-certs\") pod \"484cba5f-b415-430a-b52d-50b1293be948\" (UID: \"484cba5f-b415-430a-b52d-50b1293be948\") " Oct 30 21:25:26.578610 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:25:26.578455 5378 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f409f469-71fc-445b-bdc8-b8e84a4d1dc6-home\" (UniqueName: \"kubernetes.io/empty-dir/f409f469-71fc-445b-bdc8-b8e84a4d1dc6-home\") pod \"f409f469-71fc-445b-bdc8-b8e84a4d1dc6\" (UID: \"f409f469-71fc-445b-bdc8-b8e84a4d1dc6\") " Oct 30 21:25:26.578610 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:25:26.578501 5378 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f409f469-71fc-445b-bdc8-b8e84a4d1dc6-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f409f469-71fc-445b-bdc8-b8e84a4d1dc6-tmp-dir\") pod \"f409f469-71fc-445b-bdc8-b8e84a4d1dc6\" (UID: \"f409f469-71fc-445b-bdc8-b8e84a4d1dc6\") " Oct 30 21:25:26.578826 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:25:26.578554 5378 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f409f469-71fc-445b-bdc8-b8e84a4d1dc6-kube-api-access-gkjj8\" (UniqueName: \"kubernetes.io/projected/f409f469-71fc-445b-bdc8-b8e84a4d1dc6-kube-api-access-gkjj8\") pod \"f409f469-71fc-445b-bdc8-b8e84a4d1dc6\" (UID: \"f409f469-71fc-445b-bdc8-b8e84a4d1dc6\") " Oct 30 21:25:26.578826 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:25:26.578659 5378 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f409f469-71fc-445b-bdc8-b8e84a4d1dc6-tls-certs\" (UniqueName: \"kubernetes.io/secret/f409f469-71fc-445b-bdc8-b8e84a4d1dc6-tls-certs\") pod \"f409f469-71fc-445b-bdc8-b8e84a4d1dc6\" (UID: \"f409f469-71fc-445b-bdc8-b8e84a4d1dc6\") " Oct 30 21:25:26.578826 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:25:26.578581 5378 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/484cba5f-b415-430a-b52d-50b1293be948-home" pod "484cba5f-b415-430a-b52d-50b1293be948" (UID: "484cba5f-b415-430a-b52d-50b1293be948"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:25:26.578826 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:25:26.578656 5378 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/484cba5f-b415-430a-b52d-50b1293be948-tmp-dir" pod "484cba5f-b415-430a-b52d-50b1293be948" (UID: "484cba5f-b415-430a-b52d-50b1293be948"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:25:26.578826 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:25:26.578685 5378 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/484cba5f-b415-430a-b52d-50b1293be948-model-cache" pod "484cba5f-b415-430a-b52d-50b1293be948" (UID: "484cba5f-b415-430a-b52d-50b1293be948"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:25:26.578826 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:25:26.578693 5378 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f409f469-71fc-445b-bdc8-b8e84a4d1dc6-model-cache" pod "f409f469-71fc-445b-bdc8-b8e84a4d1dc6" (UID: "f409f469-71fc-445b-bdc8-b8e84a4d1dc6"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:25:26.578826 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:25:26.578736 5378 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/484cba5f-b415-430a-b52d-50b1293be948-dshm\" (UniqueName: \"kubernetes.io/empty-dir/484cba5f-b415-430a-b52d-50b1293be948-dshm\") pod \"484cba5f-b415-430a-b52d-50b1293be948\" (UID: \"484cba5f-b415-430a-b52d-50b1293be948\") " Oct 30 21:25:26.578826 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:25:26.578803 5378 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f409f469-71fc-445b-bdc8-b8e84a4d1dc6-home" pod "f409f469-71fc-445b-bdc8-b8e84a4d1dc6" (UID: "f409f469-71fc-445b-bdc8-b8e84a4d1dc6"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:25:26.579226 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:25:26.578931 5378 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f409f469-71fc-445b-bdc8-b8e84a4d1dc6-tmp-dir" pod "f409f469-71fc-445b-bdc8-b8e84a4d1dc6" (UID: "f409f469-71fc-445b-bdc8-b8e84a4d1dc6"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:25:26.579226 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:25:26.579216 5378 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/484cba5f-b415-430a-b52d-50b1293be948-model-cache\") on node \"ip-10-0-138-149.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:25:26.579337 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:25:26.579239 5378 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f409f469-71fc-445b-bdc8-b8e84a4d1dc6-home\") on node \"ip-10-0-138-149.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:25:26.579337 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:25:26.579253 5378 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f409f469-71fc-445b-bdc8-b8e84a4d1dc6-tmp-dir\") on node \"ip-10-0-138-149.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:25:26.579337 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:25:26.579269 5378 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/484cba5f-b415-430a-b52d-50b1293be948-home\") on node \"ip-10-0-138-149.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:25:26.579337 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:25:26.579282 5378 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/484cba5f-b415-430a-b52d-50b1293be948-tmp-dir\") on node \"ip-10-0-138-149.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:25:26.579337 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:25:26.579294 5378 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f409f469-71fc-445b-bdc8-b8e84a4d1dc6-model-cache\") on node \"ip-10-0-138-149.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:25:26.581148 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:25:26.581126 5378 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/484cba5f-b415-430a-b52d-50b1293be948-kube-api-access-78w6l" pod "484cba5f-b415-430a-b52d-50b1293be948" (UID: "484cba5f-b415-430a-b52d-50b1293be948"). InnerVolumeSpecName "kube-api-access-78w6l". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:25:26.581287 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:25:26.581163 5378 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f409f469-71fc-445b-bdc8-b8e84a4d1dc6-kube-api-access-gkjj8" pod "f409f469-71fc-445b-bdc8-b8e84a4d1dc6" (UID: "f409f469-71fc-445b-bdc8-b8e84a4d1dc6"). InnerVolumeSpecName "kube-api-access-gkjj8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:25:26.581287 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:25:26.581207 5378 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f409f469-71fc-445b-bdc8-b8e84a4d1dc6-dshm" pod "f409f469-71fc-445b-bdc8-b8e84a4d1dc6" (UID: "f409f469-71fc-445b-bdc8-b8e84a4d1dc6"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:25:26.581287 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:25:26.581261 5378 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f409f469-71fc-445b-bdc8-b8e84a4d1dc6-tls-certs" pod "f409f469-71fc-445b-bdc8-b8e84a4d1dc6" (UID: "f409f469-71fc-445b-bdc8-b8e84a4d1dc6"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:25:26.581287 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:25:26.581273 5378 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/484cba5f-b415-430a-b52d-50b1293be948-dshm" pod "484cba5f-b415-430a-b52d-50b1293be948" (UID: "484cba5f-b415-430a-b52d-50b1293be948"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:25:26.581287 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:25:26.581260 5378 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/484cba5f-b415-430a-b52d-50b1293be948-tls-certs" pod "484cba5f-b415-430a-b52d-50b1293be948" (UID: "484cba5f-b415-430a-b52d-50b1293be948"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:25:26.680113 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:25:26.680074 5378 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/484cba5f-b415-430a-b52d-50b1293be948-tls-certs\") on node \"ip-10-0-138-149.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:25:26.680113 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:25:26.680112 5378 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gkjj8\" (UniqueName: \"kubernetes.io/projected/f409f469-71fc-445b-bdc8-b8e84a4d1dc6-kube-api-access-gkjj8\") on node \"ip-10-0-138-149.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:25:26.680304 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:25:26.680128 5378 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f409f469-71fc-445b-bdc8-b8e84a4d1dc6-tls-certs\") on node \"ip-10-0-138-149.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:25:26.680304 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:25:26.680145 5378 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/484cba5f-b415-430a-b52d-50b1293be948-dshm\") on node \"ip-10-0-138-149.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:25:26.680304 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:25:26.680160 5378 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-78w6l\" (UniqueName: \"kubernetes.io/projected/484cba5f-b415-430a-b52d-50b1293be948-kube-api-access-78w6l\") on node \"ip-10-0-138-149.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:25:26.680304 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:25:26.680173 5378 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f409f469-71fc-445b-bdc8-b8e84a4d1dc6-dshm\") on node \"ip-10-0-138-149.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:25:27.038716 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:25:27.038678 5378 generic.go:356] "Generic (PLEG): container finished" podID="f409f469-71fc-445b-bdc8-b8e84a4d1dc6" containerID="ec8df931af9b57998441e9214b15235987b3a0fe13581ac2c52e3deeb7c9f85d" exitCode=0 Oct 30 21:25:27.038888 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:25:27.038729 5378 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-8fa2a7/simulator-w6-8fa2a7-kserve-6cbf5bd894-lfpcq" Oct 30 21:25:27.038888 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:25:27.038762 5378 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-8fa2a7/simulator-w6-8fa2a7-kserve-6cbf5bd894-lfpcq" event={"ID":"f409f469-71fc-445b-bdc8-b8e84a4d1dc6","Type":"ContainerDied","Data":"ec8df931af9b57998441e9214b15235987b3a0fe13581ac2c52e3deeb7c9f85d"} Oct 30 21:25:27.038888 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:25:27.038813 5378 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-8fa2a7/simulator-w6-8fa2a7-kserve-6cbf5bd894-lfpcq" event={"ID":"f409f469-71fc-445b-bdc8-b8e84a4d1dc6","Type":"ContainerDied","Data":"4e5802167291b0752fb50965139888bfe6b4f4a5c36d3c5663e8389d085c4cd8"} Oct 30 21:25:27.038888 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:25:27.038844 5378 scope.go:122] "RemoveContainer" containerID="ec8df931af9b57998441e9214b15235987b3a0fe13581ac2c52e3deeb7c9f85d" Oct 30 21:25:27.040207 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:25:27.040186 5378 generic.go:356] "Generic (PLEG): container finished" podID="484cba5f-b415-430a-b52d-50b1293be948" containerID="e39bdfd69306cfe0ef86ecd80cdbece184e6a6c7740bdee328493f0ab43ce75f" exitCode=0 Oct 30 21:25:27.040332 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:25:27.040222 5378 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-8fa2a7/premium-w6-8fa2a7-kserve-64bf4bf6c-dxrdr" event={"ID":"484cba5f-b415-430a-b52d-50b1293be948","Type":"ContainerDied","Data":"e39bdfd69306cfe0ef86ecd80cdbece184e6a6c7740bdee328493f0ab43ce75f"} Oct 30 21:25:27.040332 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:25:27.040239 5378 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-8fa2a7/premium-w6-8fa2a7-kserve-64bf4bf6c-dxrdr" Oct 30 21:25:27.040332 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:25:27.040266 5378 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-8fa2a7/premium-w6-8fa2a7-kserve-64bf4bf6c-dxrdr" event={"ID":"484cba5f-b415-430a-b52d-50b1293be948","Type":"ContainerDied","Data":"c12109ae609d50415f5a179b8abc64886aa92c88edda51853e4094c865576024"} Oct 30 21:25:27.049535 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:25:27.049505 5378 scope.go:122] "RemoveContainer" containerID="ec8df931af9b57998441e9214b15235987b3a0fe13581ac2c52e3deeb7c9f85d" Oct 30 21:25:27.049985 ip-10-0-138-149 kubenswrapper[5378]: E1030 21:25:27.049961 5378 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ec8df931af9b57998441e9214b15235987b3a0fe13581ac2c52e3deeb7c9f85d\": container with ID starting with ec8df931af9b57998441e9214b15235987b3a0fe13581ac2c52e3deeb7c9f85d not found: ID does not exist" containerID="ec8df931af9b57998441e9214b15235987b3a0fe13581ac2c52e3deeb7c9f85d" Oct 30 21:25:27.050090 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:25:27.049994 5378 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ec8df931af9b57998441e9214b15235987b3a0fe13581ac2c52e3deeb7c9f85d"} err="failed to get container status \"ec8df931af9b57998441e9214b15235987b3a0fe13581ac2c52e3deeb7c9f85d\": rpc error: code = NotFound desc = could not find container \"ec8df931af9b57998441e9214b15235987b3a0fe13581ac2c52e3deeb7c9f85d\": container with ID starting with ec8df931af9b57998441e9214b15235987b3a0fe13581ac2c52e3deeb7c9f85d not found: ID does not exist" Oct 30 21:25:27.050090 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:25:27.050021 5378 scope.go:122] "RemoveContainer" containerID="e39bdfd69306cfe0ef86ecd80cdbece184e6a6c7740bdee328493f0ab43ce75f" Oct 30 21:25:27.066923 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:25:27.066903 5378 scope.go:122] "RemoveContainer" containerID="e39bdfd69306cfe0ef86ecd80cdbece184e6a6c7740bdee328493f0ab43ce75f" Oct 30 21:25:27.067174 ip-10-0-138-149 kubenswrapper[5378]: E1030 21:25:27.067153 5378 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e39bdfd69306cfe0ef86ecd80cdbece184e6a6c7740bdee328493f0ab43ce75f\": container with ID starting with e39bdfd69306cfe0ef86ecd80cdbece184e6a6c7740bdee328493f0ab43ce75f not found: ID does not exist" containerID="e39bdfd69306cfe0ef86ecd80cdbece184e6a6c7740bdee328493f0ab43ce75f" Oct 30 21:25:27.067244 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:25:27.067186 5378 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e39bdfd69306cfe0ef86ecd80cdbece184e6a6c7740bdee328493f0ab43ce75f"} err="failed to get container status \"e39bdfd69306cfe0ef86ecd80cdbece184e6a6c7740bdee328493f0ab43ce75f\": rpc error: code = NotFound desc = could not find container \"e39bdfd69306cfe0ef86ecd80cdbece184e6a6c7740bdee328493f0ab43ce75f\": container with ID starting with e39bdfd69306cfe0ef86ecd80cdbece184e6a6c7740bdee328493f0ab43ce75f not found: ID does not exist" Oct 30 21:25:27.202993 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:25:27.202951 5378 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w6-8fa2a7/premium-w6-8fa2a7-kserve-64bf4bf6c-dxrdr"] Oct 30 21:25:27.253143 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:25:27.253112 5378 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w6-8fa2a7/premium-w6-8fa2a7-kserve-64bf4bf6c-dxrdr"] Oct 30 21:25:27.314479 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:25:27.314410 5378 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w6-8fa2a7/simulator-w6-8fa2a7-kserve-6cbf5bd894-lfpcq"] Oct 30 21:25:27.316076 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:25:27.316054 5378 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w6-8fa2a7/simulator-w6-8fa2a7-kserve-6cbf5bd894-lfpcq"] Oct 30 21:25:27.741004 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:25:27.740967 5378 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="484cba5f-b415-430a-b52d-50b1293be948" path="/var/lib/kubelet/pods/484cba5f-b415-430a-b52d-50b1293be948/volumes" Oct 30 21:25:27.741430 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:25:27.741414 5378 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f409f469-71fc-445b-bdc8-b8e84a4d1dc6" path="/var/lib/kubelet/pods/f409f469-71fc-445b-bdc8-b8e84a4d1dc6/volumes" Oct 30 21:25:28.055366 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:25:28.055330 5378 generic.go:356] "Generic (PLEG): container finished" podID="6614eb26-439d-464e-a7c2-b95744976905" containerID="0230011be2a9e2668fc4d0bf5b8c60a9168fb78cfb050d723e26b5415a110000" exitCode=0 Oct 30 21:25:28.055544 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:25:28.055454 5378 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-639b07-openshift-default-795b89858c-9q4rj" event={"ID":"6614eb26-439d-464e-a7c2-b95744976905","Type":"ContainerDied","Data":"0230011be2a9e2668fc4d0bf5b8c60a9168fb78cfb050d723e26b5415a110000"} Oct 30 21:25:28.055544 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:25:28.055514 5378 scope.go:122] "RemoveContainer" containerID="a76a56d4d1a7618806474066a486dd2bd3c8f5287141fddadcc8dfb270bdd372" Oct 30 21:25:28.114657 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:25:28.114631 5378 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-639b07-openshift-default-795b89858c-9q4rj" Oct 30 21:25:28.296033 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:25:28.295960 5378 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6614eb26-439d-464e-a7c2-b95744976905-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6614eb26-439d-464e-a7c2-b95744976905-istio-envoy\") pod \"6614eb26-439d-464e-a7c2-b95744976905\" (UID: \"6614eb26-439d-464e-a7c2-b95744976905\") " Oct 30 21:25:28.296033 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:25:28.295995 5378 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6614eb26-439d-464e-a7c2-b95744976905-istio-token\" (UniqueName: \"kubernetes.io/projected/6614eb26-439d-464e-a7c2-b95744976905-istio-token\") pod \"6614eb26-439d-464e-a7c2-b95744976905\" (UID: \"6614eb26-439d-464e-a7c2-b95744976905\") " Oct 30 21:25:28.296033 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:25:28.296021 5378 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6614eb26-439d-464e-a7c2-b95744976905-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6614eb26-439d-464e-a7c2-b95744976905-istio-data\") pod \"6614eb26-439d-464e-a7c2-b95744976905\" (UID: \"6614eb26-439d-464e-a7c2-b95744976905\") " Oct 30 21:25:28.296651 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:25:28.296085 5378 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6614eb26-439d-464e-a7c2-b95744976905-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6614eb26-439d-464e-a7c2-b95744976905-workload-socket\") pod \"6614eb26-439d-464e-a7c2-b95744976905\" (UID: \"6614eb26-439d-464e-a7c2-b95744976905\") " Oct 30 21:25:28.296651 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:25:28.296111 5378 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6614eb26-439d-464e-a7c2-b95744976905-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6614eb26-439d-464e-a7c2-b95744976905-workload-certs\") pod \"6614eb26-439d-464e-a7c2-b95744976905\" (UID: \"6614eb26-439d-464e-a7c2-b95744976905\") " Oct 30 21:25:28.296651 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:25:28.296160 5378 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6614eb26-439d-464e-a7c2-b95744976905-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6614eb26-439d-464e-a7c2-b95744976905-credential-socket\") pod \"6614eb26-439d-464e-a7c2-b95744976905\" (UID: \"6614eb26-439d-464e-a7c2-b95744976905\") " Oct 30 21:25:28.296651 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:25:28.296206 5378 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6614eb26-439d-464e-a7c2-b95744976905-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6614eb26-439d-464e-a7c2-b95744976905-istiod-ca-cert\") pod \"6614eb26-439d-464e-a7c2-b95744976905\" (UID: \"6614eb26-439d-464e-a7c2-b95744976905\") " Oct 30 21:25:28.296651 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:25:28.296234 5378 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6614eb26-439d-464e-a7c2-b95744976905-kube-api-access-c5br9\" (UniqueName: \"kubernetes.io/projected/6614eb26-439d-464e-a7c2-b95744976905-kube-api-access-c5br9\") pod \"6614eb26-439d-464e-a7c2-b95744976905\" (UID: \"6614eb26-439d-464e-a7c2-b95744976905\") " Oct 30 21:25:28.296651 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:25:28.296285 5378 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/6614eb26-439d-464e-a7c2-b95744976905-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6614eb26-439d-464e-a7c2-b95744976905-istio-podinfo\") pod \"6614eb26-439d-464e-a7c2-b95744976905\" (UID: \"6614eb26-439d-464e-a7c2-b95744976905\") " Oct 30 21:25:28.296651 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:25:28.296395 5378 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6614eb26-439d-464e-a7c2-b95744976905-istio-data" pod "6614eb26-439d-464e-a7c2-b95744976905" (UID: "6614eb26-439d-464e-a7c2-b95744976905"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:25:28.296651 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:25:28.296383 5378 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6614eb26-439d-464e-a7c2-b95744976905-workload-socket" pod "6614eb26-439d-464e-a7c2-b95744976905" (UID: "6614eb26-439d-464e-a7c2-b95744976905"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:25:28.296651 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:25:28.296538 5378 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6614eb26-439d-464e-a7c2-b95744976905-workload-certs" pod "6614eb26-439d-464e-a7c2-b95744976905" (UID: "6614eb26-439d-464e-a7c2-b95744976905"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:25:28.297052 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:25:28.296771 5378 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6614eb26-439d-464e-a7c2-b95744976905-workload-socket\") on node \"ip-10-0-138-149.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:25:28.297052 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:25:28.296797 5378 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6614eb26-439d-464e-a7c2-b95744976905-workload-certs\") on node \"ip-10-0-138-149.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:25:28.297052 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:25:28.296811 5378 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6614eb26-439d-464e-a7c2-b95744976905-istio-data\") on node \"ip-10-0-138-149.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:25:28.297228 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:25:28.297049 5378 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6614eb26-439d-464e-a7c2-b95744976905-credential-socket" pod "6614eb26-439d-464e-a7c2-b95744976905" (UID: "6614eb26-439d-464e-a7c2-b95744976905"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:25:28.297228 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:25:28.297156 5378 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6614eb26-439d-464e-a7c2-b95744976905-istiod-ca-cert" pod "6614eb26-439d-464e-a7c2-b95744976905" (UID: "6614eb26-439d-464e-a7c2-b95744976905"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:25:28.298453 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:25:28.298423 5378 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6614eb26-439d-464e-a7c2-b95744976905-istio-token" pod "6614eb26-439d-464e-a7c2-b95744976905" (UID: "6614eb26-439d-464e-a7c2-b95744976905"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:25:28.298628 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:25:28.298496 5378 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6614eb26-439d-464e-a7c2-b95744976905-istio-envoy" pod "6614eb26-439d-464e-a7c2-b95744976905" (UID: "6614eb26-439d-464e-a7c2-b95744976905"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:25:28.298912 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:25:28.298888 5378 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/6614eb26-439d-464e-a7c2-b95744976905-istio-podinfo" pod "6614eb26-439d-464e-a7c2-b95744976905" (UID: "6614eb26-439d-464e-a7c2-b95744976905"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 21:25:28.299053 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:25:28.299031 5378 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6614eb26-439d-464e-a7c2-b95744976905-kube-api-access-c5br9" pod "6614eb26-439d-464e-a7c2-b95744976905" (UID: "6614eb26-439d-464e-a7c2-b95744976905"). InnerVolumeSpecName "kube-api-access-c5br9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:25:28.397349 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:25:28.397310 5378 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6614eb26-439d-464e-a7c2-b95744976905-credential-socket\") on node \"ip-10-0-138-149.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:25:28.397349 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:25:28.397341 5378 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6614eb26-439d-464e-a7c2-b95744976905-istiod-ca-cert\") on node \"ip-10-0-138-149.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:25:28.397545 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:25:28.397355 5378 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c5br9\" (UniqueName: \"kubernetes.io/projected/6614eb26-439d-464e-a7c2-b95744976905-kube-api-access-c5br9\") on node \"ip-10-0-138-149.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:25:28.397545 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:25:28.397369 5378 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6614eb26-439d-464e-a7c2-b95744976905-istio-podinfo\") on node \"ip-10-0-138-149.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:25:28.397545 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:25:28.397381 5378 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6614eb26-439d-464e-a7c2-b95744976905-istio-envoy\") on node \"ip-10-0-138-149.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:25:28.397545 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:25:28.397392 5378 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6614eb26-439d-464e-a7c2-b95744976905-istio-token\") on node \"ip-10-0-138-149.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:25:29.062974 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:25:29.062934 5378 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-639b07-openshift-default-795b89858c-9q4rj" event={"ID":"6614eb26-439d-464e-a7c2-b95744976905","Type":"ContainerDied","Data":"34c44600a35387ef6b926480c71f4279df9258b1059604e0b044e9f8e3e80d0e"} Oct 30 21:25:29.062974 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:25:29.062964 5378 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-639b07-openshift-default-795b89858c-9q4rj" Oct 30 21:25:29.063235 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:25:29.063010 5378 scope.go:122] "RemoveContainer" containerID="0230011be2a9e2668fc4d0bf5b8c60a9168fb78cfb050d723e26b5415a110000" Oct 30 21:25:29.278914 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:25:29.278870 5378 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-639b07-openshift-default-795b89858c-9q4rj"] Oct 30 21:25:29.285409 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:25:29.285381 5378 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-639b07-openshift-default-795b89858c-9q4rj"] Oct 30 21:25:29.739848 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:25:29.739808 5378 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6614eb26-439d-464e-a7c2-b95744976905" path="/var/lib/kubelet/pods/6614eb26-439d-464e-a7c2-b95744976905/volumes" Oct 30 21:25:31.491072 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:25:31.491042 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:25:33.615467 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:25:33.615429 5378 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w6-8fa2a7-7d66df94fb-clj44" podUID="75677fa4-95cf-4a72-bced-a11307c20ca0" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 21:25:34.088558 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:25:34.088483 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:25:34.235067 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:25:34.235035 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:25:35.281072 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:25:35.281034 5378 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w4-5c45e5-7f47b69b66-6ks8n"] Oct 30 21:25:35.395097 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:25:35.395059 5378 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w4-5c45e5-76458b97d7-7vrgq"] Oct 30 21:25:35.636961 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:25:35.636916 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:25:35.762904 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:25:35.762867 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:25:35.942310 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:25:35.942229 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:25:37.526321 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:25:37.526277 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:25:37.772517 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:25:37.772476 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:25:37.891750 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:25:37.891712 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:25:38.453074 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:25:38.453035 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:25:40.281770 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:25:40.281731 5378 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-5c45e5-7f47b69b66-6ks8n" podUID="a6c2ebb4-9a45-4038-87a1-a40a242ec833" containerName="payload-processing" containerID="cri-o://8a92fb2d43ce992c25c5b749b25eb9203767ecfb98b1e4a890a653a405f5963e" gracePeriod=25 Oct 30 21:25:40.395860 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:25:40.395821 5378 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-5c45e5-76458b97d7-7vrgq" podUID="52b8c06c-c948-452c-b317-523f09618599" containerName="payload-pre-processing" containerID="cri-o://2cab747c206431d8e8afdc24b84ad6a98dcdba1b4de6597508167580089d634a" gracePeriod=25 Oct 30 21:25:40.421502 ip-10-0-138-149 kubenswrapper[5378]: W1030 21:25:40.421479 5378 logging.go:55] [core] [Channel #7953 SubChannel #7954]grpc: addrConn.createTransport failed to connect to {Addr: "10.133.0.110:50052", ServerName: "10.133.0.110:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.133.0.110:50052: connect: connection refused" Oct 30 21:25:40.421654 ip-10-0-138-149 kubenswrapper[5378]: W1030 21:25:40.421510 5378 logging.go:55] [core] [Channel #7952 SubChannel #7955]grpc: addrConn.createTransport failed to connect to {Addr: "10.133.0.111:50052", ServerName: "10.133.0.111:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.133.0.111:50052: connect: connection refused" Oct 30 21:25:40.555082 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:25:40.555055 5378 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-5c45e5-7f47b69b66-6ks8n" Oct 30 21:25:40.623482 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:25:40.623451 5378 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a6c2ebb4-9a45-4038-87a1-a40a242ec833-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a6c2ebb4-9a45-4038-87a1-a40a242ec833-plugins-config-volume\") pod \"a6c2ebb4-9a45-4038-87a1-a40a242ec833\" (UID: \"a6c2ebb4-9a45-4038-87a1-a40a242ec833\") " Oct 30 21:25:40.623683 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:25:40.623554 5378 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a6c2ebb4-9a45-4038-87a1-a40a242ec833-kube-api-access-fxs2k\" (UniqueName: \"kubernetes.io/projected/a6c2ebb4-9a45-4038-87a1-a40a242ec833-kube-api-access-fxs2k\") pod \"a6c2ebb4-9a45-4038-87a1-a40a242ec833\" (UID: \"a6c2ebb4-9a45-4038-87a1-a40a242ec833\") " Oct 30 21:25:40.623923 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:25:40.623890 5378 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a6c2ebb4-9a45-4038-87a1-a40a242ec833-plugins-config-volume" pod "a6c2ebb4-9a45-4038-87a1-a40a242ec833" (UID: "a6c2ebb4-9a45-4038-87a1-a40a242ec833"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:25:40.626310 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:25:40.626288 5378 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a6c2ebb4-9a45-4038-87a1-a40a242ec833-kube-api-access-fxs2k" pod "a6c2ebb4-9a45-4038-87a1-a40a242ec833" (UID: "a6c2ebb4-9a45-4038-87a1-a40a242ec833"). InnerVolumeSpecName "kube-api-access-fxs2k". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:25:40.654427 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:25:40.654406 5378 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-5c45e5-76458b97d7-7vrgq" Oct 30 21:25:40.724409 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:25:40.724378 5378 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/52b8c06c-c948-452c-b317-523f09618599-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/52b8c06c-c948-452c-b317-523f09618599-plugins-config-volume\") pod \"52b8c06c-c948-452c-b317-523f09618599\" (UID: \"52b8c06c-c948-452c-b317-523f09618599\") " Oct 30 21:25:40.724586 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:25:40.724510 5378 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/52b8c06c-c948-452c-b317-523f09618599-kube-api-access-gzqg7\" (UniqueName: \"kubernetes.io/projected/52b8c06c-c948-452c-b317-523f09618599-kube-api-access-gzqg7\") pod \"52b8c06c-c948-452c-b317-523f09618599\" (UID: \"52b8c06c-c948-452c-b317-523f09618599\") " Oct 30 21:25:40.724733 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:25:40.724719 5378 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a6c2ebb4-9a45-4038-87a1-a40a242ec833-plugins-config-volume\") on node \"ip-10-0-138-149.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:25:40.724807 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:25:40.724733 5378 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fxs2k\" (UniqueName: \"kubernetes.io/projected/a6c2ebb4-9a45-4038-87a1-a40a242ec833-kube-api-access-fxs2k\") on node \"ip-10-0-138-149.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:25:40.724867 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:25:40.724822 5378 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/52b8c06c-c948-452c-b317-523f09618599-plugins-config-volume" pod "52b8c06c-c948-452c-b317-523f09618599" (UID: "52b8c06c-c948-452c-b317-523f09618599"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:25:40.726459 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:25:40.726440 5378 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/52b8c06c-c948-452c-b317-523f09618599-kube-api-access-gzqg7" pod "52b8c06c-c948-452c-b317-523f09618599" (UID: "52b8c06c-c948-452c-b317-523f09618599"). InnerVolumeSpecName "kube-api-access-gzqg7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:25:40.825952 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:25:40.825877 5378 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gzqg7\" (UniqueName: \"kubernetes.io/projected/52b8c06c-c948-452c-b317-523f09618599-kube-api-access-gzqg7\") on node \"ip-10-0-138-149.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:25:40.825952 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:25:40.825900 5378 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/52b8c06c-c948-452c-b317-523f09618599-plugins-config-volume\") on node \"ip-10-0-138-149.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:25:41.123819 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:25:41.123785 5378 generic.go:356] "Generic (PLEG): container finished" podID="a6c2ebb4-9a45-4038-87a1-a40a242ec833" containerID="8a92fb2d43ce992c25c5b749b25eb9203767ecfb98b1e4a890a653a405f5963e" exitCode=0 Oct 30 21:25:41.123999 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:25:41.123830 5378 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-5c45e5-7f47b69b66-6ks8n" Oct 30 21:25:41.123999 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:25:41.123869 5378 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-5c45e5-7f47b69b66-6ks8n" event={"ID":"a6c2ebb4-9a45-4038-87a1-a40a242ec833","Type":"ContainerDied","Data":"8a92fb2d43ce992c25c5b749b25eb9203767ecfb98b1e4a890a653a405f5963e"} Oct 30 21:25:41.123999 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:25:41.123922 5378 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-5c45e5-7f47b69b66-6ks8n" event={"ID":"a6c2ebb4-9a45-4038-87a1-a40a242ec833","Type":"ContainerDied","Data":"48068a32578b21d9960579b977a170f4486d82e66a3f1c257e11207d66d9a69a"} Oct 30 21:25:41.123999 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:25:41.123951 5378 scope.go:122] "RemoveContainer" containerID="8a92fb2d43ce992c25c5b749b25eb9203767ecfb98b1e4a890a653a405f5963e" Oct 30 21:25:41.125271 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:25:41.125246 5378 generic.go:356] "Generic (PLEG): container finished" podID="52b8c06c-c948-452c-b317-523f09618599" containerID="2cab747c206431d8e8afdc24b84ad6a98dcdba1b4de6597508167580089d634a" exitCode=0 Oct 30 21:25:41.125394 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:25:41.125270 5378 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-5c45e5-76458b97d7-7vrgq" event={"ID":"52b8c06c-c948-452c-b317-523f09618599","Type":"ContainerDied","Data":"2cab747c206431d8e8afdc24b84ad6a98dcdba1b4de6597508167580089d634a"} Oct 30 21:25:41.125394 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:25:41.125303 5378 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-5c45e5-76458b97d7-7vrgq" Oct 30 21:25:41.125394 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:25:41.125326 5378 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-5c45e5-76458b97d7-7vrgq" event={"ID":"52b8c06c-c948-452c-b317-523f09618599","Type":"ContainerDied","Data":"19070c021d669e4e82893b06f13f9921cadfdefab97e446ca232e57e1d948602"} Oct 30 21:25:41.132890 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:25:41.132873 5378 scope.go:122] "RemoveContainer" containerID="8a92fb2d43ce992c25c5b749b25eb9203767ecfb98b1e4a890a653a405f5963e" Oct 30 21:25:41.133163 ip-10-0-138-149 kubenswrapper[5378]: E1030 21:25:41.133145 5378 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8a92fb2d43ce992c25c5b749b25eb9203767ecfb98b1e4a890a653a405f5963e\": container with ID starting with 8a92fb2d43ce992c25c5b749b25eb9203767ecfb98b1e4a890a653a405f5963e not found: ID does not exist" containerID="8a92fb2d43ce992c25c5b749b25eb9203767ecfb98b1e4a890a653a405f5963e" Oct 30 21:25:41.133235 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:25:41.133171 5378 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8a92fb2d43ce992c25c5b749b25eb9203767ecfb98b1e4a890a653a405f5963e"} err="failed to get container status \"8a92fb2d43ce992c25c5b749b25eb9203767ecfb98b1e4a890a653a405f5963e\": rpc error: code = NotFound desc = could not find container \"8a92fb2d43ce992c25c5b749b25eb9203767ecfb98b1e4a890a653a405f5963e\": container with ID starting with 8a92fb2d43ce992c25c5b749b25eb9203767ecfb98b1e4a890a653a405f5963e not found: ID does not exist" Oct 30 21:25:41.133235 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:25:41.133188 5378 scope.go:122] "RemoveContainer" containerID="2cab747c206431d8e8afdc24b84ad6a98dcdba1b4de6597508167580089d634a" Oct 30 21:25:41.141705 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:25:41.141685 5378 scope.go:122] "RemoveContainer" containerID="2cab747c206431d8e8afdc24b84ad6a98dcdba1b4de6597508167580089d634a" Oct 30 21:25:41.141934 ip-10-0-138-149 kubenswrapper[5378]: E1030 21:25:41.141917 5378 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2cab747c206431d8e8afdc24b84ad6a98dcdba1b4de6597508167580089d634a\": container with ID starting with 2cab747c206431d8e8afdc24b84ad6a98dcdba1b4de6597508167580089d634a not found: ID does not exist" containerID="2cab747c206431d8e8afdc24b84ad6a98dcdba1b4de6597508167580089d634a" Oct 30 21:25:41.141994 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:25:41.141942 5378 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2cab747c206431d8e8afdc24b84ad6a98dcdba1b4de6597508167580089d634a"} err="failed to get container status \"2cab747c206431d8e8afdc24b84ad6a98dcdba1b4de6597508167580089d634a\": rpc error: code = NotFound desc = could not find container \"2cab747c206431d8e8afdc24b84ad6a98dcdba1b4de6597508167580089d634a\": container with ID starting with 2cab747c206431d8e8afdc24b84ad6a98dcdba1b4de6597508167580089d634a not found: ID does not exist" Oct 30 21:25:41.290112 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:25:41.290071 5378 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w4-5c45e5-7f47b69b66-6ks8n"] Oct 30 21:25:41.294404 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:25:41.294374 5378 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w4-5c45e5-7f47b69b66-6ks8n"] Oct 30 21:25:41.408019 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:25:41.407950 5378 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w4-5c45e5-76458b97d7-7vrgq"] Oct 30 21:25:41.418544 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:25:41.418519 5378 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w4-5c45e5-76458b97d7-7vrgq"] Oct 30 21:25:41.421448 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:25:41.421418 5378 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-5c45e5-76458b97d7-7vrgq" podUID="52b8c06c-c948-452c-b317-523f09618599" containerName="payload-pre-processing" probeResult="failure" output="timeout: failed to connect service \"10.133.0.110:50052\" within 1s: context deadline exceeded" Oct 30 21:25:41.421448 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:25:41.421429 5378 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-5c45e5-7f47b69b66-6ks8n" podUID="a6c2ebb4-9a45-4038-87a1-a40a242ec833" containerName="payload-processing" probeResult="failure" output="timeout: failed to connect service \"10.133.0.111:50052\" within 1s: context deadline exceeded" Oct 30 21:25:41.740319 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:25:41.740245 5378 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="52b8c06c-c948-452c-b317-523f09618599" path="/var/lib/kubelet/pods/52b8c06c-c948-452c-b317-523f09618599/volumes" Oct 30 21:25:41.740652 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:25:41.740638 5378 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a6c2ebb4-9a45-4038-87a1-a40a242ec833" path="/var/lib/kubelet/pods/a6c2ebb4-9a45-4038-87a1-a40a242ec833/volumes" Oct 30 21:25:42.216090 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:25:42.216058 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:25:43.096416 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:25:43.096360 5378 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-r9g59"] Oct 30 21:25:43.450536 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:25:43.450507 5378 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w6-8fa2a7-7d66df94fb-clj44" Oct 30 21:25:43.558049 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:25:43.558015 5378 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/75677fa4-95cf-4a72-bced-a11307c20ca0-kube-api-access-vptpm\" (UniqueName: \"kubernetes.io/projected/75677fa4-95cf-4a72-bced-a11307c20ca0-kube-api-access-vptpm\") pod \"75677fa4-95cf-4a72-bced-a11307c20ca0\" (UID: \"75677fa4-95cf-4a72-bced-a11307c20ca0\") " Oct 30 21:25:43.558240 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:25:43.558100 5378 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/75677fa4-95cf-4a72-bced-a11307c20ca0-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/75677fa4-95cf-4a72-bced-a11307c20ca0-plugins-config-volume\") pod \"75677fa4-95cf-4a72-bced-a11307c20ca0\" (UID: \"75677fa4-95cf-4a72-bced-a11307c20ca0\") " Oct 30 21:25:43.558404 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:25:43.558380 5378 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/75677fa4-95cf-4a72-bced-a11307c20ca0-plugins-config-volume" pod "75677fa4-95cf-4a72-bced-a11307c20ca0" (UID: "75677fa4-95cf-4a72-bced-a11307c20ca0"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:25:43.558514 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:25:43.558500 5378 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/75677fa4-95cf-4a72-bced-a11307c20ca0-plugins-config-volume\") on node \"ip-10-0-138-149.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:25:43.560115 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:25:43.560092 5378 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/75677fa4-95cf-4a72-bced-a11307c20ca0-kube-api-access-vptpm" pod "75677fa4-95cf-4a72-bced-a11307c20ca0" (UID: "75677fa4-95cf-4a72-bced-a11307c20ca0"). InnerVolumeSpecName "kube-api-access-vptpm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:25:43.660153 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:25:43.660076 5378 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vptpm\" (UniqueName: \"kubernetes.io/projected/75677fa4-95cf-4a72-bced-a11307c20ca0-kube-api-access-vptpm\") on node \"ip-10-0-138-149.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:25:43.846809 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:25:43.846775 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:25:44.085384 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:25:44.085267 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:25:44.146206 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:25:44.146175 5378 generic.go:356] "Generic (PLEG): container finished" podID="75677fa4-95cf-4a72-bced-a11307c20ca0" containerID="b480467aa93228bbf419066e938f3fbbe9902aa8bec8eaff1d83a615d4fa301a" exitCode=0 Oct 30 21:25:44.146735 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:25:44.146238 5378 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w6-8fa2a7-7d66df94fb-clj44" Oct 30 21:25:44.146735 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:25:44.146256 5378 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w6-8fa2a7-7d66df94fb-clj44" event={"ID":"75677fa4-95cf-4a72-bced-a11307c20ca0","Type":"ContainerDied","Data":"b480467aa93228bbf419066e938f3fbbe9902aa8bec8eaff1d83a615d4fa301a"} Oct 30 21:25:44.146735 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:25:44.146324 5378 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w6-8fa2a7-7d66df94fb-clj44" event={"ID":"75677fa4-95cf-4a72-bced-a11307c20ca0","Type":"ContainerDied","Data":"ff5e586d01159dad4d08397592dc0c2da8f845980b2f067dc1bf269b5d854f05"} Oct 30 21:25:44.146735 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:25:44.146355 5378 scope.go:122] "RemoveContainer" containerID="b480467aa93228bbf419066e938f3fbbe9902aa8bec8eaff1d83a615d4fa301a" Oct 30 21:25:44.155091 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:25:44.155072 5378 scope.go:122] "RemoveContainer" containerID="b480467aa93228bbf419066e938f3fbbe9902aa8bec8eaff1d83a615d4fa301a" Oct 30 21:25:44.155403 ip-10-0-138-149 kubenswrapper[5378]: E1030 21:25:44.155383 5378 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b480467aa93228bbf419066e938f3fbbe9902aa8bec8eaff1d83a615d4fa301a\": container with ID starting with b480467aa93228bbf419066e938f3fbbe9902aa8bec8eaff1d83a615d4fa301a not found: ID does not exist" containerID="b480467aa93228bbf419066e938f3fbbe9902aa8bec8eaff1d83a615d4fa301a" Oct 30 21:25:44.155468 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:25:44.155412 5378 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b480467aa93228bbf419066e938f3fbbe9902aa8bec8eaff1d83a615d4fa301a"} err="failed to get container status \"b480467aa93228bbf419066e938f3fbbe9902aa8bec8eaff1d83a615d4fa301a\": rpc error: code = NotFound desc = could not find container \"b480467aa93228bbf419066e938f3fbbe9902aa8bec8eaff1d83a615d4fa301a\": container with ID starting with b480467aa93228bbf419066e938f3fbbe9902aa8bec8eaff1d83a615d4fa301a not found: ID does not exist" Oct 30 21:25:44.210155 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:25:44.210087 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:25:44.313653 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:25:44.313619 5378 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w6-8fa2a7-7d66df94fb-clj44"] Oct 30 21:25:44.317419 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:25:44.317391 5378 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w6-8fa2a7-7d66df94fb-clj44"] Oct 30 21:25:44.785554 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:25:44.785514 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:25:45.120156 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:25:45.120117 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:25:45.260767 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:25:45.260732 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:25:45.494841 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:25:45.494762 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:25:45.740510 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:25:45.740474 5378 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="75677fa4-95cf-4a72-bced-a11307c20ca0" path="/var/lib/kubelet/pods/75677fa4-95cf-4a72-bced-a11307c20ca0/volumes" Oct 30 21:25:48.587531 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:25:48.587495 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:25:48.720809 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:25:48.720776 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:25:48.836966 ip-10-0-138-149 kubenswrapper[5378]: E1030 21:25:48.836930 5378 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q6mx7" containerName="extractor" Oct 30 21:25:48.877000 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:25:48.876971 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:25:49.369985 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:25:49.369949 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:25:50.773710 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:25:50.773675 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:25:50.911972 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:25:50.911938 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:25:51.129787 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:25:51.129751 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:25:51.145366 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:25:51.145341 5378 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-5c45e5-openshift-default-9dcdd54d9-s697g"] Oct 30 21:25:51.145576 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:25:51.145541 5378 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w4-5c45e5-openshift-default-9dcdd54d9-s697g" podUID="23f08320-e0a1-42bc-8fd1-de3492b59bca" containerName="istio-proxy" containerID="cri-o://3411ad24d40a29ed63e18fa10cf40588cbb83130efc1cf2232c5acd10a3a0303" gracePeriod=30 Oct 30 21:25:52.182733 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:25:52.182691 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:25:53.792670 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:25:53.792619 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:25:54.444234 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:25:54.444201 5378 patch_prober.go:29] interesting pod/e2e-shared-a-w4-5c45e5-openshift-default-9dcdd54d9-s697g container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 21:25:54.444412 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:25:54.444265 5378 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-w4-5c45e5-openshift-default-9dcdd54d9-s697g" podUID="23f08320-e0a1-42bc-8fd1-de3492b59bca" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 21:25:54.860273 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:25:54.860239 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:25:55.504948 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:25:55.504916 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:25:55.787101 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:25:55.787027 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:25:55.912768 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:25:55.912723 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:25:56.141189 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:25:56.141154 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:25:56.421902 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:25:56.421869 5378 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-5c45e5-openshift-default-9dcdd54d9-s697g" Oct 30 21:25:56.426706 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:25:56.426669 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:25:56.493734 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:25:56.493701 5378 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/23f08320-e0a1-42bc-8fd1-de3492b59bca-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/23f08320-e0a1-42bc-8fd1-de3492b59bca-istio-podinfo\") pod \"23f08320-e0a1-42bc-8fd1-de3492b59bca\" (UID: \"23f08320-e0a1-42bc-8fd1-de3492b59bca\") " Oct 30 21:25:56.493948 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:25:56.493749 5378 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/23f08320-e0a1-42bc-8fd1-de3492b59bca-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/23f08320-e0a1-42bc-8fd1-de3492b59bca-istiod-ca-cert\") pod \"23f08320-e0a1-42bc-8fd1-de3492b59bca\" (UID: \"23f08320-e0a1-42bc-8fd1-de3492b59bca\") " Oct 30 21:25:56.493948 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:25:56.493835 5378 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/23f08320-e0a1-42bc-8fd1-de3492b59bca-kube-api-access-phqsd\" (UniqueName: \"kubernetes.io/projected/23f08320-e0a1-42bc-8fd1-de3492b59bca-kube-api-access-phqsd\") pod \"23f08320-e0a1-42bc-8fd1-de3492b59bca\" (UID: \"23f08320-e0a1-42bc-8fd1-de3492b59bca\") " Oct 30 21:25:56.493948 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:25:56.493897 5378 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/23f08320-e0a1-42bc-8fd1-de3492b59bca-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/23f08320-e0a1-42bc-8fd1-de3492b59bca-credential-socket\") pod \"23f08320-e0a1-42bc-8fd1-de3492b59bca\" (UID: \"23f08320-e0a1-42bc-8fd1-de3492b59bca\") " Oct 30 21:25:56.494145 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:25:56.493976 5378 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/23f08320-e0a1-42bc-8fd1-de3492b59bca-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/23f08320-e0a1-42bc-8fd1-de3492b59bca-workload-socket\") pod \"23f08320-e0a1-42bc-8fd1-de3492b59bca\" (UID: \"23f08320-e0a1-42bc-8fd1-de3492b59bca\") " Oct 30 21:25:56.494145 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:25:56.494007 5378 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/23f08320-e0a1-42bc-8fd1-de3492b59bca-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/23f08320-e0a1-42bc-8fd1-de3492b59bca-workload-certs\") pod \"23f08320-e0a1-42bc-8fd1-de3492b59bca\" (UID: \"23f08320-e0a1-42bc-8fd1-de3492b59bca\") " Oct 30 21:25:56.494145 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:25:56.494050 5378 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/23f08320-e0a1-42bc-8fd1-de3492b59bca-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/23f08320-e0a1-42bc-8fd1-de3492b59bca-istio-data\") pod \"23f08320-e0a1-42bc-8fd1-de3492b59bca\" (UID: \"23f08320-e0a1-42bc-8fd1-de3492b59bca\") " Oct 30 21:25:56.494145 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:25:56.494082 5378 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/23f08320-e0a1-42bc-8fd1-de3492b59bca-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/23f08320-e0a1-42bc-8fd1-de3492b59bca-istio-envoy\") pod \"23f08320-e0a1-42bc-8fd1-de3492b59bca\" (UID: \"23f08320-e0a1-42bc-8fd1-de3492b59bca\") " Oct 30 21:25:56.494145 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:25:56.494121 5378 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/23f08320-e0a1-42bc-8fd1-de3492b59bca-istio-token\" (UniqueName: \"kubernetes.io/projected/23f08320-e0a1-42bc-8fd1-de3492b59bca-istio-token\") pod \"23f08320-e0a1-42bc-8fd1-de3492b59bca\" (UID: \"23f08320-e0a1-42bc-8fd1-de3492b59bca\") " Oct 30 21:25:56.494429 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:25:56.494344 5378 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/23f08320-e0a1-42bc-8fd1-de3492b59bca-istiod-ca-cert" pod "23f08320-e0a1-42bc-8fd1-de3492b59bca" (UID: "23f08320-e0a1-42bc-8fd1-de3492b59bca"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:25:56.494521 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:25:56.494487 5378 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/23f08320-e0a1-42bc-8fd1-de3492b59bca-istio-data" pod "23f08320-e0a1-42bc-8fd1-de3492b59bca" (UID: "23f08320-e0a1-42bc-8fd1-de3492b59bca"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:25:56.494521 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:25:56.494510 5378 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/23f08320-e0a1-42bc-8fd1-de3492b59bca-istiod-ca-cert\") on node \"ip-10-0-138-149.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:25:56.494673 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:25:56.494618 5378 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/23f08320-e0a1-42bc-8fd1-de3492b59bca-workload-socket" pod "23f08320-e0a1-42bc-8fd1-de3492b59bca" (UID: "23f08320-e0a1-42bc-8fd1-de3492b59bca"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:25:56.495209 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:25:56.494840 5378 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/23f08320-e0a1-42bc-8fd1-de3492b59bca-workload-certs" pod "23f08320-e0a1-42bc-8fd1-de3492b59bca" (UID: "23f08320-e0a1-42bc-8fd1-de3492b59bca"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:25:56.495209 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:25:56.495110 5378 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/23f08320-e0a1-42bc-8fd1-de3492b59bca-credential-socket" pod "23f08320-e0a1-42bc-8fd1-de3492b59bca" (UID: "23f08320-e0a1-42bc-8fd1-de3492b59bca"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:25:56.496689 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:25:56.496626 5378 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/23f08320-e0a1-42bc-8fd1-de3492b59bca-istio-podinfo" pod "23f08320-e0a1-42bc-8fd1-de3492b59bca" (UID: "23f08320-e0a1-42bc-8fd1-de3492b59bca"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 21:25:56.497046 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:25:56.497015 5378 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/23f08320-e0a1-42bc-8fd1-de3492b59bca-kube-api-access-phqsd" pod "23f08320-e0a1-42bc-8fd1-de3492b59bca" (UID: "23f08320-e0a1-42bc-8fd1-de3492b59bca"). InnerVolumeSpecName "kube-api-access-phqsd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:25:56.497249 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:25:56.497225 5378 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/23f08320-e0a1-42bc-8fd1-de3492b59bca-istio-token" pod "23f08320-e0a1-42bc-8fd1-de3492b59bca" (UID: "23f08320-e0a1-42bc-8fd1-de3492b59bca"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:25:56.497332 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:25:56.497251 5378 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/23f08320-e0a1-42bc-8fd1-de3492b59bca-istio-envoy" pod "23f08320-e0a1-42bc-8fd1-de3492b59bca" (UID: "23f08320-e0a1-42bc-8fd1-de3492b59bca"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:25:56.595202 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:25:56.595165 5378 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/23f08320-e0a1-42bc-8fd1-de3492b59bca-workload-socket\") on node \"ip-10-0-138-149.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:25:56.595202 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:25:56.595204 5378 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/23f08320-e0a1-42bc-8fd1-de3492b59bca-workload-certs\") on node \"ip-10-0-138-149.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:25:56.595488 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:25:56.595219 5378 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/23f08320-e0a1-42bc-8fd1-de3492b59bca-istio-data\") on node \"ip-10-0-138-149.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:25:56.595488 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:25:56.595232 5378 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/23f08320-e0a1-42bc-8fd1-de3492b59bca-istio-envoy\") on node \"ip-10-0-138-149.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:25:56.595488 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:25:56.595245 5378 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/23f08320-e0a1-42bc-8fd1-de3492b59bca-istio-token\") on node \"ip-10-0-138-149.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:25:56.595488 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:25:56.595258 5378 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/23f08320-e0a1-42bc-8fd1-de3492b59bca-istio-podinfo\") on node \"ip-10-0-138-149.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:25:56.595488 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:25:56.595272 5378 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-phqsd\" (UniqueName: \"kubernetes.io/projected/23f08320-e0a1-42bc-8fd1-de3492b59bca-kube-api-access-phqsd\") on node \"ip-10-0-138-149.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:25:56.595488 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:25:56.595286 5378 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/23f08320-e0a1-42bc-8fd1-de3492b59bca-credential-socket\") on node \"ip-10-0-138-149.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:25:57.214007 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:25:57.213971 5378 generic.go:356] "Generic (PLEG): container finished" podID="23f08320-e0a1-42bc-8fd1-de3492b59bca" containerID="3411ad24d40a29ed63e18fa10cf40588cbb83130efc1cf2232c5acd10a3a0303" exitCode=0 Oct 30 21:25:57.214450 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:25:57.214017 5378 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-5c45e5-openshift-default-9dcdd54d9-s697g" Oct 30 21:25:57.214450 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:25:57.214056 5378 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-5c45e5-openshift-default-9dcdd54d9-s697g" event={"ID":"23f08320-e0a1-42bc-8fd1-de3492b59bca","Type":"ContainerDied","Data":"3411ad24d40a29ed63e18fa10cf40588cbb83130efc1cf2232c5acd10a3a0303"} Oct 30 21:25:57.214450 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:25:57.214111 5378 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-5c45e5-openshift-default-9dcdd54d9-s697g" event={"ID":"23f08320-e0a1-42bc-8fd1-de3492b59bca","Type":"ContainerDied","Data":"d502d498f085e301c28891c26418385a3ab54683c5bf0ab01b57a2a67ffbc272"} Oct 30 21:25:57.214450 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:25:57.214135 5378 scope.go:122] "RemoveContainer" containerID="3411ad24d40a29ed63e18fa10cf40588cbb83130efc1cf2232c5acd10a3a0303" Oct 30 21:25:57.223518 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:25:57.223502 5378 scope.go:122] "RemoveContainer" containerID="565200d98f097d3eda2ed71305f48ca3f2478d53f968badf2cb34cdaa05da40a" Oct 30 21:25:57.231554 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:25:57.231538 5378 scope.go:122] "RemoveContainer" containerID="3411ad24d40a29ed63e18fa10cf40588cbb83130efc1cf2232c5acd10a3a0303" Oct 30 21:25:57.231827 ip-10-0-138-149 kubenswrapper[5378]: E1030 21:25:57.231808 5378 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3411ad24d40a29ed63e18fa10cf40588cbb83130efc1cf2232c5acd10a3a0303\": container with ID starting with 3411ad24d40a29ed63e18fa10cf40588cbb83130efc1cf2232c5acd10a3a0303 not found: ID does not exist" containerID="3411ad24d40a29ed63e18fa10cf40588cbb83130efc1cf2232c5acd10a3a0303" Oct 30 21:25:57.231886 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:25:57.231835 5378 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3411ad24d40a29ed63e18fa10cf40588cbb83130efc1cf2232c5acd10a3a0303"} err="failed to get container status \"3411ad24d40a29ed63e18fa10cf40588cbb83130efc1cf2232c5acd10a3a0303\": rpc error: code = NotFound desc = could not find container \"3411ad24d40a29ed63e18fa10cf40588cbb83130efc1cf2232c5acd10a3a0303\": container with ID starting with 3411ad24d40a29ed63e18fa10cf40588cbb83130efc1cf2232c5acd10a3a0303 not found: ID does not exist" Oct 30 21:25:57.231886 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:25:57.231851 5378 scope.go:122] "RemoveContainer" containerID="565200d98f097d3eda2ed71305f48ca3f2478d53f968badf2cb34cdaa05da40a" Oct 30 21:25:57.232078 ip-10-0-138-149 kubenswrapper[5378]: E1030 21:25:57.232057 5378 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"565200d98f097d3eda2ed71305f48ca3f2478d53f968badf2cb34cdaa05da40a\": container with ID starting with 565200d98f097d3eda2ed71305f48ca3f2478d53f968badf2cb34cdaa05da40a not found: ID does not exist" containerID="565200d98f097d3eda2ed71305f48ca3f2478d53f968badf2cb34cdaa05da40a" Oct 30 21:25:57.232132 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:25:57.232088 5378 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"565200d98f097d3eda2ed71305f48ca3f2478d53f968badf2cb34cdaa05da40a"} err="failed to get container status \"565200d98f097d3eda2ed71305f48ca3f2478d53f968badf2cb34cdaa05da40a\": rpc error: code = NotFound desc = could not find container \"565200d98f097d3eda2ed71305f48ca3f2478d53f968badf2cb34cdaa05da40a\": container with ID starting with 565200d98f097d3eda2ed71305f48ca3f2478d53f968badf2cb34cdaa05da40a not found: ID does not exist" Oct 30 21:25:57.429948 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:25:57.429912 5378 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-5c45e5-openshift-default-9dcdd54d9-s697g"] Oct 30 21:25:57.432941 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:25:57.432917 5378 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-5c45e5-openshift-default-9dcdd54d9-s697g"] Oct 30 21:25:57.739603 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:25:57.739555 5378 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="23f08320-e0a1-42bc-8fd1-de3492b59bca" path="/var/lib/kubelet/pods/23f08320-e0a1-42bc-8fd1-de3492b59bca/volumes" Oct 30 21:26:00.547813 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:26:00.547776 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:26:00.678905 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:26:00.678869 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:26:00.993511 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:26:00.993448 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:26:02.175447 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:26:02.175399 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:26:02.438431 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:26:02.438325 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:26:02.599870 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:26:02.599834 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:26:04.439302 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:26:04.439270 5378 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w4-88da25-5ddcdff7f9-5p4dg"] Oct 30 21:26:04.682025 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:26:04.681996 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:26:04.804184 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:26:04.804091 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:26:05.371743 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:26:05.371707 5378 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-88da25-785647f46c-rb4g9"] Oct 30 21:26:05.372034 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:26:05.371989 5378 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-88da25-785647f46c-rb4g9" podUID="a9c904b3-edb2-426a-ac6a-136a4e6afba0" containerName="maas-api" containerID="cri-o://fd82c3d6c0c0c909006f171457c846ce2daf7bce0596dc8d8a1cc553c44d7828" gracePeriod=30 Oct 30 21:26:05.465586 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:26:05.465527 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:26:05.638201 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:26:05.638174 5378 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-88da25-785647f46c-rb4g9" Oct 30 21:26:05.786450 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:26:05.786406 5378 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a9c904b3-edb2-426a-ac6a-136a4e6afba0-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a9c904b3-edb2-426a-ac6a-136a4e6afba0-openshift-service-ca\") pod \"a9c904b3-edb2-426a-ac6a-136a4e6afba0\" (UID: \"a9c904b3-edb2-426a-ac6a-136a4e6afba0\") " Oct 30 21:26:05.786450 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:26:05.786450 5378 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a9c904b3-edb2-426a-ac6a-136a4e6afba0-kube-api-access-pjxpp\" (UniqueName: \"kubernetes.io/projected/a9c904b3-edb2-426a-ac6a-136a4e6afba0-kube-api-access-pjxpp\") pod \"a9c904b3-edb2-426a-ac6a-136a4e6afba0\" (UID: \"a9c904b3-edb2-426a-ac6a-136a4e6afba0\") " Oct 30 21:26:05.786655 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:26:05.786476 5378 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a9c904b3-edb2-426a-ac6a-136a4e6afba0-metrics-certs\" (UniqueName: \"kubernetes.io/secret/a9c904b3-edb2-426a-ac6a-136a4e6afba0-metrics-certs\") pod \"a9c904b3-edb2-426a-ac6a-136a4e6afba0\" (UID: \"a9c904b3-edb2-426a-ac6a-136a4e6afba0\") " Oct 30 21:26:05.786655 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:26:05.786602 5378 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a9c904b3-edb2-426a-ac6a-136a4e6afba0-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a9c904b3-edb2-426a-ac6a-136a4e6afba0-maas-api-tls\") pod \"a9c904b3-edb2-426a-ac6a-136a4e6afba0\" (UID: \"a9c904b3-edb2-426a-ac6a-136a4e6afba0\") " Oct 30 21:26:05.786850 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:26:05.786821 5378 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a9c904b3-edb2-426a-ac6a-136a4e6afba0-openshift-service-ca" pod "a9c904b3-edb2-426a-ac6a-136a4e6afba0" (UID: "a9c904b3-edb2-426a-ac6a-136a4e6afba0"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:26:05.786986 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:26:05.786970 5378 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a9c904b3-edb2-426a-ac6a-136a4e6afba0-openshift-service-ca\") on node \"ip-10-0-138-149.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:26:05.788728 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:26:05.788698 5378 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a9c904b3-edb2-426a-ac6a-136a4e6afba0-kube-api-access-pjxpp" pod "a9c904b3-edb2-426a-ac6a-136a4e6afba0" (UID: "a9c904b3-edb2-426a-ac6a-136a4e6afba0"). InnerVolumeSpecName "kube-api-access-pjxpp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:26:05.788843 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:26:05.788720 5378 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a9c904b3-edb2-426a-ac6a-136a4e6afba0-maas-api-tls" pod "a9c904b3-edb2-426a-ac6a-136a4e6afba0" (UID: "a9c904b3-edb2-426a-ac6a-136a4e6afba0"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:26:05.788843 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:26:05.788698 5378 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a9c904b3-edb2-426a-ac6a-136a4e6afba0-metrics-certs" pod "a9c904b3-edb2-426a-ac6a-136a4e6afba0" (UID: "a9c904b3-edb2-426a-ac6a-136a4e6afba0"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:26:05.888021 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:26:05.887919 5378 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pjxpp\" (UniqueName: \"kubernetes.io/projected/a9c904b3-edb2-426a-ac6a-136a4e6afba0-kube-api-access-pjxpp\") on node \"ip-10-0-138-149.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:26:05.888021 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:26:05.887961 5378 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a9c904b3-edb2-426a-ac6a-136a4e6afba0-metrics-certs\") on node \"ip-10-0-138-149.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:26:05.888021 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:26:05.887976 5378 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a9c904b3-edb2-426a-ac6a-136a4e6afba0-maas-api-tls\") on node \"ip-10-0-138-149.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:26:06.262403 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:26:06.262319 5378 generic.go:356] "Generic (PLEG): container finished" podID="a9c904b3-edb2-426a-ac6a-136a4e6afba0" containerID="fd82c3d6c0c0c909006f171457c846ce2daf7bce0596dc8d8a1cc553c44d7828" exitCode=0 Oct 30 21:26:06.262403 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:26:06.262373 5378 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-88da25-785647f46c-rb4g9" Oct 30 21:26:06.262797 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:26:06.262406 5378 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-88da25-785647f46c-rb4g9" event={"ID":"a9c904b3-edb2-426a-ac6a-136a4e6afba0","Type":"ContainerDied","Data":"fd82c3d6c0c0c909006f171457c846ce2daf7bce0596dc8d8a1cc553c44d7828"} Oct 30 21:26:06.262797 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:26:06.262462 5378 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-88da25-785647f46c-rb4g9" event={"ID":"a9c904b3-edb2-426a-ac6a-136a4e6afba0","Type":"ContainerDied","Data":"b836ebafb6632969db44c4a01418d7df5d53ad86877279244cc6ee43c10294b3"} Oct 30 21:26:06.262797 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:26:06.262485 5378 scope.go:122] "RemoveContainer" containerID="fd82c3d6c0c0c909006f171457c846ce2daf7bce0596dc8d8a1cc553c44d7828" Oct 30 21:26:06.273459 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:26:06.273443 5378 scope.go:122] "RemoveContainer" containerID="fd82c3d6c0c0c909006f171457c846ce2daf7bce0596dc8d8a1cc553c44d7828" Oct 30 21:26:06.273747 ip-10-0-138-149 kubenswrapper[5378]: E1030 21:26:06.273726 5378 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fd82c3d6c0c0c909006f171457c846ce2daf7bce0596dc8d8a1cc553c44d7828\": container with ID starting with fd82c3d6c0c0c909006f171457c846ce2daf7bce0596dc8d8a1cc553c44d7828 not found: ID does not exist" containerID="fd82c3d6c0c0c909006f171457c846ce2daf7bce0596dc8d8a1cc553c44d7828" Oct 30 21:26:06.273829 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:26:06.273759 5378 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fd82c3d6c0c0c909006f171457c846ce2daf7bce0596dc8d8a1cc553c44d7828"} err="failed to get container status \"fd82c3d6c0c0c909006f171457c846ce2daf7bce0596dc8d8a1cc553c44d7828\": rpc error: code = NotFound desc = could not find container \"fd82c3d6c0c0c909006f171457c846ce2daf7bce0596dc8d8a1cc553c44d7828\": container with ID starting with fd82c3d6c0c0c909006f171457c846ce2daf7bce0596dc8d8a1cc553c44d7828 not found: ID does not exist" Oct 30 21:26:06.428559 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:26:06.428523 5378 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-88da25-785647f46c-rb4g9"] Oct 30 21:26:06.478891 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:26:06.478861 5378 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-88da25-785647f46c-rb4g9"] Oct 30 21:26:07.254305 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:26:07.254270 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:26:07.741484 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:26:07.741443 5378 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a9c904b3-edb2-426a-ac6a-136a4e6afba0" path="/var/lib/kubelet/pods/a9c904b3-edb2-426a-ac6a-136a4e6afba0/volumes" Oct 30 21:26:09.440404 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:26:09.440365 5378 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-88da25-5ddcdff7f9-5p4dg" podUID="c537a2c1-13ee-488d-98b8-e8195b60d2be" containerName="payload-pre-processing" containerID="cri-o://016794a990a4b6581264a8720f413a29daa35f7c022e18e8ef04bf21897e8c92" gracePeriod=25 Oct 30 21:26:09.699844 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:26:09.699787 5378 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-88da25-5ddcdff7f9-5p4dg" Oct 30 21:26:09.828363 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:26:09.828332 5378 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c537a2c1-13ee-488d-98b8-e8195b60d2be-kube-api-access-wdq75\" (UniqueName: \"kubernetes.io/projected/c537a2c1-13ee-488d-98b8-e8195b60d2be-kube-api-access-wdq75\") pod \"c537a2c1-13ee-488d-98b8-e8195b60d2be\" (UID: \"c537a2c1-13ee-488d-98b8-e8195b60d2be\") " Oct 30 21:26:09.828529 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:26:09.828472 5378 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c537a2c1-13ee-488d-98b8-e8195b60d2be-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c537a2c1-13ee-488d-98b8-e8195b60d2be-plugins-config-volume\") pod \"c537a2c1-13ee-488d-98b8-e8195b60d2be\" (UID: \"c537a2c1-13ee-488d-98b8-e8195b60d2be\") " Oct 30 21:26:09.828873 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:26:09.828847 5378 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c537a2c1-13ee-488d-98b8-e8195b60d2be-plugins-config-volume" pod "c537a2c1-13ee-488d-98b8-e8195b60d2be" (UID: "c537a2c1-13ee-488d-98b8-e8195b60d2be"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:26:09.830500 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:26:09.830473 5378 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c537a2c1-13ee-488d-98b8-e8195b60d2be-kube-api-access-wdq75" pod "c537a2c1-13ee-488d-98b8-e8195b60d2be" (UID: "c537a2c1-13ee-488d-98b8-e8195b60d2be"). InnerVolumeSpecName "kube-api-access-wdq75". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:26:09.929924 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:26:09.929892 5378 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c537a2c1-13ee-488d-98b8-e8195b60d2be-plugins-config-volume\") on node \"ip-10-0-138-149.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:26:09.929924 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:26:09.929921 5378 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wdq75\" (UniqueName: \"kubernetes.io/projected/c537a2c1-13ee-488d-98b8-e8195b60d2be-kube-api-access-wdq75\") on node \"ip-10-0-138-149.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:26:10.283064 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:26:10.283030 5378 generic.go:356] "Generic (PLEG): container finished" podID="c537a2c1-13ee-488d-98b8-e8195b60d2be" containerID="016794a990a4b6581264a8720f413a29daa35f7c022e18e8ef04bf21897e8c92" exitCode=0 Oct 30 21:26:10.283238 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:26:10.283079 5378 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-88da25-5ddcdff7f9-5p4dg" Oct 30 21:26:10.283238 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:26:10.283099 5378 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-88da25-5ddcdff7f9-5p4dg" event={"ID":"c537a2c1-13ee-488d-98b8-e8195b60d2be","Type":"ContainerDied","Data":"016794a990a4b6581264a8720f413a29daa35f7c022e18e8ef04bf21897e8c92"} Oct 30 21:26:10.283238 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:26:10.283151 5378 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-88da25-5ddcdff7f9-5p4dg" event={"ID":"c537a2c1-13ee-488d-98b8-e8195b60d2be","Type":"ContainerDied","Data":"58ec8fca92488331b2fb594fae4d1c4d6a489bee9a64174433d3d3f9009c14ff"} Oct 30 21:26:10.283238 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:26:10.283182 5378 scope.go:122] "RemoveContainer" containerID="016794a990a4b6581264a8720f413a29daa35f7c022e18e8ef04bf21897e8c92" Oct 30 21:26:10.291911 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:26:10.291894 5378 scope.go:122] "RemoveContainer" containerID="016794a990a4b6581264a8720f413a29daa35f7c022e18e8ef04bf21897e8c92" Oct 30 21:26:10.292206 ip-10-0-138-149 kubenswrapper[5378]: E1030 21:26:10.292183 5378 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"016794a990a4b6581264a8720f413a29daa35f7c022e18e8ef04bf21897e8c92\": container with ID starting with 016794a990a4b6581264a8720f413a29daa35f7c022e18e8ef04bf21897e8c92 not found: ID does not exist" containerID="016794a990a4b6581264a8720f413a29daa35f7c022e18e8ef04bf21897e8c92" Oct 30 21:26:10.292257 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:26:10.292214 5378 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"016794a990a4b6581264a8720f413a29daa35f7c022e18e8ef04bf21897e8c92"} err="failed to get container status \"016794a990a4b6581264a8720f413a29daa35f7c022e18e8ef04bf21897e8c92\": rpc error: code = NotFound desc = could not find container \"016794a990a4b6581264a8720f413a29daa35f7c022e18e8ef04bf21897e8c92\": container with ID starting with 016794a990a4b6581264a8720f413a29daa35f7c022e18e8ef04bf21897e8c92 not found: ID does not exist" Oct 30 21:26:10.452733 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:26:10.452693 5378 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w4-88da25-5ddcdff7f9-5p4dg"] Oct 30 21:26:10.503073 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:26:10.503044 5378 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w4-88da25-5ddcdff7f9-5p4dg"] Oct 30 21:26:10.973206 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:26:10.973171 5378 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-r9g59"] Oct 30 21:26:11.348507 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:26:11.348475 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:26:11.623532 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:26:11.623450 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:26:11.739845 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:26:11.739812 5378 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c537a2c1-13ee-488d-98b8-e8195b60d2be" path="/var/lib/kubelet/pods/c537a2c1-13ee-488d-98b8-e8195b60d2be/volumes" Oct 30 21:26:12.119785 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:26:12.119750 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:26:12.282849 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:26:12.282815 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:26:17.261336 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:26:17.261306 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:26:17.921417 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:26:17.921385 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:26:18.057432 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:26:18.057399 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:26:18.281600 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:26:18.281495 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:26:18.282490 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:26:18.282469 5378 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-88da25-openshift-default-6c4ccc4cd9-tcsdd"] Oct 30 21:26:18.282718 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:26:18.282684 5378 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w4-88da25-openshift-default-6c4ccc4cd9-tcsdd" podUID="191dc733-cb36-4efc-9572-e645e2eb59d8" containerName="istio-proxy" containerID="cri-o://b32bbef49274b5f0ba3e132a3db6b2019f999b73007e1ed694bd0b5142d741e9" gracePeriod=30 Oct 30 21:26:18.790277 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:26:18.790246 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:26:18.836520 ip-10-0-138-149 kubenswrapper[5378]: E1030 21:26:18.836484 5378 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q6mx7" containerName="extractor" Oct 30 21:26:22.392284 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:26:22.392252 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:26:22.825505 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:26:22.825416 5378 patch_prober.go:29] interesting pod/e2e-shared-b-w4-88da25-openshift-default-6c4ccc4cd9-tcsdd container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 21:26:22.825505 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:26:22.825478 5378 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-w4-88da25-openshift-default-6c4ccc4cd9-tcsdd" podUID="191dc733-cb36-4efc-9572-e645e2eb59d8" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 21:26:23.539573 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:26:23.539538 5378 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w4-88da25-openshift-default-6c4ccc4cd9-tcsdd" Oct 30 21:26:23.663098 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:26:23.663068 5378 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/191dc733-cb36-4efc-9572-e645e2eb59d8-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/191dc733-cb36-4efc-9572-e645e2eb59d8-workload-certs\") pod \"191dc733-cb36-4efc-9572-e645e2eb59d8\" (UID: \"191dc733-cb36-4efc-9572-e645e2eb59d8\") " Oct 30 21:26:23.663246 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:26:23.663111 5378 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/191dc733-cb36-4efc-9572-e645e2eb59d8-kube-api-access-6n2gm\" (UniqueName: \"kubernetes.io/projected/191dc733-cb36-4efc-9572-e645e2eb59d8-kube-api-access-6n2gm\") pod \"191dc733-cb36-4efc-9572-e645e2eb59d8\" (UID: \"191dc733-cb36-4efc-9572-e645e2eb59d8\") " Oct 30 21:26:23.663246 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:26:23.663146 5378 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/191dc733-cb36-4efc-9572-e645e2eb59d8-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/191dc733-cb36-4efc-9572-e645e2eb59d8-istiod-ca-cert\") pod \"191dc733-cb36-4efc-9572-e645e2eb59d8\" (UID: \"191dc733-cb36-4efc-9572-e645e2eb59d8\") " Oct 30 21:26:23.663246 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:26:23.663208 5378 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/191dc733-cb36-4efc-9572-e645e2eb59d8-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/191dc733-cb36-4efc-9572-e645e2eb59d8-workload-socket\") pod \"191dc733-cb36-4efc-9572-e645e2eb59d8\" (UID: \"191dc733-cb36-4efc-9572-e645e2eb59d8\") " Oct 30 21:26:23.663422 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:26:23.663356 5378 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/191dc733-cb36-4efc-9572-e645e2eb59d8-workload-certs" pod "191dc733-cb36-4efc-9572-e645e2eb59d8" (UID: "191dc733-cb36-4efc-9572-e645e2eb59d8"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:26:23.663422 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:26:23.663359 5378 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/191dc733-cb36-4efc-9572-e645e2eb59d8-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/191dc733-cb36-4efc-9572-e645e2eb59d8-istio-data\") pod \"191dc733-cb36-4efc-9572-e645e2eb59d8\" (UID: \"191dc733-cb36-4efc-9572-e645e2eb59d8\") " Oct 30 21:26:23.663541 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:26:23.663465 5378 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/191dc733-cb36-4efc-9572-e645e2eb59d8-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/191dc733-cb36-4efc-9572-e645e2eb59d8-istio-podinfo\") pod \"191dc733-cb36-4efc-9572-e645e2eb59d8\" (UID: \"191dc733-cb36-4efc-9572-e645e2eb59d8\") " Oct 30 21:26:23.663541 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:26:23.663518 5378 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/191dc733-cb36-4efc-9572-e645e2eb59d8-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/191dc733-cb36-4efc-9572-e645e2eb59d8-istio-envoy\") pod \"191dc733-cb36-4efc-9572-e645e2eb59d8\" (UID: \"191dc733-cb36-4efc-9572-e645e2eb59d8\") " Oct 30 21:26:23.663541 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:26:23.663534 5378 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/191dc733-cb36-4efc-9572-e645e2eb59d8-workload-socket" pod "191dc733-cb36-4efc-9572-e645e2eb59d8" (UID: "191dc733-cb36-4efc-9572-e645e2eb59d8"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:26:23.663758 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:26:23.663554 5378 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/191dc733-cb36-4efc-9572-e645e2eb59d8-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/191dc733-cb36-4efc-9572-e645e2eb59d8-credential-socket\") pod \"191dc733-cb36-4efc-9572-e645e2eb59d8\" (UID: \"191dc733-cb36-4efc-9572-e645e2eb59d8\") " Oct 30 21:26:23.663758 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:26:23.663638 5378 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/191dc733-cb36-4efc-9572-e645e2eb59d8-istio-token\" (UniqueName: \"kubernetes.io/projected/191dc733-cb36-4efc-9572-e645e2eb59d8-istio-token\") pod \"191dc733-cb36-4efc-9572-e645e2eb59d8\" (UID: \"191dc733-cb36-4efc-9572-e645e2eb59d8\") " Oct 30 21:26:23.663758 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:26:23.663554 5378 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/191dc733-cb36-4efc-9572-e645e2eb59d8-istio-data" pod "191dc733-cb36-4efc-9572-e645e2eb59d8" (UID: "191dc733-cb36-4efc-9572-e645e2eb59d8"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:26:23.663758 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:26:23.663603 5378 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/191dc733-cb36-4efc-9572-e645e2eb59d8-istiod-ca-cert" pod "191dc733-cb36-4efc-9572-e645e2eb59d8" (UID: "191dc733-cb36-4efc-9572-e645e2eb59d8"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:26:23.663758 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:26:23.663728 5378 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/191dc733-cb36-4efc-9572-e645e2eb59d8-credential-socket" pod "191dc733-cb36-4efc-9572-e645e2eb59d8" (UID: "191dc733-cb36-4efc-9572-e645e2eb59d8"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:26:23.664042 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:26:23.663969 5378 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/191dc733-cb36-4efc-9572-e645e2eb59d8-workload-certs\") on node \"ip-10-0-138-149.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:26:23.664042 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:26:23.663991 5378 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/191dc733-cb36-4efc-9572-e645e2eb59d8-istiod-ca-cert\") on node \"ip-10-0-138-149.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:26:23.664042 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:26:23.664006 5378 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/191dc733-cb36-4efc-9572-e645e2eb59d8-workload-socket\") on node \"ip-10-0-138-149.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:26:23.664042 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:26:23.664020 5378 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/191dc733-cb36-4efc-9572-e645e2eb59d8-istio-data\") on node \"ip-10-0-138-149.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:26:23.664042 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:26:23.664036 5378 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/191dc733-cb36-4efc-9572-e645e2eb59d8-credential-socket\") on node \"ip-10-0-138-149.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:26:23.665442 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:26:23.665421 5378 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/191dc733-cb36-4efc-9572-e645e2eb59d8-kube-api-access-6n2gm" pod "191dc733-cb36-4efc-9572-e645e2eb59d8" (UID: "191dc733-cb36-4efc-9572-e645e2eb59d8"). InnerVolumeSpecName "kube-api-access-6n2gm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:26:23.665789 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:26:23.665767 5378 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/191dc733-cb36-4efc-9572-e645e2eb59d8-istio-envoy" pod "191dc733-cb36-4efc-9572-e645e2eb59d8" (UID: "191dc733-cb36-4efc-9572-e645e2eb59d8"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:26:23.665958 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:26:23.665940 5378 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/191dc733-cb36-4efc-9572-e645e2eb59d8-istio-podinfo" pod "191dc733-cb36-4efc-9572-e645e2eb59d8" (UID: "191dc733-cb36-4efc-9572-e645e2eb59d8"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 21:26:23.666041 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:26:23.666019 5378 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/191dc733-cb36-4efc-9572-e645e2eb59d8-istio-token" pod "191dc733-cb36-4efc-9572-e645e2eb59d8" (UID: "191dc733-cb36-4efc-9572-e645e2eb59d8"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:26:23.763393 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:26:23.763362 5378 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-1fdaa7/distinct-w4-1fdaa7-kserve-86dc9774b9-6x2gf"] Oct 30 21:26:23.765268 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:26:23.765241 5378 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/191dc733-cb36-4efc-9572-e645e2eb59d8-istio-podinfo\") on node \"ip-10-0-138-149.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:26:23.765268 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:26:23.765265 5378 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/191dc733-cb36-4efc-9572-e645e2eb59d8-istio-envoy\") on node \"ip-10-0-138-149.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:26:23.765485 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:26:23.765277 5378 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/191dc733-cb36-4efc-9572-e645e2eb59d8-istio-token\") on node \"ip-10-0-138-149.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:26:23.765485 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:26:23.765286 5378 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6n2gm\" (UniqueName: \"kubernetes.io/projected/191dc733-cb36-4efc-9572-e645e2eb59d8-kube-api-access-6n2gm\") on node \"ip-10-0-138-149.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:26:23.770591 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:26:23.770554 5378 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-1fdaa7/premium-w4-1fdaa7-kserve-55dffbd5cd-gvpxq"] Oct 30 21:26:23.776273 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:26:23.776247 5378 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-1fdaa7/simulator-w4-1fdaa7-kserve-9866d468c-fwmsk"] Oct 30 21:26:23.850724 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:26:23.850691 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:26:24.351773 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:26:24.351742 5378 generic.go:356] "Generic (PLEG): container finished" podID="191dc733-cb36-4efc-9572-e645e2eb59d8" containerID="b32bbef49274b5f0ba3e132a3db6b2019f999b73007e1ed694bd0b5142d741e9" exitCode=0 Oct 30 21:26:24.351977 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:26:24.351806 5378 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w4-88da25-openshift-default-6c4ccc4cd9-tcsdd" Oct 30 21:26:24.351977 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:26:24.351828 5378 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-88da25-openshift-default-6c4ccc4cd9-tcsdd" event={"ID":"191dc733-cb36-4efc-9572-e645e2eb59d8","Type":"ContainerDied","Data":"b32bbef49274b5f0ba3e132a3db6b2019f999b73007e1ed694bd0b5142d741e9"} Oct 30 21:26:24.351977 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:26:24.351923 5378 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-88da25-openshift-default-6c4ccc4cd9-tcsdd" event={"ID":"191dc733-cb36-4efc-9572-e645e2eb59d8","Type":"ContainerDied","Data":"965a8bf649a8e1af9ad3a6bcb746a8efed57d36598d839e817767c74c016f274"} Oct 30 21:26:24.351977 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:26:24.351955 5378 scope.go:122] "RemoveContainer" containerID="b32bbef49274b5f0ba3e132a3db6b2019f999b73007e1ed694bd0b5142d741e9" Oct 30 21:26:24.360843 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:26:24.360826 5378 scope.go:122] "RemoveContainer" containerID="345ba2ed9062b536d084e4b4475ec52a5838cdb19de04968cd6e6b8f301aa257" Oct 30 21:26:24.368514 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:26:24.368498 5378 scope.go:122] "RemoveContainer" containerID="b32bbef49274b5f0ba3e132a3db6b2019f999b73007e1ed694bd0b5142d741e9" Oct 30 21:26:24.368850 ip-10-0-138-149 kubenswrapper[5378]: E1030 21:26:24.368833 5378 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b32bbef49274b5f0ba3e132a3db6b2019f999b73007e1ed694bd0b5142d741e9\": container with ID starting with b32bbef49274b5f0ba3e132a3db6b2019f999b73007e1ed694bd0b5142d741e9 not found: ID does not exist" containerID="b32bbef49274b5f0ba3e132a3db6b2019f999b73007e1ed694bd0b5142d741e9" Oct 30 21:26:24.368925 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:26:24.368859 5378 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b32bbef49274b5f0ba3e132a3db6b2019f999b73007e1ed694bd0b5142d741e9"} err="failed to get container status \"b32bbef49274b5f0ba3e132a3db6b2019f999b73007e1ed694bd0b5142d741e9\": rpc error: code = NotFound desc = could not find container \"b32bbef49274b5f0ba3e132a3db6b2019f999b73007e1ed694bd0b5142d741e9\": container with ID starting with b32bbef49274b5f0ba3e132a3db6b2019f999b73007e1ed694bd0b5142d741e9 not found: ID does not exist" Oct 30 21:26:24.368925 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:26:24.368875 5378 scope.go:122] "RemoveContainer" containerID="345ba2ed9062b536d084e4b4475ec52a5838cdb19de04968cd6e6b8f301aa257" Oct 30 21:26:24.369126 ip-10-0-138-149 kubenswrapper[5378]: E1030 21:26:24.369108 5378 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"345ba2ed9062b536d084e4b4475ec52a5838cdb19de04968cd6e6b8f301aa257\": container with ID starting with 345ba2ed9062b536d084e4b4475ec52a5838cdb19de04968cd6e6b8f301aa257 not found: ID does not exist" containerID="345ba2ed9062b536d084e4b4475ec52a5838cdb19de04968cd6e6b8f301aa257" Oct 30 21:26:24.369165 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:26:24.369137 5378 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"345ba2ed9062b536d084e4b4475ec52a5838cdb19de04968cd6e6b8f301aa257"} err="failed to get container status \"345ba2ed9062b536d084e4b4475ec52a5838cdb19de04968cd6e6b8f301aa257\": rpc error: code = NotFound desc = could not find container \"345ba2ed9062b536d084e4b4475ec52a5838cdb19de04968cd6e6b8f301aa257\": container with ID starting with 345ba2ed9062b536d084e4b4475ec52a5838cdb19de04968cd6e6b8f301aa257 not found: ID does not exist" Oct 30 21:26:24.518493 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:26:24.518460 5378 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-88da25-openshift-default-6c4ccc4cd9-tcsdd"] Oct 30 21:26:24.569176 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:26:24.569143 5378 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-88da25-openshift-default-6c4ccc4cd9-tcsdd"] Oct 30 21:26:25.740078 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:26:25.740041 5378 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="191dc733-cb36-4efc-9572-e645e2eb59d8" path="/var/lib/kubelet/pods/191dc733-cb36-4efc-9572-e645e2eb59d8/volumes" Oct 30 21:26:27.245820 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:26:27.245777 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:26:31.143706 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:26:31.143669 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:26:31.263253 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:26:31.263182 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:26:32.064737 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:26:32.064689 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:26:38.783529 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:26:38.783447 5378 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w4-1fdaa7/distinct-w4-1fdaa7-kserve-86dc9774b9-6x2gf" podUID="2e162c95-0527-4cb3-85a3-795aae212d29" containerName="main" containerID="cri-o://1af171b9a3e007ba20860d03d532aace90ca2406d47479ca5ab185b23f57d478" gracePeriod=45 Oct 30 21:26:38.789728 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:26:38.789678 5378 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w4-1fdaa7/premium-w4-1fdaa7-kserve-55dffbd5cd-gvpxq" podUID="ec282a4b-742f-4382-a296-335dcd2e9636" containerName="main" containerID="cri-o://582c981270282cf4b1bf2ee8310ecda4e6f18fde9ace8f30167166f6d9be40cb" gracePeriod=45 Oct 30 21:26:38.794885 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:26:38.794856 5378 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w4-1fdaa7/simulator-w4-1fdaa7-kserve-9866d468c-fwmsk" podUID="09219832-010e-4ba8-bfb9-5a2884b358e1" containerName="main" containerID="cri-o://6be1583f151b9b2619c98618b8a94907756a9c4ca205c56dc206922b30ce5998" gracePeriod=45 Oct 30 21:26:38.836901 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:26:38.836870 5378 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-1fdaa7/simulator-w4-1fdaa7-kserve-9866d468c-fwmsk" podUID="09219832-010e-4ba8-bfb9-5a2884b358e1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": dial tcp 10.133.0.55:8000: connect: connection refused" Oct 30 21:26:38.838163 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:26:38.838136 5378 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-1fdaa7/premium-w4-1fdaa7-kserve-55dffbd5cd-gvpxq" podUID="ec282a4b-742f-4382-a296-335dcd2e9636" containerName="main" probeResult="failure" output="Get \"https://10.133.0.59:8000/ready\": dial tcp 10.133.0.59:8000: connect: connection refused" Oct 30 21:26:39.060056 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:26:39.060026 5378 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-1fdaa7/distinct-w4-1fdaa7-kserve-86dc9774b9-6x2gf" Oct 30 21:26:39.064010 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:26:39.063981 5378 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-1fdaa7/premium-w4-1fdaa7-kserve-55dffbd5cd-gvpxq" Oct 30 21:26:39.092964 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:26:39.092934 5378 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-1fdaa7/simulator-w4-1fdaa7-kserve-9866d468c-fwmsk" Oct 30 21:26:39.209516 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:26:39.209486 5378 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2e162c95-0527-4cb3-85a3-795aae212d29-dshm\" (UniqueName: \"kubernetes.io/empty-dir/2e162c95-0527-4cb3-85a3-795aae212d29-dshm\") pod \"2e162c95-0527-4cb3-85a3-795aae212d29\" (UID: \"2e162c95-0527-4cb3-85a3-795aae212d29\") " Oct 30 21:26:39.209725 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:26:39.209608 5378 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2e162c95-0527-4cb3-85a3-795aae212d29-kube-api-access-fq5tf\" (UniqueName: \"kubernetes.io/projected/2e162c95-0527-4cb3-85a3-795aae212d29-kube-api-access-fq5tf\") pod \"2e162c95-0527-4cb3-85a3-795aae212d29\" (UID: \"2e162c95-0527-4cb3-85a3-795aae212d29\") " Oct 30 21:26:39.209725 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:26:39.209630 5378 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2e162c95-0527-4cb3-85a3-795aae212d29-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2e162c95-0527-4cb3-85a3-795aae212d29-model-cache\") pod \"2e162c95-0527-4cb3-85a3-795aae212d29\" (UID: \"2e162c95-0527-4cb3-85a3-795aae212d29\") " Oct 30 21:26:39.209725 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:26:39.209652 5378 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/09219832-010e-4ba8-bfb9-5a2884b358e1-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/09219832-010e-4ba8-bfb9-5a2884b358e1-model-cache\") pod \"09219832-010e-4ba8-bfb9-5a2884b358e1\" (UID: \"09219832-010e-4ba8-bfb9-5a2884b358e1\") " Oct 30 21:26:39.209725 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:26:39.209680 5378 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ec282a4b-742f-4382-a296-335dcd2e9636-tls-certs\" (UniqueName: \"kubernetes.io/secret/ec282a4b-742f-4382-a296-335dcd2e9636-tls-certs\") pod \"ec282a4b-742f-4382-a296-335dcd2e9636\" (UID: \"ec282a4b-742f-4382-a296-335dcd2e9636\") " Oct 30 21:26:39.210096 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:26:39.210069 5378 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2e162c95-0527-4cb3-85a3-795aae212d29-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2e162c95-0527-4cb3-85a3-795aae212d29-tmp-dir\") pod \"2e162c95-0527-4cb3-85a3-795aae212d29\" (UID: \"2e162c95-0527-4cb3-85a3-795aae212d29\") " Oct 30 21:26:39.210159 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:26:39.210104 5378 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2e162c95-0527-4cb3-85a3-795aae212d29-home\" (UniqueName: \"kubernetes.io/empty-dir/2e162c95-0527-4cb3-85a3-795aae212d29-home\") pod \"2e162c95-0527-4cb3-85a3-795aae212d29\" (UID: \"2e162c95-0527-4cb3-85a3-795aae212d29\") " Oct 30 21:26:39.210159 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:26:39.210136 5378 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/09219832-010e-4ba8-bfb9-5a2884b358e1-home\" (UniqueName: \"kubernetes.io/empty-dir/09219832-010e-4ba8-bfb9-5a2884b358e1-home\") pod \"09219832-010e-4ba8-bfb9-5a2884b358e1\" (UID: \"09219832-010e-4ba8-bfb9-5a2884b358e1\") " Oct 30 21:26:39.210264 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:26:39.210171 5378 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ec282a4b-742f-4382-a296-335dcd2e9636-kube-api-access-2qfhl\" (UniqueName: \"kubernetes.io/projected/ec282a4b-742f-4382-a296-335dcd2e9636-kube-api-access-2qfhl\") pod \"ec282a4b-742f-4382-a296-335dcd2e9636\" (UID: \"ec282a4b-742f-4382-a296-335dcd2e9636\") " Oct 30 21:26:39.210264 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:26:39.210209 5378 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ec282a4b-742f-4382-a296-335dcd2e9636-dshm\" (UniqueName: \"kubernetes.io/empty-dir/ec282a4b-742f-4382-a296-335dcd2e9636-dshm\") pod \"ec282a4b-742f-4382-a296-335dcd2e9636\" (UID: \"ec282a4b-742f-4382-a296-335dcd2e9636\") " Oct 30 21:26:39.210384 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:26:39.210249 5378 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/09219832-010e-4ba8-bfb9-5a2884b358e1-kube-api-access-bxm9w\" (UniqueName: \"kubernetes.io/projected/09219832-010e-4ba8-bfb9-5a2884b358e1-kube-api-access-bxm9w\") pod \"09219832-010e-4ba8-bfb9-5a2884b358e1\" (UID: \"09219832-010e-4ba8-bfb9-5a2884b358e1\") " Oct 30 21:26:39.210384 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:26:39.210316 5378 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2e162c95-0527-4cb3-85a3-795aae212d29-tls-certs\" (UniqueName: \"kubernetes.io/secret/2e162c95-0527-4cb3-85a3-795aae212d29-tls-certs\") pod \"2e162c95-0527-4cb3-85a3-795aae212d29\" (UID: \"2e162c95-0527-4cb3-85a3-795aae212d29\") " Oct 30 21:26:39.210503 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:26:39.210382 5378 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ec282a4b-742f-4382-a296-335dcd2e9636-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ec282a4b-742f-4382-a296-335dcd2e9636-tmp-dir\") pod \"ec282a4b-742f-4382-a296-335dcd2e9636\" (UID: \"ec282a4b-742f-4382-a296-335dcd2e9636\") " Oct 30 21:26:39.210503 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:26:39.210419 5378 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/09219832-010e-4ba8-bfb9-5a2884b358e1-dshm\" (UniqueName: \"kubernetes.io/empty-dir/09219832-010e-4ba8-bfb9-5a2884b358e1-dshm\") pod \"09219832-010e-4ba8-bfb9-5a2884b358e1\" (UID: \"09219832-010e-4ba8-bfb9-5a2884b358e1\") " Oct 30 21:26:39.210655 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:26:39.209922 5378 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/09219832-010e-4ba8-bfb9-5a2884b358e1-model-cache" pod "09219832-010e-4ba8-bfb9-5a2884b358e1" (UID: "09219832-010e-4ba8-bfb9-5a2884b358e1"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:26:39.210655 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:26:39.209927 5378 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2e162c95-0527-4cb3-85a3-795aae212d29-model-cache" pod "2e162c95-0527-4cb3-85a3-795aae212d29" (UID: "2e162c95-0527-4cb3-85a3-795aae212d29"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:26:39.210800 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:26:39.210300 5378 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2e162c95-0527-4cb3-85a3-795aae212d29-tmp-dir" pod "2e162c95-0527-4cb3-85a3-795aae212d29" (UID: "2e162c95-0527-4cb3-85a3-795aae212d29"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:26:39.210800 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:26:39.210396 5378 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2e162c95-0527-4cb3-85a3-795aae212d29-home" pod "2e162c95-0527-4cb3-85a3-795aae212d29" (UID: "2e162c95-0527-4cb3-85a3-795aae212d29"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:26:39.210800 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:26:39.210440 5378 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/09219832-010e-4ba8-bfb9-5a2884b358e1-home" pod "09219832-010e-4ba8-bfb9-5a2884b358e1" (UID: "09219832-010e-4ba8-bfb9-5a2884b358e1"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:26:39.210800 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:26:39.210686 5378 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ec282a4b-742f-4382-a296-335dcd2e9636-tmp-dir" pod "ec282a4b-742f-4382-a296-335dcd2e9636" (UID: "ec282a4b-742f-4382-a296-335dcd2e9636"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:26:39.211023 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:26:39.210953 5378 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/09219832-010e-4ba8-bfb9-5a2884b358e1-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/09219832-010e-4ba8-bfb9-5a2884b358e1-tmp-dir\") pod \"09219832-010e-4ba8-bfb9-5a2884b358e1\" (UID: \"09219832-010e-4ba8-bfb9-5a2884b358e1\") " Oct 30 21:26:39.211084 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:26:39.211023 5378 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/09219832-010e-4ba8-bfb9-5a2884b358e1-tls-certs\" (UniqueName: \"kubernetes.io/secret/09219832-010e-4ba8-bfb9-5a2884b358e1-tls-certs\") pod \"09219832-010e-4ba8-bfb9-5a2884b358e1\" (UID: \"09219832-010e-4ba8-bfb9-5a2884b358e1\") " Oct 30 21:26:39.211084 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:26:39.211060 5378 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ec282a4b-742f-4382-a296-335dcd2e9636-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ec282a4b-742f-4382-a296-335dcd2e9636-model-cache\") pod \"ec282a4b-742f-4382-a296-335dcd2e9636\" (UID: \"ec282a4b-742f-4382-a296-335dcd2e9636\") " Oct 30 21:26:39.211186 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:26:39.211091 5378 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ec282a4b-742f-4382-a296-335dcd2e9636-home\" (UniqueName: \"kubernetes.io/empty-dir/ec282a4b-742f-4382-a296-335dcd2e9636-home\") pod \"ec282a4b-742f-4382-a296-335dcd2e9636\" (UID: \"ec282a4b-742f-4382-a296-335dcd2e9636\") " Oct 30 21:26:39.211309 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:26:39.211288 5378 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/09219832-010e-4ba8-bfb9-5a2884b358e1-tmp-dir" pod "09219832-010e-4ba8-bfb9-5a2884b358e1" (UID: "09219832-010e-4ba8-bfb9-5a2884b358e1"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:26:39.211483 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:26:39.211465 5378 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2e162c95-0527-4cb3-85a3-795aae212d29-model-cache\") on node \"ip-10-0-138-149.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:26:39.211592 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:26:39.211479 5378 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ec282a4b-742f-4382-a296-335dcd2e9636-home" pod "ec282a4b-742f-4382-a296-335dcd2e9636" (UID: "ec282a4b-742f-4382-a296-335dcd2e9636"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:26:39.211592 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:26:39.211490 5378 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/09219832-010e-4ba8-bfb9-5a2884b358e1-model-cache\") on node \"ip-10-0-138-149.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:26:39.211592 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:26:39.211506 5378 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2e162c95-0527-4cb3-85a3-795aae212d29-tmp-dir\") on node \"ip-10-0-138-149.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:26:39.211592 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:26:39.211520 5378 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2e162c95-0527-4cb3-85a3-795aae212d29-home\") on node \"ip-10-0-138-149.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:26:39.211592 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:26:39.211533 5378 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/09219832-010e-4ba8-bfb9-5a2884b358e1-home\") on node \"ip-10-0-138-149.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:26:39.211592 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:26:39.211546 5378 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ec282a4b-742f-4382-a296-335dcd2e9636-tmp-dir\") on node \"ip-10-0-138-149.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:26:39.211592 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:26:39.211580 5378 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/09219832-010e-4ba8-bfb9-5a2884b358e1-tmp-dir\") on node \"ip-10-0-138-149.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:26:39.211994 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:26:39.211788 5378 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ec282a4b-742f-4382-a296-335dcd2e9636-model-cache" pod "ec282a4b-742f-4382-a296-335dcd2e9636" (UID: "ec282a4b-742f-4382-a296-335dcd2e9636"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:26:39.213754 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:26:39.213726 5378 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/09219832-010e-4ba8-bfb9-5a2884b358e1-dshm" pod "09219832-010e-4ba8-bfb9-5a2884b358e1" (UID: "09219832-010e-4ba8-bfb9-5a2884b358e1"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:26:39.214054 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:26:39.213773 5378 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ec282a4b-742f-4382-a296-335dcd2e9636-tls-certs" pod "ec282a4b-742f-4382-a296-335dcd2e9636" (UID: "ec282a4b-742f-4382-a296-335dcd2e9636"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:26:39.214054 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:26:39.213791 5378 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2e162c95-0527-4cb3-85a3-795aae212d29-dshm" pod "2e162c95-0527-4cb3-85a3-795aae212d29" (UID: "2e162c95-0527-4cb3-85a3-795aae212d29"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:26:39.214054 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:26:39.213835 5378 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2e162c95-0527-4cb3-85a3-795aae212d29-kube-api-access-fq5tf" pod "2e162c95-0527-4cb3-85a3-795aae212d29" (UID: "2e162c95-0527-4cb3-85a3-795aae212d29"). InnerVolumeSpecName "kube-api-access-fq5tf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:26:39.215189 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:26:39.215164 5378 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/09219832-010e-4ba8-bfb9-5a2884b358e1-kube-api-access-bxm9w" pod "09219832-010e-4ba8-bfb9-5a2884b358e1" (UID: "09219832-010e-4ba8-bfb9-5a2884b358e1"). InnerVolumeSpecName "kube-api-access-bxm9w". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:26:39.215303 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:26:39.215270 5378 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2e162c95-0527-4cb3-85a3-795aae212d29-tls-certs" pod "2e162c95-0527-4cb3-85a3-795aae212d29" (UID: "2e162c95-0527-4cb3-85a3-795aae212d29"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:26:39.215303 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:26:39.215273 5378 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/09219832-010e-4ba8-bfb9-5a2884b358e1-tls-certs" pod "09219832-010e-4ba8-bfb9-5a2884b358e1" (UID: "09219832-010e-4ba8-bfb9-5a2884b358e1"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:26:39.215447 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:26:39.215328 5378 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ec282a4b-742f-4382-a296-335dcd2e9636-dshm" pod "ec282a4b-742f-4382-a296-335dcd2e9636" (UID: "ec282a4b-742f-4382-a296-335dcd2e9636"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:26:39.216799 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:26:39.216776 5378 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ec282a4b-742f-4382-a296-335dcd2e9636-kube-api-access-2qfhl" pod "ec282a4b-742f-4382-a296-335dcd2e9636" (UID: "ec282a4b-742f-4382-a296-335dcd2e9636"). InnerVolumeSpecName "kube-api-access-2qfhl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:26:39.312376 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:26:39.312294 5378 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ec282a4b-742f-4382-a296-335dcd2e9636-home\") on node \"ip-10-0-138-149.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:26:39.312376 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:26:39.312321 5378 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2e162c95-0527-4cb3-85a3-795aae212d29-dshm\") on node \"ip-10-0-138-149.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:26:39.312376 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:26:39.312331 5378 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fq5tf\" (UniqueName: \"kubernetes.io/projected/2e162c95-0527-4cb3-85a3-795aae212d29-kube-api-access-fq5tf\") on node \"ip-10-0-138-149.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:26:39.312376 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:26:39.312342 5378 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ec282a4b-742f-4382-a296-335dcd2e9636-tls-certs\") on node \"ip-10-0-138-149.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:26:39.312376 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:26:39.312352 5378 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2qfhl\" (UniqueName: \"kubernetes.io/projected/ec282a4b-742f-4382-a296-335dcd2e9636-kube-api-access-2qfhl\") on node \"ip-10-0-138-149.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:26:39.312376 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:26:39.312360 5378 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ec282a4b-742f-4382-a296-335dcd2e9636-dshm\") on node \"ip-10-0-138-149.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:26:39.312376 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:26:39.312371 5378 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bxm9w\" (UniqueName: \"kubernetes.io/projected/09219832-010e-4ba8-bfb9-5a2884b358e1-kube-api-access-bxm9w\") on node \"ip-10-0-138-149.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:26:39.312376 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:26:39.312379 5378 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2e162c95-0527-4cb3-85a3-795aae212d29-tls-certs\") on node \"ip-10-0-138-149.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:26:39.312792 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:26:39.312389 5378 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/09219832-010e-4ba8-bfb9-5a2884b358e1-dshm\") on node \"ip-10-0-138-149.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:26:39.312792 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:26:39.312398 5378 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/09219832-010e-4ba8-bfb9-5a2884b358e1-tls-certs\") on node \"ip-10-0-138-149.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:26:39.312792 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:26:39.312405 5378 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ec282a4b-742f-4382-a296-335dcd2e9636-model-cache\") on node \"ip-10-0-138-149.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:26:39.426312 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:26:39.426280 5378 generic.go:356] "Generic (PLEG): container finished" podID="ec282a4b-742f-4382-a296-335dcd2e9636" containerID="582c981270282cf4b1bf2ee8310ecda4e6f18fde9ace8f30167166f6d9be40cb" exitCode=0 Oct 30 21:26:39.426507 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:26:39.426327 5378 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-1fdaa7/premium-w4-1fdaa7-kserve-55dffbd5cd-gvpxq" Oct 30 21:26:39.426507 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:26:39.426385 5378 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-1fdaa7/premium-w4-1fdaa7-kserve-55dffbd5cd-gvpxq" event={"ID":"ec282a4b-742f-4382-a296-335dcd2e9636","Type":"ContainerDied","Data":"582c981270282cf4b1bf2ee8310ecda4e6f18fde9ace8f30167166f6d9be40cb"} Oct 30 21:26:39.426507 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:26:39.426439 5378 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-1fdaa7/premium-w4-1fdaa7-kserve-55dffbd5cd-gvpxq" event={"ID":"ec282a4b-742f-4382-a296-335dcd2e9636","Type":"ContainerDied","Data":"f455a19947696c1ad00004267e6fde85cdaa293e6c9291f8698ec4178792ed5a"} Oct 30 21:26:39.426507 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:26:39.426468 5378 scope.go:122] "RemoveContainer" containerID="582c981270282cf4b1bf2ee8310ecda4e6f18fde9ace8f30167166f6d9be40cb" Oct 30 21:26:39.427964 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:26:39.427940 5378 generic.go:356] "Generic (PLEG): container finished" podID="09219832-010e-4ba8-bfb9-5a2884b358e1" containerID="6be1583f151b9b2619c98618b8a94907756a9c4ca205c56dc206922b30ce5998" exitCode=0 Oct 30 21:26:39.428065 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:26:39.428043 5378 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-1fdaa7/simulator-w4-1fdaa7-kserve-9866d468c-fwmsk" event={"ID":"09219832-010e-4ba8-bfb9-5a2884b358e1","Type":"ContainerDied","Data":"6be1583f151b9b2619c98618b8a94907756a9c4ca205c56dc206922b30ce5998"} Oct 30 21:26:39.428138 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:26:39.428076 5378 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-1fdaa7/simulator-w4-1fdaa7-kserve-9866d468c-fwmsk" event={"ID":"09219832-010e-4ba8-bfb9-5a2884b358e1","Type":"ContainerDied","Data":"d1292e3c66c3d98af24e6c68f24d992a1a06c1982f6b391d6a7ac03d12b445f8"} Oct 30 21:26:39.428201 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:26:39.428143 5378 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-1fdaa7/simulator-w4-1fdaa7-kserve-9866d468c-fwmsk" Oct 30 21:26:39.432155 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:26:39.432123 5378 generic.go:356] "Generic (PLEG): container finished" podID="2e162c95-0527-4cb3-85a3-795aae212d29" containerID="1af171b9a3e007ba20860d03d532aace90ca2406d47479ca5ab185b23f57d478" exitCode=0 Oct 30 21:26:39.432282 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:26:39.432173 5378 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-1fdaa7/distinct-w4-1fdaa7-kserve-86dc9774b9-6x2gf" event={"ID":"2e162c95-0527-4cb3-85a3-795aae212d29","Type":"ContainerDied","Data":"1af171b9a3e007ba20860d03d532aace90ca2406d47479ca5ab185b23f57d478"} Oct 30 21:26:39.432282 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:26:39.432202 5378 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-1fdaa7/distinct-w4-1fdaa7-kserve-86dc9774b9-6x2gf" event={"ID":"2e162c95-0527-4cb3-85a3-795aae212d29","Type":"ContainerDied","Data":"3bfb60bdda754fc1a736253ae1a6ec39cbb9a14594806ba8447c256d96e2e264"} Oct 30 21:26:39.432282 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:26:39.432265 5378 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-1fdaa7/distinct-w4-1fdaa7-kserve-86dc9774b9-6x2gf" Oct 30 21:26:39.438786 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:26:39.438764 5378 scope.go:122] "RemoveContainer" containerID="582c981270282cf4b1bf2ee8310ecda4e6f18fde9ace8f30167166f6d9be40cb" Oct 30 21:26:39.439093 ip-10-0-138-149 kubenswrapper[5378]: E1030 21:26:39.439068 5378 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"582c981270282cf4b1bf2ee8310ecda4e6f18fde9ace8f30167166f6d9be40cb\": container with ID starting with 582c981270282cf4b1bf2ee8310ecda4e6f18fde9ace8f30167166f6d9be40cb not found: ID does not exist" containerID="582c981270282cf4b1bf2ee8310ecda4e6f18fde9ace8f30167166f6d9be40cb" Oct 30 21:26:39.439190 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:26:39.439102 5378 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"582c981270282cf4b1bf2ee8310ecda4e6f18fde9ace8f30167166f6d9be40cb"} err="failed to get container status \"582c981270282cf4b1bf2ee8310ecda4e6f18fde9ace8f30167166f6d9be40cb\": rpc error: code = NotFound desc = could not find container \"582c981270282cf4b1bf2ee8310ecda4e6f18fde9ace8f30167166f6d9be40cb\": container with ID starting with 582c981270282cf4b1bf2ee8310ecda4e6f18fde9ace8f30167166f6d9be40cb not found: ID does not exist" Oct 30 21:26:39.439190 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:26:39.439129 5378 scope.go:122] "RemoveContainer" containerID="6be1583f151b9b2619c98618b8a94907756a9c4ca205c56dc206922b30ce5998" Oct 30 21:26:39.449474 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:26:39.449456 5378 scope.go:122] "RemoveContainer" containerID="6be1583f151b9b2619c98618b8a94907756a9c4ca205c56dc206922b30ce5998" Oct 30 21:26:39.449770 ip-10-0-138-149 kubenswrapper[5378]: E1030 21:26:39.449752 5378 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6be1583f151b9b2619c98618b8a94907756a9c4ca205c56dc206922b30ce5998\": container with ID starting with 6be1583f151b9b2619c98618b8a94907756a9c4ca205c56dc206922b30ce5998 not found: ID does not exist" containerID="6be1583f151b9b2619c98618b8a94907756a9c4ca205c56dc206922b30ce5998" Oct 30 21:26:39.449823 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:26:39.449778 5378 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6be1583f151b9b2619c98618b8a94907756a9c4ca205c56dc206922b30ce5998"} err="failed to get container status \"6be1583f151b9b2619c98618b8a94907756a9c4ca205c56dc206922b30ce5998\": rpc error: code = NotFound desc = could not find container \"6be1583f151b9b2619c98618b8a94907756a9c4ca205c56dc206922b30ce5998\": container with ID starting with 6be1583f151b9b2619c98618b8a94907756a9c4ca205c56dc206922b30ce5998 not found: ID does not exist" Oct 30 21:26:39.449823 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:26:39.449793 5378 scope.go:122] "RemoveContainer" containerID="1af171b9a3e007ba20860d03d532aace90ca2406d47479ca5ab185b23f57d478" Oct 30 21:26:39.457197 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:26:39.457180 5378 scope.go:122] "RemoveContainer" containerID="1af171b9a3e007ba20860d03d532aace90ca2406d47479ca5ab185b23f57d478" Oct 30 21:26:39.457502 ip-10-0-138-149 kubenswrapper[5378]: E1030 21:26:39.457483 5378 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1af171b9a3e007ba20860d03d532aace90ca2406d47479ca5ab185b23f57d478\": container with ID starting with 1af171b9a3e007ba20860d03d532aace90ca2406d47479ca5ab185b23f57d478 not found: ID does not exist" containerID="1af171b9a3e007ba20860d03d532aace90ca2406d47479ca5ab185b23f57d478" Oct 30 21:26:39.457617 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:26:39.457506 5378 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1af171b9a3e007ba20860d03d532aace90ca2406d47479ca5ab185b23f57d478"} err="failed to get container status \"1af171b9a3e007ba20860d03d532aace90ca2406d47479ca5ab185b23f57d478\": rpc error: code = NotFound desc = could not find container \"1af171b9a3e007ba20860d03d532aace90ca2406d47479ca5ab185b23f57d478\": container with ID starting with 1af171b9a3e007ba20860d03d532aace90ca2406d47479ca5ab185b23f57d478 not found: ID does not exist" Oct 30 21:26:39.591263 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:26:39.591181 5378 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-1fdaa7/distinct-w4-1fdaa7-kserve-86dc9774b9-6x2gf"] Oct 30 21:26:39.592853 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:26:39.592830 5378 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w4-1fdaa7/distinct-w4-1fdaa7-kserve-86dc9774b9-6x2gf"] Oct 30 21:26:39.702812 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:26:39.702779 5378 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-1fdaa7/simulator-w4-1fdaa7-kserve-9866d468c-fwmsk"] Oct 30 21:26:39.706597 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:26:39.706555 5378 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w4-1fdaa7/simulator-w4-1fdaa7-kserve-9866d468c-fwmsk"] Oct 30 21:26:39.747602 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:26:39.747557 5378 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="09219832-010e-4ba8-bfb9-5a2884b358e1" path="/var/lib/kubelet/pods/09219832-010e-4ba8-bfb9-5a2884b358e1/volumes" Oct 30 21:26:39.747985 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:26:39.747972 5378 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2e162c95-0527-4cb3-85a3-795aae212d29" path="/var/lib/kubelet/pods/2e162c95-0527-4cb3-85a3-795aae212d29/volumes" Oct 30 21:26:39.816632 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:26:39.816604 5378 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-1fdaa7/premium-w4-1fdaa7-kserve-55dffbd5cd-gvpxq"] Oct 30 21:26:39.822910 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:26:39.822888 5378 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w4-1fdaa7/premium-w4-1fdaa7-kserve-55dffbd5cd-gvpxq"] Oct 30 21:26:41.740510 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:26:41.740477 5378 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ec282a4b-742f-4382-a296-335dcd2e9636" path="/var/lib/kubelet/pods/ec282a4b-742f-4382-a296-335dcd2e9636/volumes" Oct 30 21:26:48.835359 ip-10-0-138-149 kubenswrapper[5378]: E1030 21:26:48.835322 5378 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q6mx7" containerName="extractor" Oct 30 21:26:51.739413 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:26:51.739374 5378 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-1fdaa7-547f998845-pckbc"] Oct 30 21:26:51.739910 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:26:51.739607 5378 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-1fdaa7-547f998845-pckbc" podUID="c0b8557d-4625-4dc6-a769-e03f2d95c684" containerName="maas-api" containerID="cri-o://32f99068fbd0e980e08f61e39cf04ab37074e01a9924799fce5b86401365823e" gracePeriod=30 Oct 30 21:26:51.816865 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:26:51.816827 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:26:51.997802 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:26:51.997749 5378 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-1fdaa7-547f998845-pckbc" Oct 30 21:26:52.038431 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:26:52.038397 5378 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c0b8557d-4625-4dc6-a769-e03f2d95c684-kube-api-access-pxlzl\" (UniqueName: \"kubernetes.io/projected/c0b8557d-4625-4dc6-a769-e03f2d95c684-kube-api-access-pxlzl\") pod \"c0b8557d-4625-4dc6-a769-e03f2d95c684\" (UID: \"c0b8557d-4625-4dc6-a769-e03f2d95c684\") " Oct 30 21:26:52.038603 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:26:52.038590 5378 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c0b8557d-4625-4dc6-a769-e03f2d95c684-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c0b8557d-4625-4dc6-a769-e03f2d95c684-openshift-service-ca\") pod \"c0b8557d-4625-4dc6-a769-e03f2d95c684\" (UID: \"c0b8557d-4625-4dc6-a769-e03f2d95c684\") " Oct 30 21:26:52.038669 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:26:52.038612 5378 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c0b8557d-4625-4dc6-a769-e03f2d95c684-metrics-certs\" (UniqueName: \"kubernetes.io/secret/c0b8557d-4625-4dc6-a769-e03f2d95c684-metrics-certs\") pod \"c0b8557d-4625-4dc6-a769-e03f2d95c684\" (UID: \"c0b8557d-4625-4dc6-a769-e03f2d95c684\") " Oct 30 21:26:52.038669 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:26:52.038641 5378 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c0b8557d-4625-4dc6-a769-e03f2d95c684-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c0b8557d-4625-4dc6-a769-e03f2d95c684-maas-api-tls\") pod \"c0b8557d-4625-4dc6-a769-e03f2d95c684\" (UID: \"c0b8557d-4625-4dc6-a769-e03f2d95c684\") " Oct 30 21:26:52.038952 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:26:52.038930 5378 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c0b8557d-4625-4dc6-a769-e03f2d95c684-openshift-service-ca" pod "c0b8557d-4625-4dc6-a769-e03f2d95c684" (UID: "c0b8557d-4625-4dc6-a769-e03f2d95c684"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:26:52.040724 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:26:52.040700 5378 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c0b8557d-4625-4dc6-a769-e03f2d95c684-maas-api-tls" pod "c0b8557d-4625-4dc6-a769-e03f2d95c684" (UID: "c0b8557d-4625-4dc6-a769-e03f2d95c684"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:26:52.040724 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:26:52.040707 5378 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c0b8557d-4625-4dc6-a769-e03f2d95c684-kube-api-access-pxlzl" pod "c0b8557d-4625-4dc6-a769-e03f2d95c684" (UID: "c0b8557d-4625-4dc6-a769-e03f2d95c684"). InnerVolumeSpecName "kube-api-access-pxlzl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:26:52.040865 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:26:52.040712 5378 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c0b8557d-4625-4dc6-a769-e03f2d95c684-metrics-certs" pod "c0b8557d-4625-4dc6-a769-e03f2d95c684" (UID: "c0b8557d-4625-4dc6-a769-e03f2d95c684"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:26:52.139668 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:26:52.139632 5378 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pxlzl\" (UniqueName: \"kubernetes.io/projected/c0b8557d-4625-4dc6-a769-e03f2d95c684-kube-api-access-pxlzl\") on node \"ip-10-0-138-149.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:26:52.139668 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:26:52.139666 5378 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c0b8557d-4625-4dc6-a769-e03f2d95c684-openshift-service-ca\") on node \"ip-10-0-138-149.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:26:52.139868 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:26:52.139684 5378 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c0b8557d-4625-4dc6-a769-e03f2d95c684-metrics-certs\") on node \"ip-10-0-138-149.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:26:52.139868 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:26:52.139698 5378 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c0b8557d-4625-4dc6-a769-e03f2d95c684-maas-api-tls\") on node \"ip-10-0-138-149.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:26:52.497029 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:26:52.496992 5378 generic.go:356] "Generic (PLEG): container finished" podID="c0b8557d-4625-4dc6-a769-e03f2d95c684" containerID="32f99068fbd0e980e08f61e39cf04ab37074e01a9924799fce5b86401365823e" exitCode=0 Oct 30 21:26:52.497029 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:26:52.497035 5378 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-1fdaa7-547f998845-pckbc" event={"ID":"c0b8557d-4625-4dc6-a769-e03f2d95c684","Type":"ContainerDied","Data":"32f99068fbd0e980e08f61e39cf04ab37074e01a9924799fce5b86401365823e"} Oct 30 21:26:52.497291 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:26:52.497050 5378 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-1fdaa7-547f998845-pckbc" Oct 30 21:26:52.497291 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:26:52.497073 5378 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-1fdaa7-547f998845-pckbc" event={"ID":"c0b8557d-4625-4dc6-a769-e03f2d95c684","Type":"ContainerDied","Data":"bc45038a976bd0d96c825fc779d6ea6b14ed3ecb52b62ee041f9abdfc6b2b2da"} Oct 30 21:26:52.497291 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:26:52.497094 5378 scope.go:122] "RemoveContainer" containerID="32f99068fbd0e980e08f61e39cf04ab37074e01a9924799fce5b86401365823e" Oct 30 21:26:52.504580 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:26:52.504545 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:26:52.507071 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:26:52.507045 5378 scope.go:122] "RemoveContainer" containerID="32f99068fbd0e980e08f61e39cf04ab37074e01a9924799fce5b86401365823e" Oct 30 21:26:52.507325 ip-10-0-138-149 kubenswrapper[5378]: E1030 21:26:52.507304 5378 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"32f99068fbd0e980e08f61e39cf04ab37074e01a9924799fce5b86401365823e\": container with ID starting with 32f99068fbd0e980e08f61e39cf04ab37074e01a9924799fce5b86401365823e not found: ID does not exist" containerID="32f99068fbd0e980e08f61e39cf04ab37074e01a9924799fce5b86401365823e" Oct 30 21:26:52.507384 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:26:52.507334 5378 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"32f99068fbd0e980e08f61e39cf04ab37074e01a9924799fce5b86401365823e"} err="failed to get container status \"32f99068fbd0e980e08f61e39cf04ab37074e01a9924799fce5b86401365823e\": rpc error: code = NotFound desc = could not find container \"32f99068fbd0e980e08f61e39cf04ab37074e01a9924799fce5b86401365823e\": container with ID starting with 32f99068fbd0e980e08f61e39cf04ab37074e01a9924799fce5b86401365823e not found: ID does not exist" Oct 30 21:26:52.663743 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:26:52.663684 5378 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-1fdaa7-547f998845-pckbc"] Oct 30 21:26:52.668395 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:26:52.668339 5378 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-r9g59"] Oct 30 21:26:52.670006 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:26:52.669983 5378 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-1fdaa7-547f998845-pckbc"] Oct 30 21:26:53.329985 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:26:53.329948 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:26:53.681176 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:26:53.681128 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:26:53.740013 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:26:53.739978 5378 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c0b8557d-4625-4dc6-a769-e03f2d95c684" path="/var/lib/kubelet/pods/c0b8557d-4625-4dc6-a769-e03f2d95c684/volumes" Oct 30 21:26:54.303802 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:26:54.303740 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:27:00.197829 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:27:00.197785 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:27:00.325351 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:27:00.325314 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:27:01.280643 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:27:01.280611 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:27:01.413729 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:27:01.413692 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:27:01.685470 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:27:01.685439 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:27:01.838249 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:27:01.838211 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:27:06.925874 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:27:06.925832 5378 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-6cb6cd7964-gj5sw"] Oct 30 21:27:06.926315 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:27:06.926103 5378 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/maas-controller-6cb6cd7964-gj5sw" podUID="62138235-e57b-42e6-a892-67123e9b2478" containerName="manager" containerID="cri-o://b1900f549a152d00785af325909d9c6af17e2c07763d7433baf6aabcbaa0e3a3" gracePeriod=30 Oct 30 21:27:07.185636 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:27:07.185584 5378 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-6cb6cd7964-gj5sw" Oct 30 21:27:07.286998 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:27:07.286966 5378 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/62138235-e57b-42e6-a892-67123e9b2478-kube-api-access-2g2kz\" (UniqueName: \"kubernetes.io/projected/62138235-e57b-42e6-a892-67123e9b2478-kube-api-access-2g2kz\") pod \"62138235-e57b-42e6-a892-67123e9b2478\" (UID: \"62138235-e57b-42e6-a892-67123e9b2478\") " Oct 30 21:27:07.287159 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:27:07.287018 5378 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/62138235-e57b-42e6-a892-67123e9b2478-webhook-cert\" (UniqueName: \"kubernetes.io/secret/62138235-e57b-42e6-a892-67123e9b2478-webhook-cert\") pod \"62138235-e57b-42e6-a892-67123e9b2478\" (UID: \"62138235-e57b-42e6-a892-67123e9b2478\") " Oct 30 21:27:07.287159 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:27:07.287122 5378 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/62138235-e57b-42e6-a892-67123e9b2478-metrics-certs\" (UniqueName: \"kubernetes.io/secret/62138235-e57b-42e6-a892-67123e9b2478-metrics-certs\") pod \"62138235-e57b-42e6-a892-67123e9b2478\" (UID: \"62138235-e57b-42e6-a892-67123e9b2478\") " Oct 30 21:27:07.289123 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:27:07.289091 5378 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/62138235-e57b-42e6-a892-67123e9b2478-metrics-certs" pod "62138235-e57b-42e6-a892-67123e9b2478" (UID: "62138235-e57b-42e6-a892-67123e9b2478"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:27:07.289230 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:27:07.289150 5378 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/62138235-e57b-42e6-a892-67123e9b2478-webhook-cert" pod "62138235-e57b-42e6-a892-67123e9b2478" (UID: "62138235-e57b-42e6-a892-67123e9b2478"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:27:07.289230 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:27:07.289189 5378 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/62138235-e57b-42e6-a892-67123e9b2478-kube-api-access-2g2kz" pod "62138235-e57b-42e6-a892-67123e9b2478" (UID: "62138235-e57b-42e6-a892-67123e9b2478"). InnerVolumeSpecName "kube-api-access-2g2kz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:27:07.388658 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:27:07.388627 5378 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2g2kz\" (UniqueName: \"kubernetes.io/projected/62138235-e57b-42e6-a892-67123e9b2478-kube-api-access-2g2kz\") on node \"ip-10-0-138-149.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:27:07.388658 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:27:07.388657 5378 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/62138235-e57b-42e6-a892-67123e9b2478-webhook-cert\") on node \"ip-10-0-138-149.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:27:07.388947 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:27:07.388668 5378 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/62138235-e57b-42e6-a892-67123e9b2478-metrics-certs\") on node \"ip-10-0-138-149.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:27:07.565618 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:27:07.565516 5378 generic.go:356] "Generic (PLEG): container finished" podID="62138235-e57b-42e6-a892-67123e9b2478" containerID="b1900f549a152d00785af325909d9c6af17e2c07763d7433baf6aabcbaa0e3a3" exitCode=0 Oct 30 21:27:07.565618 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:27:07.565580 5378 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-6cb6cd7964-gj5sw" Oct 30 21:27:07.565618 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:27:07.565605 5378 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-6cb6cd7964-gj5sw" event={"ID":"62138235-e57b-42e6-a892-67123e9b2478","Type":"ContainerDied","Data":"b1900f549a152d00785af325909d9c6af17e2c07763d7433baf6aabcbaa0e3a3"} Oct 30 21:27:07.565871 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:27:07.565657 5378 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-6cb6cd7964-gj5sw" event={"ID":"62138235-e57b-42e6-a892-67123e9b2478","Type":"ContainerDied","Data":"6871633649c7b49e7f98cd3ed6e811492c31f93f811d77cec9173906ead6c3ef"} Oct 30 21:27:07.565871 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:27:07.565681 5378 scope.go:122] "RemoveContainer" containerID="b1900f549a152d00785af325909d9c6af17e2c07763d7433baf6aabcbaa0e3a3" Oct 30 21:27:07.576484 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:27:07.576468 5378 scope.go:122] "RemoveContainer" containerID="b1900f549a152d00785af325909d9c6af17e2c07763d7433baf6aabcbaa0e3a3" Oct 30 21:27:07.576825 ip-10-0-138-149 kubenswrapper[5378]: E1030 21:27:07.576804 5378 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b1900f549a152d00785af325909d9c6af17e2c07763d7433baf6aabcbaa0e3a3\": container with ID starting with b1900f549a152d00785af325909d9c6af17e2c07763d7433baf6aabcbaa0e3a3 not found: ID does not exist" containerID="b1900f549a152d00785af325909d9c6af17e2c07763d7433baf6aabcbaa0e3a3" Oct 30 21:27:07.576906 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:27:07.576837 5378 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b1900f549a152d00785af325909d9c6af17e2c07763d7433baf6aabcbaa0e3a3"} err="failed to get container status \"b1900f549a152d00785af325909d9c6af17e2c07763d7433baf6aabcbaa0e3a3\": rpc error: code = NotFound desc = could not find container \"b1900f549a152d00785af325909d9c6af17e2c07763d7433baf6aabcbaa0e3a3\": container with ID starting with b1900f549a152d00785af325909d9c6af17e2c07763d7433baf6aabcbaa0e3a3 not found: ID does not exist" Oct 30 21:27:07.652699 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:27:07.652647 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:27:07.665807 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:27:07.665782 5378 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-r9g59"] Oct 30 21:27:07.732227 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:27:07.732182 5378 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-6cb6cd7964-gj5sw"] Oct 30 21:27:07.739832 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:27:07.739807 5378 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-6cb6cd7964-gj5sw"] Oct 30 21:27:08.609041 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:27:08.609010 5378 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-6cb6cd7964-lgjgv"] Oct 30 21:27:08.609435 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:27:08.609375 5378 cpu_manager.go:414] "containerMap: removing container" podUID="f409f469-71fc-445b-bdc8-b8e84a4d1dc6" containerName="main" Oct 30 21:27:08.609435 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:27:08.609387 5378 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f409f469-71fc-445b-bdc8-b8e84a4d1dc6" containerName="main" Oct 30 21:27:08.609435 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:27:08.609398 5378 cpu_manager.go:414] "containerMap: removing container" podUID="d848cad5-c34e-4841-a2a0-e3aab4a69029" containerName="istio-proxy" Oct 30 21:27:08.609435 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:27:08.609403 5378 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d848cad5-c34e-4841-a2a0-e3aab4a69029" containerName="istio-proxy" Oct 30 21:27:08.609435 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:27:08.609415 5378 cpu_manager.go:414] "containerMap: removing container" podUID="ec282a4b-742f-4382-a296-335dcd2e9636" containerName="main" Oct 30 21:27:08.609435 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:27:08.609420 5378 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ec282a4b-742f-4382-a296-335dcd2e9636" containerName="main" Oct 30 21:27:08.609435 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:27:08.609437 5378 cpu_manager.go:414] "containerMap: removing container" podUID="52b8c06c-c948-452c-b317-523f09618599" containerName="payload-pre-processing" Oct 30 21:27:08.609696 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:27:08.609443 5378 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="52b8c06c-c948-452c-b317-523f09618599" containerName="payload-pre-processing" Oct 30 21:27:08.609696 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:27:08.609450 5378 cpu_manager.go:414] "containerMap: removing container" podUID="6614eb26-439d-464e-a7c2-b95744976905" containerName="istio-proxy" Oct 30 21:27:08.609696 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:27:08.609455 5378 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6614eb26-439d-464e-a7c2-b95744976905" containerName="istio-proxy" Oct 30 21:27:08.609696 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:27:08.609463 5378 cpu_manager.go:414] "containerMap: removing container" podUID="6351b682-cca8-4a28-bd5f-9cb3fd38085b" containerName="payload-processing" Oct 30 21:27:08.609696 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:27:08.609468 5378 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6351b682-cca8-4a28-bd5f-9cb3fd38085b" containerName="payload-processing" Oct 30 21:27:08.609696 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:27:08.609477 5378 cpu_manager.go:414] "containerMap: removing container" podUID="75677fa4-95cf-4a72-bced-a11307c20ca0" containerName="payload-processing" Oct 30 21:27:08.609696 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:27:08.609482 5378 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="75677fa4-95cf-4a72-bced-a11307c20ca0" containerName="payload-processing" Oct 30 21:27:08.609696 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:27:08.609488 5378 cpu_manager.go:414] "containerMap: removing container" podUID="09219832-010e-4ba8-bfb9-5a2884b358e1" containerName="main" Oct 30 21:27:08.609696 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:27:08.609493 5378 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="09219832-010e-4ba8-bfb9-5a2884b358e1" containerName="main" Oct 30 21:27:08.609696 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:27:08.609499 5378 cpu_manager.go:414] "containerMap: removing container" podUID="191dc733-cb36-4efc-9572-e645e2eb59d8" containerName="istio-proxy" Oct 30 21:27:08.609696 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:27:08.609505 5378 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="191dc733-cb36-4efc-9572-e645e2eb59d8" containerName="istio-proxy" Oct 30 21:27:08.609696 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:27:08.609515 5378 cpu_manager.go:414] "containerMap: removing container" podUID="04d5e742-36a9-41a8-9bb7-c9c8808d74cc" containerName="istio-proxy" Oct 30 21:27:08.609696 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:27:08.609520 5378 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="04d5e742-36a9-41a8-9bb7-c9c8808d74cc" containerName="istio-proxy" Oct 30 21:27:08.609696 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:27:08.609526 5378 cpu_manager.go:414] "containerMap: removing container" podUID="23f08320-e0a1-42bc-8fd1-de3492b59bca" containerName="istio-proxy" Oct 30 21:27:08.609696 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:27:08.609530 5378 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="23f08320-e0a1-42bc-8fd1-de3492b59bca" containerName="istio-proxy" Oct 30 21:27:08.609696 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:27:08.609538 5378 cpu_manager.go:414] "containerMap: removing container" podUID="a6c2ebb4-9a45-4038-87a1-a40a242ec833" containerName="payload-processing" Oct 30 21:27:08.609696 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:27:08.609542 5378 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a6c2ebb4-9a45-4038-87a1-a40a242ec833" containerName="payload-processing" Oct 30 21:27:08.609696 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:27:08.609548 5378 cpu_manager.go:414] "containerMap: removing container" podUID="ef431fc4-98f7-4a6f-85c1-5989dc6665f6" containerName="payload-pre-processing" Oct 30 21:27:08.609696 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:27:08.609553 5378 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ef431fc4-98f7-4a6f-85c1-5989dc6665f6" containerName="payload-pre-processing" Oct 30 21:27:08.609696 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:27:08.609559 5378 cpu_manager.go:414] "containerMap: removing container" podUID="fd6d4723-fe0b-4eae-94fb-c7b61de829c5" containerName="maas-api" Oct 30 21:27:08.609696 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:27:08.609589 5378 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fd6d4723-fe0b-4eae-94fb-c7b61de829c5" containerName="maas-api" Oct 30 21:27:08.609696 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:27:08.609597 5378 cpu_manager.go:414] "containerMap: removing container" podUID="d251ff08-e87c-4819-a8f9-4cc702bc75ac" containerName="maas-api" Oct 30 21:27:08.609696 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:27:08.609602 5378 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d251ff08-e87c-4819-a8f9-4cc702bc75ac" containerName="maas-api" Oct 30 21:27:08.609696 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:27:08.609607 5378 cpu_manager.go:414] "containerMap: removing container" podUID="8fdfc4fa-d488-4267-bfb0-113448a60fc2" containerName="istio-proxy" Oct 30 21:27:08.609696 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:27:08.609614 5378 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8fdfc4fa-d488-4267-bfb0-113448a60fc2" containerName="istio-proxy" Oct 30 21:27:08.609696 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:27:08.609622 5378 cpu_manager.go:414] "containerMap: removing container" podUID="c537a2c1-13ee-488d-98b8-e8195b60d2be" containerName="payload-pre-processing" Oct 30 21:27:08.609696 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:27:08.609626 5378 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c537a2c1-13ee-488d-98b8-e8195b60d2be" containerName="payload-pre-processing" Oct 30 21:27:08.609696 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:27:08.609635 5378 cpu_manager.go:414] "containerMap: removing container" podUID="0acdf79f-8312-4818-a1fb-8e50c7948634" containerName="maas-api" Oct 30 21:27:08.609696 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:27:08.609639 5378 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0acdf79f-8312-4818-a1fb-8e50c7948634" containerName="maas-api" Oct 30 21:27:08.609696 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:27:08.609645 5378 cpu_manager.go:414] "containerMap: removing container" podUID="e96f3b44-86e7-4389-a90f-e0fa736cffd1" containerName="main" Oct 30 21:27:08.609696 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:27:08.609649 5378 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e96f3b44-86e7-4389-a90f-e0fa736cffd1" containerName="main" Oct 30 21:27:08.609696 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:27:08.609655 5378 cpu_manager.go:414] "containerMap: removing container" podUID="a9c904b3-edb2-426a-ac6a-136a4e6afba0" containerName="maas-api" Oct 30 21:27:08.609696 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:27:08.609661 5378 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a9c904b3-edb2-426a-ac6a-136a4e6afba0" containerName="maas-api" Oct 30 21:27:08.609696 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:27:08.609670 5378 cpu_manager.go:414] "containerMap: removing container" podUID="2391a719-0c9b-4c56-b229-8f286a9aac96" containerName="istio-proxy" Oct 30 21:27:08.609696 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:27:08.609677 5378 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2391a719-0c9b-4c56-b229-8f286a9aac96" containerName="istio-proxy" Oct 30 21:27:08.609696 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:27:08.609686 5378 cpu_manager.go:414] "containerMap: removing container" podUID="2e162c95-0527-4cb3-85a3-795aae212d29" containerName="main" Oct 30 21:27:08.609696 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:27:08.609690 5378 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2e162c95-0527-4cb3-85a3-795aae212d29" containerName="main" Oct 30 21:27:08.609696 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:27:08.609698 5378 cpu_manager.go:414] "containerMap: removing container" podUID="484cba5f-b415-430a-b52d-50b1293be948" containerName="main" Oct 30 21:27:08.609696 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:27:08.609702 5378 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="484cba5f-b415-430a-b52d-50b1293be948" containerName="main" Oct 30 21:27:08.609696 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:27:08.609709 5378 cpu_manager.go:414] "containerMap: removing container" podUID="c5b6e9d2-ca83-4956-819e-d7ec7997acd8" containerName="istio-proxy" Oct 30 21:27:08.609696 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:27:08.609713 5378 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c5b6e9d2-ca83-4956-819e-d7ec7997acd8" containerName="istio-proxy" Oct 30 21:27:08.611200 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:27:08.609719 5378 cpu_manager.go:414] "containerMap: removing container" podUID="97e8a2e3-2092-4e54-8621-c33b9404e163" containerName="payload-processing" Oct 30 21:27:08.611200 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:27:08.609723 5378 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="97e8a2e3-2092-4e54-8621-c33b9404e163" containerName="payload-processing" Oct 30 21:27:08.611200 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:27:08.609729 5378 cpu_manager.go:414] "containerMap: removing container" podUID="bebe8fda-f6a0-4527-9798-2cb425757af6" containerName="istio-proxy" Oct 30 21:27:08.611200 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:27:08.609734 5378 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bebe8fda-f6a0-4527-9798-2cb425757af6" containerName="istio-proxy" Oct 30 21:27:08.611200 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:27:08.609746 5378 cpu_manager.go:414] "containerMap: removing container" podUID="62138235-e57b-42e6-a892-67123e9b2478" containerName="manager" Oct 30 21:27:08.611200 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:27:08.609752 5378 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="62138235-e57b-42e6-a892-67123e9b2478" containerName="manager" Oct 30 21:27:08.611200 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:27:08.609763 5378 cpu_manager.go:414] "containerMap: removing container" podUID="23f08320-e0a1-42bc-8fd1-de3492b59bca" containerName="istio-proxy" Oct 30 21:27:08.611200 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:27:08.609768 5378 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="23f08320-e0a1-42bc-8fd1-de3492b59bca" containerName="istio-proxy" Oct 30 21:27:08.611200 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:27:08.609774 5378 cpu_manager.go:414] "containerMap: removing container" podUID="191dc733-cb36-4efc-9572-e645e2eb59d8" containerName="istio-proxy" Oct 30 21:27:08.611200 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:27:08.609778 5378 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="191dc733-cb36-4efc-9572-e645e2eb59d8" containerName="istio-proxy" Oct 30 21:27:08.611200 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:27:08.609785 5378 cpu_manager.go:414] "containerMap: removing container" podUID="c0b8557d-4625-4dc6-a769-e03f2d95c684" containerName="maas-api" Oct 30 21:27:08.611200 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:27:08.609789 5378 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c0b8557d-4625-4dc6-a769-e03f2d95c684" containerName="maas-api" Oct 30 21:27:08.611200 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:27:08.609797 5378 cpu_manager.go:414] "containerMap: removing container" podUID="7bfe56ea-dd63-441e-a5c1-9b116f068419" containerName="istio-proxy" Oct 30 21:27:08.611200 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:27:08.609801 5378 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7bfe56ea-dd63-441e-a5c1-9b116f068419" containerName="istio-proxy" Oct 30 21:27:08.611200 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:27:08.609875 5378 memory_manager.go:365] "RemoveStaleState removing state" podUID="a6c2ebb4-9a45-4038-87a1-a40a242ec833" containerName="payload-processing" Oct 30 21:27:08.611200 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:27:08.609885 5378 memory_manager.go:365] "RemoveStaleState removing state" podUID="ef431fc4-98f7-4a6f-85c1-5989dc6665f6" containerName="payload-pre-processing" Oct 30 21:27:08.611200 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:27:08.609891 5378 memory_manager.go:365] "RemoveStaleState removing state" podUID="fd6d4723-fe0b-4eae-94fb-c7b61de829c5" containerName="maas-api" Oct 30 21:27:08.611200 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:27:08.609899 5378 memory_manager.go:365] "RemoveStaleState removing state" podUID="d251ff08-e87c-4819-a8f9-4cc702bc75ac" containerName="maas-api" Oct 30 21:27:08.611200 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:27:08.609908 5378 memory_manager.go:365] "RemoveStaleState removing state" podUID="8fdfc4fa-d488-4267-bfb0-113448a60fc2" containerName="istio-proxy" Oct 30 21:27:08.611200 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:27:08.609915 5378 memory_manager.go:365] "RemoveStaleState removing state" podUID="c537a2c1-13ee-488d-98b8-e8195b60d2be" containerName="payload-pre-processing" Oct 30 21:27:08.611200 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:27:08.609922 5378 memory_manager.go:365] "RemoveStaleState removing state" podUID="0acdf79f-8312-4818-a1fb-8e50c7948634" containerName="maas-api" Oct 30 21:27:08.611200 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:27:08.609927 5378 memory_manager.go:365] "RemoveStaleState removing state" podUID="e96f3b44-86e7-4389-a90f-e0fa736cffd1" containerName="main" Oct 30 21:27:08.611200 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:27:08.609932 5378 memory_manager.go:365] "RemoveStaleState removing state" podUID="a9c904b3-edb2-426a-ac6a-136a4e6afba0" containerName="maas-api" Oct 30 21:27:08.611200 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:27:08.609939 5378 memory_manager.go:365] "RemoveStaleState removing state" podUID="2391a719-0c9b-4c56-b229-8f286a9aac96" containerName="istio-proxy" Oct 30 21:27:08.611200 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:27:08.609944 5378 memory_manager.go:365] "RemoveStaleState removing state" podUID="2e162c95-0527-4cb3-85a3-795aae212d29" containerName="main" Oct 30 21:27:08.611200 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:27:08.609949 5378 memory_manager.go:365] "RemoveStaleState removing state" podUID="484cba5f-b415-430a-b52d-50b1293be948" containerName="main" Oct 30 21:27:08.611200 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:27:08.609955 5378 memory_manager.go:365] "RemoveStaleState removing state" podUID="c5b6e9d2-ca83-4956-819e-d7ec7997acd8" containerName="istio-proxy" Oct 30 21:27:08.611200 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:27:08.609960 5378 memory_manager.go:365] "RemoveStaleState removing state" podUID="97e8a2e3-2092-4e54-8621-c33b9404e163" containerName="payload-processing" Oct 30 21:27:08.611200 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:27:08.609965 5378 memory_manager.go:365] "RemoveStaleState removing state" podUID="bebe8fda-f6a0-4527-9798-2cb425757af6" containerName="istio-proxy" Oct 30 21:27:08.611200 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:27:08.609970 5378 memory_manager.go:365] "RemoveStaleState removing state" podUID="62138235-e57b-42e6-a892-67123e9b2478" containerName="manager" Oct 30 21:27:08.611200 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:27:08.609979 5378 memory_manager.go:365] "RemoveStaleState removing state" podUID="23f08320-e0a1-42bc-8fd1-de3492b59bca" containerName="istio-proxy" Oct 30 21:27:08.611200 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:27:08.609986 5378 memory_manager.go:365] "RemoveStaleState removing state" podUID="191dc733-cb36-4efc-9572-e645e2eb59d8" containerName="istio-proxy" Oct 30 21:27:08.611200 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:27:08.609990 5378 memory_manager.go:365] "RemoveStaleState removing state" podUID="c0b8557d-4625-4dc6-a769-e03f2d95c684" containerName="maas-api" Oct 30 21:27:08.611200 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:27:08.609999 5378 memory_manager.go:365] "RemoveStaleState removing state" podUID="6614eb26-439d-464e-a7c2-b95744976905" containerName="istio-proxy" Oct 30 21:27:08.611200 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:27:08.610004 5378 memory_manager.go:365] "RemoveStaleState removing state" podUID="7bfe56ea-dd63-441e-a5c1-9b116f068419" containerName="istio-proxy" Oct 30 21:27:08.611200 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:27:08.610009 5378 memory_manager.go:365] "RemoveStaleState removing state" podUID="f409f469-71fc-445b-bdc8-b8e84a4d1dc6" containerName="main" Oct 30 21:27:08.611200 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:27:08.610014 5378 memory_manager.go:365] "RemoveStaleState removing state" podUID="d848cad5-c34e-4841-a2a0-e3aab4a69029" containerName="istio-proxy" Oct 30 21:27:08.611200 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:27:08.610019 5378 memory_manager.go:365] "RemoveStaleState removing state" podUID="ec282a4b-742f-4382-a296-335dcd2e9636" containerName="main" Oct 30 21:27:08.611200 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:27:08.610024 5378 memory_manager.go:365] "RemoveStaleState removing state" podUID="52b8c06c-c948-452c-b317-523f09618599" containerName="payload-pre-processing" Oct 30 21:27:08.611200 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:27:08.610030 5378 memory_manager.go:365] "RemoveStaleState removing state" podUID="6614eb26-439d-464e-a7c2-b95744976905" containerName="istio-proxy" Oct 30 21:27:08.611200 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:27:08.610034 5378 memory_manager.go:365] "RemoveStaleState removing state" podUID="6351b682-cca8-4a28-bd5f-9cb3fd38085b" containerName="payload-processing" Oct 30 21:27:08.611200 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:27:08.610040 5378 memory_manager.go:365] "RemoveStaleState removing state" podUID="75677fa4-95cf-4a72-bced-a11307c20ca0" containerName="payload-processing" Oct 30 21:27:08.611200 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:27:08.610046 5378 memory_manager.go:365] "RemoveStaleState removing state" podUID="09219832-010e-4ba8-bfb9-5a2884b358e1" containerName="main" Oct 30 21:27:08.611200 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:27:08.610052 5378 memory_manager.go:365] "RemoveStaleState removing state" podUID="191dc733-cb36-4efc-9572-e645e2eb59d8" containerName="istio-proxy" Oct 30 21:27:08.611200 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:27:08.610061 5378 memory_manager.go:365] "RemoveStaleState removing state" podUID="04d5e742-36a9-41a8-9bb7-c9c8808d74cc" containerName="istio-proxy" Oct 30 21:27:08.614455 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:27:08.614438 5378 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-6cb6cd7964-lgjgv" Oct 30 21:27:08.702169 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:27:08.702140 5378 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xjf5n\" (UniqueName: \"kubernetes.io/projected/6e78dacd-7440-4904-ae86-7083302ea675-kube-api-access-xjf5n\") pod \"maas-controller-6cb6cd7964-lgjgv\" (UID: \"6e78dacd-7440-4904-ae86-7083302ea675\") " pod="opendatahub/maas-controller-6cb6cd7964-lgjgv" Oct 30 21:27:08.702169 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:27:08.702171 5378 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/6e78dacd-7440-4904-ae86-7083302ea675-webhook-cert\") pod \"maas-controller-6cb6cd7964-lgjgv\" (UID: \"6e78dacd-7440-4904-ae86-7083302ea675\") " pod="opendatahub/maas-controller-6cb6cd7964-lgjgv" Oct 30 21:27:08.702377 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:27:08.702204 5378 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6e78dacd-7440-4904-ae86-7083302ea675-metrics-certs\") pod \"maas-controller-6cb6cd7964-lgjgv\" (UID: \"6e78dacd-7440-4904-ae86-7083302ea675\") " pod="opendatahub/maas-controller-6cb6cd7964-lgjgv" Oct 30 21:27:08.718425 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:27:08.718397 5378 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\"" Oct 30 21:27:08.718591 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:27:08.718507 5378 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-kpvgd\"" Oct 30 21:27:08.718675 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:27:08.718615 5378 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\"" Oct 30 21:27:08.718675 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:27:08.718647 5378 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-metrics-tls\"" Oct 30 21:27:08.720485 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:27:08.720462 5378 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-6cb6cd7964-lgjgv"] Oct 30 21:27:08.803729 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:27:08.803688 5378 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xjf5n\" (UniqueName: \"kubernetes.io/projected/6e78dacd-7440-4904-ae86-7083302ea675-kube-api-access-xjf5n\") pod \"maas-controller-6cb6cd7964-lgjgv\" (UID: \"6e78dacd-7440-4904-ae86-7083302ea675\") " pod="opendatahub/maas-controller-6cb6cd7964-lgjgv" Oct 30 21:27:08.803729 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:27:08.803725 5378 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/6e78dacd-7440-4904-ae86-7083302ea675-webhook-cert\") pod \"maas-controller-6cb6cd7964-lgjgv\" (UID: \"6e78dacd-7440-4904-ae86-7083302ea675\") " pod="opendatahub/maas-controller-6cb6cd7964-lgjgv" Oct 30 21:27:08.804003 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:27:08.803771 5378 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6e78dacd-7440-4904-ae86-7083302ea675-metrics-certs\") pod \"maas-controller-6cb6cd7964-lgjgv\" (UID: \"6e78dacd-7440-4904-ae86-7083302ea675\") " pod="opendatahub/maas-controller-6cb6cd7964-lgjgv" Oct 30 21:27:08.806438 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:27:08.806402 5378 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6e78dacd-7440-4904-ae86-7083302ea675-metrics-certs\") pod \"maas-controller-6cb6cd7964-lgjgv\" (UID: \"6e78dacd-7440-4904-ae86-7083302ea675\") " pod="opendatahub/maas-controller-6cb6cd7964-lgjgv" Oct 30 21:27:08.806867 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:27:08.806843 5378 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/6e78dacd-7440-4904-ae86-7083302ea675-webhook-cert\") pod \"maas-controller-6cb6cd7964-lgjgv\" (UID: \"6e78dacd-7440-4904-ae86-7083302ea675\") " pod="opendatahub/maas-controller-6cb6cd7964-lgjgv" Oct 30 21:27:08.861497 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:27:08.861423 5378 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xjf5n\" (UniqueName: \"kubernetes.io/projected/6e78dacd-7440-4904-ae86-7083302ea675-kube-api-access-xjf5n\") pod \"maas-controller-6cb6cd7964-lgjgv\" (UID: \"6e78dacd-7440-4904-ae86-7083302ea675\") " pod="opendatahub/maas-controller-6cb6cd7964-lgjgv" Oct 30 21:27:08.925978 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:27:08.925945 5378 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-6cb6cd7964-lgjgv" Oct 30 21:27:09.302419 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:27:09.302388 5378 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-6cb6cd7964-lgjgv"] Oct 30 21:27:09.303644 ip-10-0-138-149 kubenswrapper[5378]: W1030 21:27:09.303610 5378 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6e78dacd_7440_4904_ae86_7083302ea675.slice/crio-33afc3897b51617060a5b1e0dd097a6dd2f09384f6359bd53cf262c772855d80 WatchSource:0}: Error finding container 33afc3897b51617060a5b1e0dd097a6dd2f09384f6359bd53cf262c772855d80: Status 404 returned error can't find the container with id 33afc3897b51617060a5b1e0dd097a6dd2f09384f6359bd53cf262c772855d80 Oct 30 21:27:09.305294 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:27:09.305271 5378 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 30 21:27:09.578021 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:27:09.577939 5378 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-6cb6cd7964-lgjgv" event={"ID":"6e78dacd-7440-4904-ae86-7083302ea675","Type":"ContainerStarted","Data":"33afc3897b51617060a5b1e0dd097a6dd2f09384f6359bd53cf262c772855d80"} Oct 30 21:27:09.738990 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:27:09.738956 5378 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="62138235-e57b-42e6-a892-67123e9b2478" path="/var/lib/kubelet/pods/62138235-e57b-42e6-a892-67123e9b2478/volumes" Oct 30 21:27:10.584369 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:27:10.584280 5378 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-6cb6cd7964-lgjgv" event={"ID":"6e78dacd-7440-4904-ae86-7083302ea675","Type":"ContainerStarted","Data":"d4d072d75d0a788277cb884e14d67810f2601891dff6e34390ef5118c86a34de"} Oct 30 21:27:10.584369 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:27:10.584354 5378 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-6cb6cd7964-lgjgv" Oct 30 21:27:10.701854 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:27:10.701804 5378 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/maas-controller-6cb6cd7964-lgjgv" podStartSLOduration=1.906722182 podStartE2EDuration="2.701791815s" totalImagesPullingTime="795.069633ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:27:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:27:10.700711666 +0000 UTC m=+2921.128825585" watchObservedRunningTime="2026-10-30 21:27:10.701791815 +0000 UTC m=+2921.129905734" Oct 30 21:27:18.836061 ip-10-0-138-149 kubenswrapper[5378]: E1030 21:27:18.836027 5378 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q6mx7" containerName="extractor" Oct 30 21:27:21.591811 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:27:21.591779 5378 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-6cb6cd7964-lgjgv" Oct 30 21:27:25.000245 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:27:25.000209 5378 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-769df7846c-99vbx"] Oct 30 21:27:25.000673 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:27:25.000587 5378 cpu_manager.go:414] "containerMap: removing container" podUID="6614eb26-439d-464e-a7c2-b95744976905" containerName="istio-proxy" Oct 30 21:27:25.000673 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:27:25.000598 5378 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6614eb26-439d-464e-a7c2-b95744976905" containerName="istio-proxy" Oct 30 21:27:25.000673 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:27:25.000615 5378 cpu_manager.go:414] "containerMap: removing container" podUID="2391a719-0c9b-4c56-b229-8f286a9aac96" containerName="istio-proxy" Oct 30 21:27:25.000673 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:27:25.000621 5378 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2391a719-0c9b-4c56-b229-8f286a9aac96" containerName="istio-proxy" Oct 30 21:27:25.000812 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:27:25.000678 5378 memory_manager.go:365] "RemoveStaleState removing state" podUID="2391a719-0c9b-4c56-b229-8f286a9aac96" containerName="istio-proxy" Oct 30 21:27:25.000812 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:27:25.000689 5378 memory_manager.go:365] "RemoveStaleState removing state" podUID="23f08320-e0a1-42bc-8fd1-de3492b59bca" containerName="istio-proxy" Oct 30 21:27:25.003817 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:27:25.003796 5378 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-99vbx" Oct 30 21:27:25.105774 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:27:25.105743 5378 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-769df7846c-99vbx"] Oct 30 21:27:25.143853 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:27:25.143820 5378 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ccc2fbc1-82a9-46e7-a6a6-102582f572d3-workload-socket\") pod \"maas-default-gateway-openshift-default-769df7846c-99vbx\" (UID: \"ccc2fbc1-82a9-46e7-a6a6-102582f572d3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-99vbx" Oct 30 21:27:25.144035 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:27:25.143857 5378 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ccc2fbc1-82a9-46e7-a6a6-102582f572d3-workload-certs\") pod \"maas-default-gateway-openshift-default-769df7846c-99vbx\" (UID: \"ccc2fbc1-82a9-46e7-a6a6-102582f572d3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-99vbx" Oct 30 21:27:25.144035 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:27:25.143879 5378 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ccc2fbc1-82a9-46e7-a6a6-102582f572d3-credential-socket\") pod \"maas-default-gateway-openshift-default-769df7846c-99vbx\" (UID: \"ccc2fbc1-82a9-46e7-a6a6-102582f572d3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-99vbx" Oct 30 21:27:25.144035 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:27:25.144008 5378 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ccc2fbc1-82a9-46e7-a6a6-102582f572d3-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-769df7846c-99vbx\" (UID: \"ccc2fbc1-82a9-46e7-a6a6-102582f572d3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-99vbx" Oct 30 21:27:25.144219 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:27:25.144119 5378 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ccc2fbc1-82a9-46e7-a6a6-102582f572d3-istio-envoy\") pod \"maas-default-gateway-openshift-default-769df7846c-99vbx\" (UID: \"ccc2fbc1-82a9-46e7-a6a6-102582f572d3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-99vbx" Oct 30 21:27:25.144219 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:27:25.144154 5378 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ccc2fbc1-82a9-46e7-a6a6-102582f572d3-istio-data\") pod \"maas-default-gateway-openshift-default-769df7846c-99vbx\" (UID: \"ccc2fbc1-82a9-46e7-a6a6-102582f572d3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-99vbx" Oct 30 21:27:25.144318 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:27:25.144293 5378 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ccc2fbc1-82a9-46e7-a6a6-102582f572d3-istio-token\") pod \"maas-default-gateway-openshift-default-769df7846c-99vbx\" (UID: \"ccc2fbc1-82a9-46e7-a6a6-102582f572d3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-99vbx" Oct 30 21:27:25.144364 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:27:25.144322 5378 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8m6jr\" (UniqueName: \"kubernetes.io/projected/ccc2fbc1-82a9-46e7-a6a6-102582f572d3-kube-api-access-8m6jr\") pod \"maas-default-gateway-openshift-default-769df7846c-99vbx\" (UID: \"ccc2fbc1-82a9-46e7-a6a6-102582f572d3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-99vbx" Oct 30 21:27:25.144364 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:27:25.144350 5378 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ccc2fbc1-82a9-46e7-a6a6-102582f572d3-istio-podinfo\") pod \"maas-default-gateway-openshift-default-769df7846c-99vbx\" (UID: \"ccc2fbc1-82a9-46e7-a6a6-102582f572d3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-99vbx" Oct 30 21:27:25.244925 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:27:25.244881 5378 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ccc2fbc1-82a9-46e7-a6a6-102582f572d3-workload-certs\") pod \"maas-default-gateway-openshift-default-769df7846c-99vbx\" (UID: \"ccc2fbc1-82a9-46e7-a6a6-102582f572d3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-99vbx" Oct 30 21:27:25.245121 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:27:25.244939 5378 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ccc2fbc1-82a9-46e7-a6a6-102582f572d3-credential-socket\") pod \"maas-default-gateway-openshift-default-769df7846c-99vbx\" (UID: \"ccc2fbc1-82a9-46e7-a6a6-102582f572d3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-99vbx" Oct 30 21:27:25.245121 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:27:25.244967 5378 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ccc2fbc1-82a9-46e7-a6a6-102582f572d3-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-769df7846c-99vbx\" (UID: \"ccc2fbc1-82a9-46e7-a6a6-102582f572d3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-99vbx" Oct 30 21:27:25.245121 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:27:25.245052 5378 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ccc2fbc1-82a9-46e7-a6a6-102582f572d3-istio-envoy\") pod \"maas-default-gateway-openshift-default-769df7846c-99vbx\" (UID: \"ccc2fbc1-82a9-46e7-a6a6-102582f572d3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-99vbx" Oct 30 21:27:25.245121 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:27:25.245081 5378 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ccc2fbc1-82a9-46e7-a6a6-102582f572d3-istio-data\") pod \"maas-default-gateway-openshift-default-769df7846c-99vbx\" (UID: \"ccc2fbc1-82a9-46e7-a6a6-102582f572d3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-99vbx" Oct 30 21:27:25.245369 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:27:25.245168 5378 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ccc2fbc1-82a9-46e7-a6a6-102582f572d3-istio-token\") pod \"maas-default-gateway-openshift-default-769df7846c-99vbx\" (UID: \"ccc2fbc1-82a9-46e7-a6a6-102582f572d3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-99vbx" Oct 30 21:27:25.245369 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:27:25.245195 5378 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8m6jr\" (UniqueName: \"kubernetes.io/projected/ccc2fbc1-82a9-46e7-a6a6-102582f572d3-kube-api-access-8m6jr\") pod \"maas-default-gateway-openshift-default-769df7846c-99vbx\" (UID: \"ccc2fbc1-82a9-46e7-a6a6-102582f572d3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-99vbx" Oct 30 21:27:25.245369 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:27:25.245240 5378 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ccc2fbc1-82a9-46e7-a6a6-102582f572d3-istio-podinfo\") pod \"maas-default-gateway-openshift-default-769df7846c-99vbx\" (UID: \"ccc2fbc1-82a9-46e7-a6a6-102582f572d3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-99vbx" Oct 30 21:27:25.245369 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:27:25.245277 5378 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ccc2fbc1-82a9-46e7-a6a6-102582f572d3-workload-socket\") pod \"maas-default-gateway-openshift-default-769df7846c-99vbx\" (UID: \"ccc2fbc1-82a9-46e7-a6a6-102582f572d3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-99vbx" Oct 30 21:27:25.245667 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:27:25.245368 5378 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ccc2fbc1-82a9-46e7-a6a6-102582f572d3-workload-certs\") pod \"maas-default-gateway-openshift-default-769df7846c-99vbx\" (UID: \"ccc2fbc1-82a9-46e7-a6a6-102582f572d3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-99vbx" Oct 30 21:27:25.245667 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:27:25.245395 5378 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ccc2fbc1-82a9-46e7-a6a6-102582f572d3-credential-socket\") pod \"maas-default-gateway-openshift-default-769df7846c-99vbx\" (UID: \"ccc2fbc1-82a9-46e7-a6a6-102582f572d3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-99vbx" Oct 30 21:27:25.245667 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:27:25.245468 5378 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ccc2fbc1-82a9-46e7-a6a6-102582f572d3-istio-data\") pod \"maas-default-gateway-openshift-default-769df7846c-99vbx\" (UID: \"ccc2fbc1-82a9-46e7-a6a6-102582f572d3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-99vbx" Oct 30 21:27:25.245838 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:27:25.245686 5378 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ccc2fbc1-82a9-46e7-a6a6-102582f572d3-workload-socket\") pod \"maas-default-gateway-openshift-default-769df7846c-99vbx\" (UID: \"ccc2fbc1-82a9-46e7-a6a6-102582f572d3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-99vbx" Oct 30 21:27:25.246041 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:27:25.246006 5378 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ccc2fbc1-82a9-46e7-a6a6-102582f572d3-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-769df7846c-99vbx\" (UID: \"ccc2fbc1-82a9-46e7-a6a6-102582f572d3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-99vbx" Oct 30 21:27:25.247469 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:27:25.247444 5378 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ccc2fbc1-82a9-46e7-a6a6-102582f572d3-istio-envoy\") pod \"maas-default-gateway-openshift-default-769df7846c-99vbx\" (UID: \"ccc2fbc1-82a9-46e7-a6a6-102582f572d3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-99vbx" Oct 30 21:27:25.247771 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:27:25.247752 5378 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ccc2fbc1-82a9-46e7-a6a6-102582f572d3-istio-podinfo\") pod \"maas-default-gateway-openshift-default-769df7846c-99vbx\" (UID: \"ccc2fbc1-82a9-46e7-a6a6-102582f572d3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-99vbx" Oct 30 21:27:25.303029 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:27:25.302960 5378 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8m6jr\" (UniqueName: \"kubernetes.io/projected/ccc2fbc1-82a9-46e7-a6a6-102582f572d3-kube-api-access-8m6jr\") pod \"maas-default-gateway-openshift-default-769df7846c-99vbx\" (UID: \"ccc2fbc1-82a9-46e7-a6a6-102582f572d3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-99vbx" Oct 30 21:27:25.303183 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:27:25.303163 5378 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ccc2fbc1-82a9-46e7-a6a6-102582f572d3-istio-token\") pod \"maas-default-gateway-openshift-default-769df7846c-99vbx\" (UID: \"ccc2fbc1-82a9-46e7-a6a6-102582f572d3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-99vbx" Oct 30 21:27:25.316063 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:27:25.316042 5378 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-99vbx" Oct 30 21:27:25.738506 ip-10-0-138-149 kubenswrapper[5378]: W1030 21:27:25.738480 5378 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podccc2fbc1_82a9_46e7_a6a6_102582f572d3.slice/crio-cdde0fa416f6c8419ae6eb011bc2afe3f31e8d6b1f2f13ea1a1e71ec1b3da71a WatchSource:0}: Error finding container cdde0fa416f6c8419ae6eb011bc2afe3f31e8d6b1f2f13ea1a1e71ec1b3da71a: Status 404 returned error can't find the container with id cdde0fa416f6c8419ae6eb011bc2afe3f31e8d6b1f2f13ea1a1e71ec1b3da71a Oct 30 21:27:25.740627 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:27:25.740596 5378 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 21:27:25.740731 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:27:25.740659 5378 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 21:27:25.740731 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:27:25.740689 5378 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 21:27:25.740943 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:27:25.740922 5378 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-769df7846c-99vbx"] Oct 30 21:27:26.683430 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:27:26.683384 5378 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-99vbx" event={"ID":"ccc2fbc1-82a9-46e7-a6a6-102582f572d3","Type":"ContainerStarted","Data":"ee408ac66234e3fbc77d567aa59ebb7627053aea5564c6f9ae7c51ec396598cc"} Oct 30 21:27:26.683943 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:27:26.683465 5378 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-99vbx" event={"ID":"ccc2fbc1-82a9-46e7-a6a6-102582f572d3","Type":"ContainerStarted","Data":"cdde0fa416f6c8419ae6eb011bc2afe3f31e8d6b1f2f13ea1a1e71ec1b3da71a"} Oct 30 21:27:26.852360 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:27:26.852302 5378 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-99vbx" podStartSLOduration=2.852289365 podStartE2EDuration="2.852289365s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:27:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:27:26.851210504 +0000 UTC m=+2937.279324424" watchObservedRunningTime="2026-10-30 21:27:26.852289365 +0000 UTC m=+2937.280403345" Oct 30 21:27:27.316668 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:27:27.316630 5378 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-99vbx" Oct 30 21:27:27.318307 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:27:27.318280 5378 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-769df7846c-99vbx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.167:15021/healthz/ready\": dial tcp 10.133.0.167:15021: connect: connection refused" start-of-body= Oct 30 21:27:27.318442 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:27:27.318338 5378 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-99vbx" podUID="ccc2fbc1-82a9-46e7-a6a6-102582f572d3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.167:15021/healthz/ready\": dial tcp 10.133.0.167:15021: connect: connection refused" Oct 30 21:27:28.317128 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:27:28.317094 5378 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-769df7846c-99vbx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.167:15021/healthz/ready\": dial tcp 10.133.0.167:15021: connect: connection refused" start-of-body= Oct 30 21:27:28.317524 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:27:28.317148 5378 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-99vbx" podUID="ccc2fbc1-82a9-46e7-a6a6-102582f572d3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.167:15021/healthz/ready\": dial tcp 10.133.0.167:15021: connect: connection refused" Oct 30 21:27:29.316478 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:27:29.316445 5378 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-769df7846c-99vbx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.167:15021/healthz/ready\": dial tcp 10.133.0.167:15021: connect: connection refused" start-of-body= Oct 30 21:27:29.316683 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:27:29.316505 5378 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-99vbx" podUID="ccc2fbc1-82a9-46e7-a6a6-102582f572d3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.167:15021/healthz/ready\": dial tcp 10.133.0.167:15021: connect: connection refused" Oct 30 21:27:29.526211 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:27:29.526168 5378 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-x75wp"] Oct 30 21:27:29.530430 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:27:29.530402 5378 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-x75wp" Oct 30 21:27:29.585406 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:27:29.585320 5378 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-x75wp"] Oct 30 21:27:29.598311 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:27:29.598275 5378 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ac2d133c-d6a1-4bba-b6f6-a201a06ac557-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-85237e45-x75wp\" (UID: \"ac2d133c-d6a1-4bba-b6f6-a201a06ac557\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-x75wp" Oct 30 21:27:29.598479 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:27:29.598325 5378 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ac2d133c-d6a1-4bba-b6f6-a201a06ac557-maas-api-xks-ca\") pod \"maas-api-revoke-sub-85237e45-x75wp\" (UID: \"ac2d133c-d6a1-4bba-b6f6-a201a06ac557\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-x75wp" Oct 30 21:27:29.598479 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:27:29.598435 5378 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p8nbg\" (UniqueName: \"kubernetes.io/projected/ac2d133c-d6a1-4bba-b6f6-a201a06ac557-kube-api-access-p8nbg\") pod \"maas-api-revoke-sub-85237e45-x75wp\" (UID: \"ac2d133c-d6a1-4bba-b6f6-a201a06ac557\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-x75wp" Oct 30 21:27:29.634008 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:27:29.633981 5378 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 21:27:29.634272 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:27:29.634256 5378 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 21:27:29.634322 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:27:29.634256 5378 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 21:27:29.634370 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:27:29.634331 5378 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-q2mgc\"" Oct 30 21:27:29.699117 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:27:29.699079 5378 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ac2d133c-d6a1-4bba-b6f6-a201a06ac557-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-85237e45-x75wp\" (UID: \"ac2d133c-d6a1-4bba-b6f6-a201a06ac557\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-x75wp" Oct 30 21:27:29.699311 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:27:29.699132 5378 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ac2d133c-d6a1-4bba-b6f6-a201a06ac557-maas-api-xks-ca\") pod \"maas-api-revoke-sub-85237e45-x75wp\" (UID: \"ac2d133c-d6a1-4bba-b6f6-a201a06ac557\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-x75wp" Oct 30 21:27:29.699311 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:27:29.699158 5378 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p8nbg\" (UniqueName: \"kubernetes.io/projected/ac2d133c-d6a1-4bba-b6f6-a201a06ac557-kube-api-access-p8nbg\") pod \"maas-api-revoke-sub-85237e45-x75wp\" (UID: \"ac2d133c-d6a1-4bba-b6f6-a201a06ac557\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-x75wp" Oct 30 21:27:29.699704 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:27:29.699681 5378 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ac2d133c-d6a1-4bba-b6f6-a201a06ac557-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-85237e45-x75wp\" (UID: \"ac2d133c-d6a1-4bba-b6f6-a201a06ac557\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-x75wp" Oct 30 21:27:29.701504 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:27:29.701484 5378 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ac2d133c-d6a1-4bba-b6f6-a201a06ac557-maas-api-xks-ca\") pod \"maas-api-revoke-sub-85237e45-x75wp\" (UID: \"ac2d133c-d6a1-4bba-b6f6-a201a06ac557\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-x75wp" Oct 30 21:27:29.824634 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:27:29.824597 5378 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 21:27:29.836044 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:27:29.835976 5378 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p8nbg\" (UniqueName: \"kubernetes.io/projected/ac2d133c-d6a1-4bba-b6f6-a201a06ac557-kube-api-access-p8nbg\") pod \"maas-api-revoke-sub-85237e45-x75wp\" (UID: \"ac2d133c-d6a1-4bba-b6f6-a201a06ac557\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-x75wp" Oct 30 21:27:29.894758 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:27:29.894721 5378 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-q2mgc\"" Oct 30 21:27:29.903933 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:27:29.903906 5378 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-x75wp" Oct 30 21:27:30.294379 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:27:30.294343 5378 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-x75wp"] Oct 30 21:27:30.296619 ip-10-0-138-149 kubenswrapper[5378]: W1030 21:27:30.296595 5378 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podac2d133c_d6a1_4bba_b6f6_a201a06ac557.slice/crio-f0ef2f7e3ca0585c1ac58ba4154133668bfeaa49fdd904caa86a77fa98db3a13 WatchSource:0}: Error finding container f0ef2f7e3ca0585c1ac58ba4154133668bfeaa49fdd904caa86a77fa98db3a13: Status 404 returned error can't find the container with id f0ef2f7e3ca0585c1ac58ba4154133668bfeaa49fdd904caa86a77fa98db3a13 Oct 30 21:27:30.316465 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:27:30.316438 5378 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-769df7846c-99vbx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.167:15021/healthz/ready\": dial tcp 10.133.0.167:15021: connect: connection refused" start-of-body= Oct 30 21:27:30.316550 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:27:30.316498 5378 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-99vbx" podUID="ccc2fbc1-82a9-46e7-a6a6-102582f572d3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.167:15021/healthz/ready\": dial tcp 10.133.0.167:15021: connect: connection refused" Oct 30 21:27:30.701599 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:27:30.701553 5378 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-x75wp" event={"ID":"ac2d133c-d6a1-4bba-b6f6-a201a06ac557","Type":"ContainerStarted","Data":"7ac3c2dc859196e5a12ba602831109ee176c438631b53d1584e4faad487e2c2f"} Oct 30 21:27:30.701986 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:27:30.701622 5378 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-x75wp" event={"ID":"ac2d133c-d6a1-4bba-b6f6-a201a06ac557","Type":"ContainerStarted","Data":"f0ef2f7e3ca0585c1ac58ba4154133668bfeaa49fdd904caa86a77fa98db3a13"} Oct 30 21:27:30.822129 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:27:30.822078 5378 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-x75wp" podStartSLOduration=1.8220648659999998 podStartE2EDuration="1.822064866s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:27:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:27:30.821805802 +0000 UTC m=+2941.249919722" watchObservedRunningTime="2026-10-30 21:27:30.822064866 +0000 UTC m=+2941.250178790" Oct 30 21:27:31.316446 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:27:31.316414 5378 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-769df7846c-99vbx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.167:15021/healthz/ready\": dial tcp 10.133.0.167:15021: connect: connection refused" start-of-body= Oct 30 21:27:31.316616 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:27:31.316467 5378 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-99vbx" podUID="ccc2fbc1-82a9-46e7-a6a6-102582f572d3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.167:15021/healthz/ready\": dial tcp 10.133.0.167:15021: connect: connection refused" Oct 30 21:27:31.707798 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:27:31.707757 5378 generic.go:356] "Generic (PLEG): container finished" podID="ac2d133c-d6a1-4bba-b6f6-a201a06ac557" containerID="7ac3c2dc859196e5a12ba602831109ee176c438631b53d1584e4faad487e2c2f" exitCode=0 Oct 30 21:27:31.708213 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:27:31.707816 5378 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-x75wp" event={"ID":"ac2d133c-d6a1-4bba-b6f6-a201a06ac557","Type":"ContainerDied","Data":"7ac3c2dc859196e5a12ba602831109ee176c438631b53d1584e4faad487e2c2f"} Oct 30 21:27:32.316724 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:27:32.316690 5378 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-769df7846c-99vbx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.167:15021/healthz/ready\": dial tcp 10.133.0.167:15021: connect: connection refused" start-of-body= Oct 30 21:27:32.316907 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:27:32.316748 5378 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-99vbx" podUID="ccc2fbc1-82a9-46e7-a6a6-102582f572d3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.167:15021/healthz/ready\": dial tcp 10.133.0.167:15021: connect: connection refused" Oct 30 21:27:32.865586 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:27:32.865552 5378 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-x75wp" Oct 30 21:27:32.927659 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:27:32.927628 5378 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ac2d133c-d6a1-4bba-b6f6-a201a06ac557-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ac2d133c-d6a1-4bba-b6f6-a201a06ac557-maas-api-xks-ca\") pod \"ac2d133c-d6a1-4bba-b6f6-a201a06ac557\" (UID: \"ac2d133c-d6a1-4bba-b6f6-a201a06ac557\") " Oct 30 21:27:32.927804 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:27:32.927697 5378 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ac2d133c-d6a1-4bba-b6f6-a201a06ac557-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ac2d133c-d6a1-4bba-b6f6-a201a06ac557-maas-api-openshift-ca\") pod \"ac2d133c-d6a1-4bba-b6f6-a201a06ac557\" (UID: \"ac2d133c-d6a1-4bba-b6f6-a201a06ac557\") " Oct 30 21:27:32.927871 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:27:32.927813 5378 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ac2d133c-d6a1-4bba-b6f6-a201a06ac557-kube-api-access-p8nbg\" (UniqueName: \"kubernetes.io/projected/ac2d133c-d6a1-4bba-b6f6-a201a06ac557-kube-api-access-p8nbg\") pod \"ac2d133c-d6a1-4bba-b6f6-a201a06ac557\" (UID: \"ac2d133c-d6a1-4bba-b6f6-a201a06ac557\") " Oct 30 21:27:32.928078 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:27:32.928048 5378 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ac2d133c-d6a1-4bba-b6f6-a201a06ac557-maas-api-openshift-ca" pod "ac2d133c-d6a1-4bba-b6f6-a201a06ac557" (UID: "ac2d133c-d6a1-4bba-b6f6-a201a06ac557"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:27:32.929791 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:27:32.929767 5378 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ac2d133c-d6a1-4bba-b6f6-a201a06ac557-maas-api-xks-ca" pod "ac2d133c-d6a1-4bba-b6f6-a201a06ac557" (UID: "ac2d133c-d6a1-4bba-b6f6-a201a06ac557"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:27:32.929920 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:27:32.929907 5378 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ac2d133c-d6a1-4bba-b6f6-a201a06ac557-kube-api-access-p8nbg" pod "ac2d133c-d6a1-4bba-b6f6-a201a06ac557" (UID: "ac2d133c-d6a1-4bba-b6f6-a201a06ac557"). InnerVolumeSpecName "kube-api-access-p8nbg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:27:33.029406 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:27:33.029326 5378 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p8nbg\" (UniqueName: \"kubernetes.io/projected/ac2d133c-d6a1-4bba-b6f6-a201a06ac557-kube-api-access-p8nbg\") on node \"ip-10-0-138-149.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:27:33.029406 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:27:33.029353 5378 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ac2d133c-d6a1-4bba-b6f6-a201a06ac557-maas-api-xks-ca\") on node \"ip-10-0-138-149.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:27:33.029406 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:27:33.029368 5378 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ac2d133c-d6a1-4bba-b6f6-a201a06ac557-maas-api-openshift-ca\") on node \"ip-10-0-138-149.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:27:33.718941 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:27:33.718903 5378 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-x75wp" event={"ID":"ac2d133c-d6a1-4bba-b6f6-a201a06ac557","Type":"ContainerDied","Data":"f0ef2f7e3ca0585c1ac58ba4154133668bfeaa49fdd904caa86a77fa98db3a13"} Oct 30 21:27:33.719152 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:27:33.718967 5378 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f0ef2f7e3ca0585c1ac58ba4154133668bfeaa49fdd904caa86a77fa98db3a13" Oct 30 21:27:33.719152 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:27:33.718914 5378 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-x75wp" Oct 30 21:27:34.052325 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:27:34.052237 5378 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-99vbx" Oct 30 21:27:34.053117 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:27:34.053095 5378 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-99vbx" Oct 30 21:27:34.054024 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:27:34.054002 5378 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-99vbx" Oct 30 21:27:40.120548 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:27:40.120514 5378 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-769df7846c-rszcl"] Oct 30 21:27:40.121051 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:27:40.121032 5378 cpu_manager.go:414] "containerMap: removing container" podUID="ac2d133c-d6a1-4bba-b6f6-a201a06ac557" containerName="revoke-keys" Oct 30 21:27:40.121108 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:27:40.121053 5378 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ac2d133c-d6a1-4bba-b6f6-a201a06ac557" containerName="revoke-keys" Oct 30 21:27:40.121149 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:27:40.121140 5378 memory_manager.go:365] "RemoveStaleState removing state" podUID="ac2d133c-d6a1-4bba-b6f6-a201a06ac557" containerName="revoke-keys" Oct 30 21:27:40.124327 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:27:40.124306 5378 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-rszcl" Oct 30 21:27:40.193495 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:27:40.193466 5378 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4tv6t\" (UniqueName: \"kubernetes.io/projected/946e32af-526e-4bae-855d-8062b58d497e-kube-api-access-4tv6t\") pod \"maas-default-gateway-openshift-default-769df7846c-rszcl\" (UID: \"946e32af-526e-4bae-855d-8062b58d497e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-rszcl" Oct 30 21:27:40.193673 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:27:40.193502 5378 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/946e32af-526e-4bae-855d-8062b58d497e-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-769df7846c-rszcl\" (UID: \"946e32af-526e-4bae-855d-8062b58d497e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-rszcl" Oct 30 21:27:40.193673 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:27:40.193524 5378 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/946e32af-526e-4bae-855d-8062b58d497e-workload-certs\") pod \"maas-default-gateway-openshift-default-769df7846c-rszcl\" (UID: \"946e32af-526e-4bae-855d-8062b58d497e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-rszcl" Oct 30 21:27:40.193673 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:27:40.193651 5378 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/946e32af-526e-4bae-855d-8062b58d497e-workload-socket\") pod \"maas-default-gateway-openshift-default-769df7846c-rszcl\" (UID: \"946e32af-526e-4bae-855d-8062b58d497e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-rszcl" Oct 30 21:27:40.193787 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:27:40.193686 5378 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/946e32af-526e-4bae-855d-8062b58d497e-istio-data\") pod \"maas-default-gateway-openshift-default-769df7846c-rszcl\" (UID: \"946e32af-526e-4bae-855d-8062b58d497e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-rszcl" Oct 30 21:27:40.193787 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:27:40.193712 5378 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/946e32af-526e-4bae-855d-8062b58d497e-istio-envoy\") pod \"maas-default-gateway-openshift-default-769df7846c-rszcl\" (UID: \"946e32af-526e-4bae-855d-8062b58d497e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-rszcl" Oct 30 21:27:40.193854 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:27:40.193793 5378 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/946e32af-526e-4bae-855d-8062b58d497e-istio-token\") pod \"maas-default-gateway-openshift-default-769df7846c-rszcl\" (UID: \"946e32af-526e-4bae-855d-8062b58d497e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-rszcl" Oct 30 21:27:40.193890 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:27:40.193865 5378 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/946e32af-526e-4bae-855d-8062b58d497e-istio-podinfo\") pod \"maas-default-gateway-openshift-default-769df7846c-rszcl\" (UID: \"946e32af-526e-4bae-855d-8062b58d497e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-rszcl" Oct 30 21:27:40.193890 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:27:40.193882 5378 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/946e32af-526e-4bae-855d-8062b58d497e-credential-socket\") pod \"maas-default-gateway-openshift-default-769df7846c-rszcl\" (UID: \"946e32af-526e-4bae-855d-8062b58d497e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-rszcl" Oct 30 21:27:40.229637 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:27:40.229609 5378 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-769df7846c-rszcl"] Oct 30 21:27:40.294628 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:27:40.294595 5378 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/946e32af-526e-4bae-855d-8062b58d497e-istio-data\") pod \"maas-default-gateway-openshift-default-769df7846c-rszcl\" (UID: \"946e32af-526e-4bae-855d-8062b58d497e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-rszcl" Oct 30 21:27:40.294628 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:27:40.294630 5378 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/946e32af-526e-4bae-855d-8062b58d497e-istio-envoy\") pod \"maas-default-gateway-openshift-default-769df7846c-rszcl\" (UID: \"946e32af-526e-4bae-855d-8062b58d497e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-rszcl" Oct 30 21:27:40.294819 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:27:40.294657 5378 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/946e32af-526e-4bae-855d-8062b58d497e-istio-token\") pod \"maas-default-gateway-openshift-default-769df7846c-rszcl\" (UID: \"946e32af-526e-4bae-855d-8062b58d497e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-rszcl" Oct 30 21:27:40.294819 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:27:40.294690 5378 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/946e32af-526e-4bae-855d-8062b58d497e-istio-podinfo\") pod \"maas-default-gateway-openshift-default-769df7846c-rszcl\" (UID: \"946e32af-526e-4bae-855d-8062b58d497e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-rszcl" Oct 30 21:27:40.294819 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:27:40.294715 5378 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/946e32af-526e-4bae-855d-8062b58d497e-credential-socket\") pod \"maas-default-gateway-openshift-default-769df7846c-rszcl\" (UID: \"946e32af-526e-4bae-855d-8062b58d497e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-rszcl" Oct 30 21:27:40.294984 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:27:40.294955 5378 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4tv6t\" (UniqueName: \"kubernetes.io/projected/946e32af-526e-4bae-855d-8062b58d497e-kube-api-access-4tv6t\") pod \"maas-default-gateway-openshift-default-769df7846c-rszcl\" (UID: \"946e32af-526e-4bae-855d-8062b58d497e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-rszcl" Oct 30 21:27:40.295051 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:27:40.295022 5378 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/946e32af-526e-4bae-855d-8062b58d497e-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-769df7846c-rszcl\" (UID: \"946e32af-526e-4bae-855d-8062b58d497e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-rszcl" Oct 30 21:27:40.295110 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:27:40.295058 5378 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/946e32af-526e-4bae-855d-8062b58d497e-workload-certs\") pod \"maas-default-gateway-openshift-default-769df7846c-rszcl\" (UID: \"946e32af-526e-4bae-855d-8062b58d497e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-rszcl" Oct 30 21:27:40.295110 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:27:40.295025 5378 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/946e32af-526e-4bae-855d-8062b58d497e-istio-data\") pod \"maas-default-gateway-openshift-default-769df7846c-rszcl\" (UID: \"946e32af-526e-4bae-855d-8062b58d497e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-rszcl" Oct 30 21:27:40.295296 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:27:40.295102 5378 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/946e32af-526e-4bae-855d-8062b58d497e-credential-socket\") pod \"maas-default-gateway-openshift-default-769df7846c-rszcl\" (UID: \"946e32af-526e-4bae-855d-8062b58d497e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-rszcl" Oct 30 21:27:40.295296 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:27:40.295168 5378 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/946e32af-526e-4bae-855d-8062b58d497e-workload-socket\") pod \"maas-default-gateway-openshift-default-769df7846c-rszcl\" (UID: \"946e32af-526e-4bae-855d-8062b58d497e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-rszcl" Oct 30 21:27:40.295425 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:27:40.295403 5378 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/946e32af-526e-4bae-855d-8062b58d497e-workload-certs\") pod \"maas-default-gateway-openshift-default-769df7846c-rszcl\" (UID: \"946e32af-526e-4bae-855d-8062b58d497e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-rszcl" Oct 30 21:27:40.295497 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:27:40.295468 5378 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/946e32af-526e-4bae-855d-8062b58d497e-workload-socket\") pod \"maas-default-gateway-openshift-default-769df7846c-rszcl\" (UID: \"946e32af-526e-4bae-855d-8062b58d497e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-rszcl" Oct 30 21:27:40.295705 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:27:40.295688 5378 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/946e32af-526e-4bae-855d-8062b58d497e-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-769df7846c-rszcl\" (UID: \"946e32af-526e-4bae-855d-8062b58d497e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-rszcl" Oct 30 21:27:40.296983 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:27:40.296964 5378 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/946e32af-526e-4bae-855d-8062b58d497e-istio-envoy\") pod \"maas-default-gateway-openshift-default-769df7846c-rszcl\" (UID: \"946e32af-526e-4bae-855d-8062b58d497e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-rszcl" Oct 30 21:27:40.297326 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:27:40.297304 5378 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/946e32af-526e-4bae-855d-8062b58d497e-istio-podinfo\") pod \"maas-default-gateway-openshift-default-769df7846c-rszcl\" (UID: \"946e32af-526e-4bae-855d-8062b58d497e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-rszcl" Oct 30 21:27:40.352863 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:27:40.352836 5378 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4tv6t\" (UniqueName: \"kubernetes.io/projected/946e32af-526e-4bae-855d-8062b58d497e-kube-api-access-4tv6t\") pod \"maas-default-gateway-openshift-default-769df7846c-rszcl\" (UID: \"946e32af-526e-4bae-855d-8062b58d497e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-rszcl" Oct 30 21:27:40.352863 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:27:40.352837 5378 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/946e32af-526e-4bae-855d-8062b58d497e-istio-token\") pod \"maas-default-gateway-openshift-default-769df7846c-rszcl\" (UID: \"946e32af-526e-4bae-855d-8062b58d497e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-rszcl" Oct 30 21:27:40.436756 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:27:40.436692 5378 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-rszcl" Oct 30 21:27:40.855244 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:27:40.855220 5378 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-769df7846c-rszcl"] Oct 30 21:27:40.857241 ip-10-0-138-149 kubenswrapper[5378]: W1030 21:27:40.857216 5378 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod946e32af_526e_4bae_855d_8062b58d497e.slice/crio-0e32c0c078b29a3f6c9ef3ba678b9793dd5acf7e1146b68c6cbcb6d2a07148ef WatchSource:0}: Error finding container 0e32c0c078b29a3f6c9ef3ba678b9793dd5acf7e1146b68c6cbcb6d2a07148ef: Status 404 returned error can't find the container with id 0e32c0c078b29a3f6c9ef3ba678b9793dd5acf7e1146b68c6cbcb6d2a07148ef Oct 30 21:27:40.859425 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:27:40.859394 5378 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 21:27:40.859529 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:27:40.859469 5378 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 21:27:40.859529 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:27:40.859518 5378 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 21:27:41.775587 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:27:41.774713 5378 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-rszcl" event={"ID":"946e32af-526e-4bae-855d-8062b58d497e","Type":"ContainerStarted","Data":"2dfb5663008ce686dc2f55b8c191de67bca56624c6283451d51f3f8bea4876aa"} Oct 30 21:27:41.775587 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:27:41.774788 5378 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-rszcl" event={"ID":"946e32af-526e-4bae-855d-8062b58d497e","Type":"ContainerStarted","Data":"0e32c0c078b29a3f6c9ef3ba678b9793dd5acf7e1146b68c6cbcb6d2a07148ef"} Oct 30 21:27:41.972587 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:27:41.971591 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:27:41.996428 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:27:41.996374 5378 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-rszcl" podStartSLOduration=1.996359922 podStartE2EDuration="1.996359922s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:27:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:27:41.944898894 +0000 UTC m=+2952.373012812" watchObservedRunningTime="2026-10-30 21:27:41.996359922 +0000 UTC m=+2952.424473835" Oct 30 21:27:42.086546 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:27:42.086460 5378 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-6cb6cd7964-lgjgv"] Oct 30 21:27:42.086731 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:27:42.086692 5378 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/maas-controller-6cb6cd7964-lgjgv" podUID="6e78dacd-7440-4904-ae86-7083302ea675" containerName="manager" containerID="cri-o://d4d072d75d0a788277cb884e14d67810f2601891dff6e34390ef5118c86a34de" gracePeriod=30 Oct 30 21:27:42.437418 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:27:42.437391 5378 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-rszcl" Oct 30 21:27:42.438635 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:27:42.438608 5378 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-769df7846c-rszcl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.169:15021/healthz/ready\": dial tcp 10.133.0.169:15021: connect: connection refused" start-of-body= Oct 30 21:27:42.438768 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:27:42.438655 5378 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-rszcl" podUID="946e32af-526e-4bae-855d-8062b58d497e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.169:15021/healthz/ready\": dial tcp 10.133.0.169:15021: connect: connection refused" Oct 30 21:27:42.485638 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:27:42.485612 5378 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-6cb6cd7964-lgjgv" Oct 30 21:27:42.520537 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:27:42.520506 5378 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6e78dacd-7440-4904-ae86-7083302ea675-metrics-certs\" (UniqueName: \"kubernetes.io/secret/6e78dacd-7440-4904-ae86-7083302ea675-metrics-certs\") pod \"6e78dacd-7440-4904-ae86-7083302ea675\" (UID: \"6e78dacd-7440-4904-ae86-7083302ea675\") " Oct 30 21:27:42.520694 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:27:42.520541 5378 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6e78dacd-7440-4904-ae86-7083302ea675-webhook-cert\" (UniqueName: \"kubernetes.io/secret/6e78dacd-7440-4904-ae86-7083302ea675-webhook-cert\") pod \"6e78dacd-7440-4904-ae86-7083302ea675\" (UID: \"6e78dacd-7440-4904-ae86-7083302ea675\") " Oct 30 21:27:42.520694 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:27:42.520676 5378 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6e78dacd-7440-4904-ae86-7083302ea675-kube-api-access-xjf5n\" (UniqueName: \"kubernetes.io/projected/6e78dacd-7440-4904-ae86-7083302ea675-kube-api-access-xjf5n\") pod \"6e78dacd-7440-4904-ae86-7083302ea675\" (UID: \"6e78dacd-7440-4904-ae86-7083302ea675\") " Oct 30 21:27:42.522539 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:27:42.522513 5378 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6e78dacd-7440-4904-ae86-7083302ea675-metrics-certs" pod "6e78dacd-7440-4904-ae86-7083302ea675" (UID: "6e78dacd-7440-4904-ae86-7083302ea675"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:27:42.522663 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:27:42.522550 5378 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6e78dacd-7440-4904-ae86-7083302ea675-webhook-cert" pod "6e78dacd-7440-4904-ae86-7083302ea675" (UID: "6e78dacd-7440-4904-ae86-7083302ea675"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:27:42.522712 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:27:42.522660 5378 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6e78dacd-7440-4904-ae86-7083302ea675-kube-api-access-xjf5n" pod "6e78dacd-7440-4904-ae86-7083302ea675" (UID: "6e78dacd-7440-4904-ae86-7083302ea675"). InnerVolumeSpecName "kube-api-access-xjf5n". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:27:42.621956 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:27:42.621918 5378 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xjf5n\" (UniqueName: \"kubernetes.io/projected/6e78dacd-7440-4904-ae86-7083302ea675-kube-api-access-xjf5n\") on node \"ip-10-0-138-149.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:27:42.621956 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:27:42.621959 5378 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6e78dacd-7440-4904-ae86-7083302ea675-metrics-certs\") on node \"ip-10-0-138-149.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:27:42.622252 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:27:42.621975 5378 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/6e78dacd-7440-4904-ae86-7083302ea675-webhook-cert\") on node \"ip-10-0-138-149.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:27:42.783550 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:27:42.783466 5378 generic.go:356] "Generic (PLEG): container finished" podID="6e78dacd-7440-4904-ae86-7083302ea675" containerID="d4d072d75d0a788277cb884e14d67810f2601891dff6e34390ef5118c86a34de" exitCode=0 Oct 30 21:27:42.783550 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:27:42.783512 5378 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-6cb6cd7964-lgjgv" Oct 30 21:27:42.784016 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:27:42.783551 5378 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-6cb6cd7964-lgjgv" event={"ID":"6e78dacd-7440-4904-ae86-7083302ea675","Type":"ContainerDied","Data":"d4d072d75d0a788277cb884e14d67810f2601891dff6e34390ef5118c86a34de"} Oct 30 21:27:42.784016 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:27:42.783622 5378 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-6cb6cd7964-lgjgv" event={"ID":"6e78dacd-7440-4904-ae86-7083302ea675","Type":"ContainerDied","Data":"33afc3897b51617060a5b1e0dd097a6dd2f09384f6359bd53cf262c772855d80"} Oct 30 21:27:42.784016 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:27:42.783644 5378 scope.go:122] "RemoveContainer" containerID="d4d072d75d0a788277cb884e14d67810f2601891dff6e34390ef5118c86a34de" Oct 30 21:27:42.792754 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:27:42.792739 5378 scope.go:122] "RemoveContainer" containerID="d4d072d75d0a788277cb884e14d67810f2601891dff6e34390ef5118c86a34de" Oct 30 21:27:42.793017 ip-10-0-138-149 kubenswrapper[5378]: E1030 21:27:42.792998 5378 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d4d072d75d0a788277cb884e14d67810f2601891dff6e34390ef5118c86a34de\": container with ID starting with d4d072d75d0a788277cb884e14d67810f2601891dff6e34390ef5118c86a34de not found: ID does not exist" containerID="d4d072d75d0a788277cb884e14d67810f2601891dff6e34390ef5118c86a34de" Oct 30 21:27:42.793082 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:27:42.793028 5378 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d4d072d75d0a788277cb884e14d67810f2601891dff6e34390ef5118c86a34de"} err="failed to get container status \"d4d072d75d0a788277cb884e14d67810f2601891dff6e34390ef5118c86a34de\": rpc error: code = NotFound desc = could not find container \"d4d072d75d0a788277cb884e14d67810f2601891dff6e34390ef5118c86a34de\": container with ID starting with d4d072d75d0a788277cb884e14d67810f2601891dff6e34390ef5118c86a34de not found: ID does not exist" Oct 30 21:27:42.949473 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:27:42.949437 5378 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-6cb6cd7964-lgjgv"] Oct 30 21:27:42.951982 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:27:42.951958 5378 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-6cb6cd7964-lgjgv"] Oct 30 21:27:43.437930 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:27:43.437897 5378 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-769df7846c-rszcl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.169:15021/healthz/ready\": dial tcp 10.133.0.169:15021: connect: connection refused" start-of-body= Oct 30 21:27:43.438111 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:27:43.437948 5378 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-rszcl" podUID="946e32af-526e-4bae-855d-8062b58d497e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.169:15021/healthz/ready\": dial tcp 10.133.0.169:15021: connect: connection refused" Oct 30 21:27:43.740342 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:27:43.740267 5378 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6e78dacd-7440-4904-ae86-7083302ea675" path="/var/lib/kubelet/pods/6e78dacd-7440-4904-ae86-7083302ea675/volumes" Oct 30 21:27:44.185600 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:27:44.185551 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:27:44.335677 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:27:44.335641 5378 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-6cb6cd7964-tnzlz"] Oct 30 21:27:44.336100 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:27:44.336081 5378 cpu_manager.go:414] "containerMap: removing container" podUID="6e78dacd-7440-4904-ae86-7083302ea675" containerName="manager" Oct 30 21:27:44.336197 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:27:44.336104 5378 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6e78dacd-7440-4904-ae86-7083302ea675" containerName="manager" Oct 30 21:27:44.336254 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:27:44.336215 5378 memory_manager.go:365] "RemoveStaleState removing state" podUID="6e78dacd-7440-4904-ae86-7083302ea675" containerName="manager" Oct 30 21:27:44.340718 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:27:44.340696 5378 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-6cb6cd7964-tnzlz" Oct 30 21:27:44.393777 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:27:44.393748 5378 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-kpvgd\"" Oct 30 21:27:44.393953 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:27:44.393807 5378 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\"" Oct 30 21:27:44.437738 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:27:44.437661 5378 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-769df7846c-rszcl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.169:15021/healthz/ready\": dial tcp 10.133.0.169:15021: connect: connection refused" start-of-body= Oct 30 21:27:44.437875 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:27:44.437738 5378 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-rszcl" podUID="946e32af-526e-4bae-855d-8062b58d497e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.169:15021/healthz/ready\": dial tcp 10.133.0.169:15021: connect: connection refused" Oct 30 21:27:44.437875 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:27:44.437860 5378 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7fjrl\" (UniqueName: \"kubernetes.io/projected/a4403f57-3bb7-4040-80ef-e582130b97cc-kube-api-access-7fjrl\") pod \"maas-controller-6cb6cd7964-tnzlz\" (UID: \"a4403f57-3bb7-4040-80ef-e582130b97cc\") " pod="opendatahub/maas-controller-6cb6cd7964-tnzlz" Oct 30 21:27:44.438044 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:27:44.437985 5378 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a4403f57-3bb7-4040-80ef-e582130b97cc-metrics-certs\") pod \"maas-controller-6cb6cd7964-tnzlz\" (UID: \"a4403f57-3bb7-4040-80ef-e582130b97cc\") " pod="opendatahub/maas-controller-6cb6cd7964-tnzlz" Oct 30 21:27:44.438100 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:27:44.438045 5378 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/a4403f57-3bb7-4040-80ef-e582130b97cc-webhook-cert\") pod \"maas-controller-6cb6cd7964-tnzlz\" (UID: \"a4403f57-3bb7-4040-80ef-e582130b97cc\") " pod="opendatahub/maas-controller-6cb6cd7964-tnzlz" Oct 30 21:27:44.444231 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:27:44.444206 5378 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\"" Oct 30 21:27:44.444231 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:27:44.444208 5378 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-metrics-tls\"" Oct 30 21:27:44.446470 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:27:44.446447 5378 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-6cb6cd7964-tnzlz"] Oct 30 21:27:44.539396 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:27:44.539364 5378 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a4403f57-3bb7-4040-80ef-e582130b97cc-metrics-certs\") pod \"maas-controller-6cb6cd7964-tnzlz\" (UID: \"a4403f57-3bb7-4040-80ef-e582130b97cc\") " pod="opendatahub/maas-controller-6cb6cd7964-tnzlz" Oct 30 21:27:44.539595 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:27:44.539407 5378 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/a4403f57-3bb7-4040-80ef-e582130b97cc-webhook-cert\") pod \"maas-controller-6cb6cd7964-tnzlz\" (UID: \"a4403f57-3bb7-4040-80ef-e582130b97cc\") " pod="opendatahub/maas-controller-6cb6cd7964-tnzlz" Oct 30 21:27:44.539595 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:27:44.539452 5378 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7fjrl\" (UniqueName: \"kubernetes.io/projected/a4403f57-3bb7-4040-80ef-e582130b97cc-kube-api-access-7fjrl\") pod \"maas-controller-6cb6cd7964-tnzlz\" (UID: \"a4403f57-3bb7-4040-80ef-e582130b97cc\") " pod="opendatahub/maas-controller-6cb6cd7964-tnzlz" Oct 30 21:27:44.541891 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:27:44.541859 5378 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a4403f57-3bb7-4040-80ef-e582130b97cc-metrics-certs\") pod \"maas-controller-6cb6cd7964-tnzlz\" (UID: \"a4403f57-3bb7-4040-80ef-e582130b97cc\") " pod="opendatahub/maas-controller-6cb6cd7964-tnzlz" Oct 30 21:27:44.542015 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:27:44.541955 5378 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/a4403f57-3bb7-4040-80ef-e582130b97cc-webhook-cert\") pod \"maas-controller-6cb6cd7964-tnzlz\" (UID: \"a4403f57-3bb7-4040-80ef-e582130b97cc\") " pod="opendatahub/maas-controller-6cb6cd7964-tnzlz" Oct 30 21:27:44.598607 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:27:44.598557 5378 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7fjrl\" (UniqueName: \"kubernetes.io/projected/a4403f57-3bb7-4040-80ef-e582130b97cc-kube-api-access-7fjrl\") pod \"maas-controller-6cb6cd7964-tnzlz\" (UID: \"a4403f57-3bb7-4040-80ef-e582130b97cc\") " pod="opendatahub/maas-controller-6cb6cd7964-tnzlz" Oct 30 21:27:44.664591 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:27:44.652635 5378 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-6cb6cd7964-tnzlz" Oct 30 21:27:45.047702 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:27:45.047672 5378 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-6cb6cd7964-tnzlz"] Oct 30 21:27:45.049653 ip-10-0-138-149 kubenswrapper[5378]: W1030 21:27:45.049617 5378 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda4403f57_3bb7_4040_80ef_e582130b97cc.slice/crio-6ad127d97b106228d77b9082fe55898ca37dda43b50ae3bad2dd23336ddaa6ff WatchSource:0}: Error finding container 6ad127d97b106228d77b9082fe55898ca37dda43b50ae3bad2dd23336ddaa6ff: Status 404 returned error can't find the container with id 6ad127d97b106228d77b9082fe55898ca37dda43b50ae3bad2dd23336ddaa6ff Oct 30 21:27:45.438020 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:27:45.437982 5378 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-769df7846c-rszcl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.169:15021/healthz/ready\": dial tcp 10.133.0.169:15021: connect: connection refused" start-of-body= Oct 30 21:27:45.438441 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:27:45.438039 5378 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-rszcl" podUID="946e32af-526e-4bae-855d-8062b58d497e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.169:15021/healthz/ready\": dial tcp 10.133.0.169:15021: connect: connection refused" Oct 30 21:27:45.799001 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:27:45.798921 5378 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-6cb6cd7964-tnzlz" event={"ID":"a4403f57-3bb7-4040-80ef-e582130b97cc","Type":"ContainerStarted","Data":"6ad127d97b106228d77b9082fe55898ca37dda43b50ae3bad2dd23336ddaa6ff"} Oct 30 21:27:46.437696 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:27:46.437658 5378 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-769df7846c-rszcl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.169:15021/healthz/ready\": dial tcp 10.133.0.169:15021: connect: connection refused" start-of-body= Oct 30 21:27:46.437874 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:27:46.437715 5378 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-rszcl" podUID="946e32af-526e-4bae-855d-8062b58d497e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.169:15021/healthz/ready\": dial tcp 10.133.0.169:15021: connect: connection refused" Oct 30 21:27:46.815090 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:27:46.815000 5378 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-6cb6cd7964-tnzlz" event={"ID":"a4403f57-3bb7-4040-80ef-e582130b97cc","Type":"ContainerStarted","Data":"da2814248ba467c1c48dc8fb756a396c0f22050f7b9c5782dcab09a9c479c5ef"} Oct 30 21:27:46.815544 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:27:46.815184 5378 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-6cb6cd7964-tnzlz" Oct 30 21:27:46.932447 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:27:46.932378 5378 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/maas-controller-6cb6cd7964-tnzlz" podStartSLOduration=1.98093067 podStartE2EDuration="2.93236213s" totalImagesPullingTime="951.43146ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:27:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:27:46.931603078 +0000 UTC m=+2957.359716997" watchObservedRunningTime="2026-10-30 21:27:46.93236213 +0000 UTC m=+2957.360476075" Oct 30 21:27:47.437703 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:27:47.437668 5378 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-769df7846c-rszcl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.169:15021/healthz/ready\": dial tcp 10.133.0.169:15021: connect: connection refused" start-of-body= Oct 30 21:27:47.437886 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:27:47.437726 5378 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-rszcl" podUID="946e32af-526e-4bae-855d-8062b58d497e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.169:15021/healthz/ready\": dial tcp 10.133.0.169:15021: connect: connection refused" Oct 30 21:27:48.836778 ip-10-0-138-149 kubenswrapper[5378]: E1030 21:27:48.836736 5378 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q6mx7" containerName="extractor" Oct 30 21:27:49.046819 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:27:49.046780 5378 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-rszcl" Oct 30 21:27:49.047147 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:27:49.047115 5378 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-rszcl" Oct 30 21:27:49.047829 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:27:49.047796 5378 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-rszcl" Oct 30 21:27:52.852975 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:27:52.852939 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:27:57.821608 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:27:57.821579 5378 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-6cb6cd7964-tnzlz" Oct 30 21:28:18.836208 ip-10-0-138-149 kubenswrapper[5378]: E1030 21:28:18.836128 5378 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q6mx7" containerName="extractor" Oct 30 21:28:19.104302 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:28:19.104267 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:28:21.440378 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:28:21.440344 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:28:21.561217 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:28:21.561180 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:28:29.775194 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:28:29.775154 5378 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-r9g59"] Oct 30 21:28:30.165012 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:28:30.164982 5378 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-7x9jg_c2ff359c-cf92-4a5b-8610-f6eb41acbe73/ovn-acl-logging/0.log" Oct 30 21:28:30.165209 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:28:30.165066 5378 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-7x9jg_c2ff359c-cf92-4a5b-8610-f6eb41acbe73/ovn-acl-logging/0.log" Oct 30 21:28:32.330819 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:28:32.330698 5378 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-769df7846c-xghmf container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.40:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 21:28:32.331428 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:28:32.331375 5378 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-xghmf" podUID="f93de8dc-026f-4868-94fb-d1b2006dfb63" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.40:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 21:28:35.334263 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:28:35.334136 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:28:38.363575 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:28:38.363542 5378 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-8r6mj"] Oct 30 21:28:38.366970 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:28:38.366953 5378 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-8r6mj" Oct 30 21:28:38.421657 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:28:38.421626 5378 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 21:28:38.472042 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:28:38.472015 5378 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 21:28:38.472208 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:28:38.472061 5378 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 21:28:38.472273 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:28:38.472212 5378 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-q2mgc\"" Oct 30 21:28:38.473581 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:28:38.473549 5378 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-8r6mj"] Oct 30 21:28:38.482418 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:28:38.482386 5378 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9qkdh\" (UniqueName: \"kubernetes.io/projected/8252d1e2-e618-4753-a491-fcfe070b0c3c-kube-api-access-9qkdh\") pod \"maas-api-revoke-sub-c1e271a9-8r6mj\" (UID: \"8252d1e2-e618-4753-a491-fcfe070b0c3c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-8r6mj" Oct 30 21:28:38.482560 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:28:38.482464 5378 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8252d1e2-e618-4753-a491-fcfe070b0c3c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c1e271a9-8r6mj\" (UID: \"8252d1e2-e618-4753-a491-fcfe070b0c3c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-8r6mj" Oct 30 21:28:38.482560 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:28:38.482552 5378 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8252d1e2-e618-4753-a491-fcfe070b0c3c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c1e271a9-8r6mj\" (UID: \"8252d1e2-e618-4753-a491-fcfe070b0c3c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-8r6mj" Oct 30 21:28:38.583271 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:28:38.583244 5378 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8252d1e2-e618-4753-a491-fcfe070b0c3c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c1e271a9-8r6mj\" (UID: \"8252d1e2-e618-4753-a491-fcfe070b0c3c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-8r6mj" Oct 30 21:28:38.583404 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:28:38.583275 5378 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8252d1e2-e618-4753-a491-fcfe070b0c3c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c1e271a9-8r6mj\" (UID: \"8252d1e2-e618-4753-a491-fcfe070b0c3c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-8r6mj" Oct 30 21:28:38.583404 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:28:38.583345 5378 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9qkdh\" (UniqueName: \"kubernetes.io/projected/8252d1e2-e618-4753-a491-fcfe070b0c3c-kube-api-access-9qkdh\") pod \"maas-api-revoke-sub-c1e271a9-8r6mj\" (UID: \"8252d1e2-e618-4753-a491-fcfe070b0c3c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-8r6mj" Oct 30 21:28:38.583972 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:28:38.583949 5378 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8252d1e2-e618-4753-a491-fcfe070b0c3c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c1e271a9-8r6mj\" (UID: \"8252d1e2-e618-4753-a491-fcfe070b0c3c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-8r6mj" Oct 30 21:28:38.585740 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:28:38.585719 5378 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8252d1e2-e618-4753-a491-fcfe070b0c3c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c1e271a9-8r6mj\" (UID: \"8252d1e2-e618-4753-a491-fcfe070b0c3c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-8r6mj" Oct 30 21:28:38.641141 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:28:38.641081 5378 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9qkdh\" (UniqueName: \"kubernetes.io/projected/8252d1e2-e618-4753-a491-fcfe070b0c3c-kube-api-access-9qkdh\") pod \"maas-api-revoke-sub-c1e271a9-8r6mj\" (UID: \"8252d1e2-e618-4753-a491-fcfe070b0c3c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-8r6mj" Oct 30 21:28:38.678989 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:28:38.678965 5378 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-8r6mj" Oct 30 21:28:39.056044 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:28:39.049957 5378 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-8r6mj"] Oct 30 21:28:39.077050 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:28:39.077026 5378 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-8r6mj" event={"ID":"8252d1e2-e618-4753-a491-fcfe070b0c3c","Type":"ContainerStarted","Data":"6223d489ddfac4e2e2126c6c9b17f7acb8a27d1161d2f845d74f57afdd0df85f"} Oct 30 21:28:40.082591 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:28:40.082534 5378 generic.go:356] "Generic (PLEG): container finished" podID="8252d1e2-e618-4753-a491-fcfe070b0c3c" containerID="d444ca7e546754214817edb86b5868055cfc1a8cd9a9b2ced5cdd549bc87ecda" exitCode=0 Oct 30 21:28:40.083017 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:28:40.082603 5378 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-8r6mj" event={"ID":"8252d1e2-e618-4753-a491-fcfe070b0c3c","Type":"ContainerDied","Data":"d444ca7e546754214817edb86b5868055cfc1a8cd9a9b2ced5cdd549bc87ecda"} Oct 30 21:28:40.601846 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:28:40.601814 5378 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-769df7846c-hr6hd"] Oct 30 21:28:40.605245 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:28:40.605218 5378 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-hr6hd" Oct 30 21:28:40.702305 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:28:40.702272 5378 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dc49893f-0dba-4aef-bc77-d6e3c962dd69-credential-socket\") pod \"maas-default-gateway-openshift-default-769df7846c-hr6hd\" (UID: \"dc49893f-0dba-4aef-bc77-d6e3c962dd69\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-hr6hd" Oct 30 21:28:40.702606 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:28:40.702314 5378 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dc49893f-0dba-4aef-bc77-d6e3c962dd69-istio-podinfo\") pod \"maas-default-gateway-openshift-default-769df7846c-hr6hd\" (UID: \"dc49893f-0dba-4aef-bc77-d6e3c962dd69\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-hr6hd" Oct 30 21:28:40.702606 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:28:40.702366 5378 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dc49893f-0dba-4aef-bc77-d6e3c962dd69-istio-envoy\") pod \"maas-default-gateway-openshift-default-769df7846c-hr6hd\" (UID: \"dc49893f-0dba-4aef-bc77-d6e3c962dd69\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-hr6hd" Oct 30 21:28:40.702606 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:28:40.702394 5378 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dc49893f-0dba-4aef-bc77-d6e3c962dd69-workload-certs\") pod \"maas-default-gateway-openshift-default-769df7846c-hr6hd\" (UID: \"dc49893f-0dba-4aef-bc77-d6e3c962dd69\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-hr6hd" Oct 30 21:28:40.702606 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:28:40.702480 5378 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dc49893f-0dba-4aef-bc77-d6e3c962dd69-istio-data\") pod \"maas-default-gateway-openshift-default-769df7846c-hr6hd\" (UID: \"dc49893f-0dba-4aef-bc77-d6e3c962dd69\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-hr6hd" Oct 30 21:28:40.702945 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:28:40.702727 5378 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dc49893f-0dba-4aef-bc77-d6e3c962dd69-istio-token\") pod \"maas-default-gateway-openshift-default-769df7846c-hr6hd\" (UID: \"dc49893f-0dba-4aef-bc77-d6e3c962dd69\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-hr6hd" Oct 30 21:28:40.702945 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:28:40.702857 5378 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9nbhg\" (UniqueName: \"kubernetes.io/projected/dc49893f-0dba-4aef-bc77-d6e3c962dd69-kube-api-access-9nbhg\") pod \"maas-default-gateway-openshift-default-769df7846c-hr6hd\" (UID: \"dc49893f-0dba-4aef-bc77-d6e3c962dd69\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-hr6hd" Oct 30 21:28:40.702945 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:28:40.702891 5378 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dc49893f-0dba-4aef-bc77-d6e3c962dd69-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-769df7846c-hr6hd\" (UID: \"dc49893f-0dba-4aef-bc77-d6e3c962dd69\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-hr6hd" Oct 30 21:28:40.703120 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:28:40.702945 5378 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dc49893f-0dba-4aef-bc77-d6e3c962dd69-workload-socket\") pod \"maas-default-gateway-openshift-default-769df7846c-hr6hd\" (UID: \"dc49893f-0dba-4aef-bc77-d6e3c962dd69\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-hr6hd" Oct 30 21:28:40.704859 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:28:40.704834 5378 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-769df7846c-hr6hd"] Oct 30 21:28:40.803925 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:28:40.803884 5378 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9nbhg\" (UniqueName: \"kubernetes.io/projected/dc49893f-0dba-4aef-bc77-d6e3c962dd69-kube-api-access-9nbhg\") pod \"maas-default-gateway-openshift-default-769df7846c-hr6hd\" (UID: \"dc49893f-0dba-4aef-bc77-d6e3c962dd69\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-hr6hd" Oct 30 21:28:40.803925 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:28:40.803928 5378 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dc49893f-0dba-4aef-bc77-d6e3c962dd69-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-769df7846c-hr6hd\" (UID: \"dc49893f-0dba-4aef-bc77-d6e3c962dd69\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-hr6hd" Oct 30 21:28:40.804165 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:28:40.803954 5378 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dc49893f-0dba-4aef-bc77-d6e3c962dd69-workload-socket\") pod \"maas-default-gateway-openshift-default-769df7846c-hr6hd\" (UID: \"dc49893f-0dba-4aef-bc77-d6e3c962dd69\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-hr6hd" Oct 30 21:28:40.804165 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:28:40.804001 5378 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dc49893f-0dba-4aef-bc77-d6e3c962dd69-credential-socket\") pod \"maas-default-gateway-openshift-default-769df7846c-hr6hd\" (UID: \"dc49893f-0dba-4aef-bc77-d6e3c962dd69\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-hr6hd" Oct 30 21:28:40.804165 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:28:40.804029 5378 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dc49893f-0dba-4aef-bc77-d6e3c962dd69-istio-podinfo\") pod \"maas-default-gateway-openshift-default-769df7846c-hr6hd\" (UID: \"dc49893f-0dba-4aef-bc77-d6e3c962dd69\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-hr6hd" Oct 30 21:28:40.804165 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:28:40.804073 5378 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dc49893f-0dba-4aef-bc77-d6e3c962dd69-istio-envoy\") pod \"maas-default-gateway-openshift-default-769df7846c-hr6hd\" (UID: \"dc49893f-0dba-4aef-bc77-d6e3c962dd69\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-hr6hd" Oct 30 21:28:40.804165 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:28:40.804103 5378 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dc49893f-0dba-4aef-bc77-d6e3c962dd69-workload-certs\") pod \"maas-default-gateway-openshift-default-769df7846c-hr6hd\" (UID: \"dc49893f-0dba-4aef-bc77-d6e3c962dd69\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-hr6hd" Oct 30 21:28:40.804165 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:28:40.804126 5378 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dc49893f-0dba-4aef-bc77-d6e3c962dd69-istio-data\") pod \"maas-default-gateway-openshift-default-769df7846c-hr6hd\" (UID: \"dc49893f-0dba-4aef-bc77-d6e3c962dd69\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-hr6hd" Oct 30 21:28:40.804511 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:28:40.804196 5378 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dc49893f-0dba-4aef-bc77-d6e3c962dd69-istio-token\") pod \"maas-default-gateway-openshift-default-769df7846c-hr6hd\" (UID: \"dc49893f-0dba-4aef-bc77-d6e3c962dd69\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-hr6hd" Oct 30 21:28:40.804511 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:28:40.804455 5378 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dc49893f-0dba-4aef-bc77-d6e3c962dd69-workload-socket\") pod \"maas-default-gateway-openshift-default-769df7846c-hr6hd\" (UID: \"dc49893f-0dba-4aef-bc77-d6e3c962dd69\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-hr6hd" Oct 30 21:28:40.804668 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:28:40.804644 5378 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dc49893f-0dba-4aef-bc77-d6e3c962dd69-istio-data\") pod \"maas-default-gateway-openshift-default-769df7846c-hr6hd\" (UID: \"dc49893f-0dba-4aef-bc77-d6e3c962dd69\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-hr6hd" Oct 30 21:28:40.804718 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:28:40.804657 5378 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dc49893f-0dba-4aef-bc77-d6e3c962dd69-workload-certs\") pod \"maas-default-gateway-openshift-default-769df7846c-hr6hd\" (UID: \"dc49893f-0dba-4aef-bc77-d6e3c962dd69\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-hr6hd" Oct 30 21:28:40.804780 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:28:40.804764 5378 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dc49893f-0dba-4aef-bc77-d6e3c962dd69-credential-socket\") pod \"maas-default-gateway-openshift-default-769df7846c-hr6hd\" (UID: \"dc49893f-0dba-4aef-bc77-d6e3c962dd69\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-hr6hd" Oct 30 21:28:40.804832 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:28:40.804790 5378 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dc49893f-0dba-4aef-bc77-d6e3c962dd69-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-769df7846c-hr6hd\" (UID: \"dc49893f-0dba-4aef-bc77-d6e3c962dd69\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-hr6hd" Oct 30 21:28:40.806442 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:28:40.806419 5378 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dc49893f-0dba-4aef-bc77-d6e3c962dd69-istio-envoy\") pod \"maas-default-gateway-openshift-default-769df7846c-hr6hd\" (UID: \"dc49893f-0dba-4aef-bc77-d6e3c962dd69\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-hr6hd" Oct 30 21:28:40.806671 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:28:40.806651 5378 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dc49893f-0dba-4aef-bc77-d6e3c962dd69-istio-podinfo\") pod \"maas-default-gateway-openshift-default-769df7846c-hr6hd\" (UID: \"dc49893f-0dba-4aef-bc77-d6e3c962dd69\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-hr6hd" Oct 30 21:28:40.860840 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:28:40.860786 5378 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dc49893f-0dba-4aef-bc77-d6e3c962dd69-istio-token\") pod \"maas-default-gateway-openshift-default-769df7846c-hr6hd\" (UID: \"dc49893f-0dba-4aef-bc77-d6e3c962dd69\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-hr6hd" Oct 30 21:28:40.861098 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:28:40.861074 5378 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9nbhg\" (UniqueName: \"kubernetes.io/projected/dc49893f-0dba-4aef-bc77-d6e3c962dd69-kube-api-access-9nbhg\") pod \"maas-default-gateway-openshift-default-769df7846c-hr6hd\" (UID: \"dc49893f-0dba-4aef-bc77-d6e3c962dd69\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-hr6hd" Oct 30 21:28:40.918012 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:28:40.917991 5378 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-hr6hd" Oct 30 21:28:41.203770 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:28:41.203737 5378 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-8r6mj" Oct 30 21:28:41.208034 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:28:41.208011 5378 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8252d1e2-e618-4753-a491-fcfe070b0c3c-kube-api-access-9qkdh\" (UniqueName: \"kubernetes.io/projected/8252d1e2-e618-4753-a491-fcfe070b0c3c-kube-api-access-9qkdh\") pod \"8252d1e2-e618-4753-a491-fcfe070b0c3c\" (UID: \"8252d1e2-e618-4753-a491-fcfe070b0c3c\") " Oct 30 21:28:41.208123 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:28:41.208099 5378 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8252d1e2-e618-4753-a491-fcfe070b0c3c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8252d1e2-e618-4753-a491-fcfe070b0c3c-maas-api-openshift-ca\") pod \"8252d1e2-e618-4753-a491-fcfe070b0c3c\" (UID: \"8252d1e2-e618-4753-a491-fcfe070b0c3c\") " Oct 30 21:28:41.208163 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:28:41.208145 5378 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8252d1e2-e618-4753-a491-fcfe070b0c3c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8252d1e2-e618-4753-a491-fcfe070b0c3c-maas-api-xks-ca\") pod \"8252d1e2-e618-4753-a491-fcfe070b0c3c\" (UID: \"8252d1e2-e618-4753-a491-fcfe070b0c3c\") " Oct 30 21:28:41.208423 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:28:41.208401 5378 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8252d1e2-e618-4753-a491-fcfe070b0c3c-maas-api-openshift-ca" pod "8252d1e2-e618-4753-a491-fcfe070b0c3c" (UID: "8252d1e2-e618-4753-a491-fcfe070b0c3c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:28:41.210126 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:28:41.210105 5378 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8252d1e2-e618-4753-a491-fcfe070b0c3c-kube-api-access-9qkdh" pod "8252d1e2-e618-4753-a491-fcfe070b0c3c" (UID: "8252d1e2-e618-4753-a491-fcfe070b0c3c"). InnerVolumeSpecName "kube-api-access-9qkdh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:28:41.210277 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:28:41.210178 5378 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8252d1e2-e618-4753-a491-fcfe070b0c3c-maas-api-xks-ca" pod "8252d1e2-e618-4753-a491-fcfe070b0c3c" (UID: "8252d1e2-e618-4753-a491-fcfe070b0c3c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:28:41.309129 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:28:41.309097 5378 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9qkdh\" (UniqueName: \"kubernetes.io/projected/8252d1e2-e618-4753-a491-fcfe070b0c3c-kube-api-access-9qkdh\") on node \"ip-10-0-138-149.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:28:41.309129 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:28:41.309125 5378 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8252d1e2-e618-4753-a491-fcfe070b0c3c-maas-api-openshift-ca\") on node \"ip-10-0-138-149.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:28:41.309308 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:28:41.309138 5378 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8252d1e2-e618-4753-a491-fcfe070b0c3c-maas-api-xks-ca\") on node \"ip-10-0-138-149.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:28:41.336745 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:28:41.336713 5378 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-769df7846c-hr6hd"] Oct 30 21:28:41.371291 ip-10-0-138-149 kubenswrapper[5378]: W1030 21:28:41.371262 5378 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddc49893f_0dba_4aef_bc77_d6e3c962dd69.slice/crio-9c5ae594d150ecdc575242c941aef66234b83bdbeedf14632d2eef040337efa1 WatchSource:0}: Error finding container 9c5ae594d150ecdc575242c941aef66234b83bdbeedf14632d2eef040337efa1: Status 404 returned error can't find the container with id 9c5ae594d150ecdc575242c941aef66234b83bdbeedf14632d2eef040337efa1 Oct 30 21:28:41.373506 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:28:41.373475 5378 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 21:28:41.373596 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:28:41.373544 5378 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 21:28:41.373645 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:28:41.373592 5378 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 21:28:42.093556 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:28:42.093472 5378 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-8r6mj" Oct 30 21:28:42.093760 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:28:42.093470 5378 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-8r6mj" event={"ID":"8252d1e2-e618-4753-a491-fcfe070b0c3c","Type":"ContainerDied","Data":"6223d489ddfac4e2e2126c6c9b17f7acb8a27d1161d2f845d74f57afdd0df85f"} Oct 30 21:28:42.093760 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:28:42.093608 5378 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6223d489ddfac4e2e2126c6c9b17f7acb8a27d1161d2f845d74f57afdd0df85f" Oct 30 21:28:42.095298 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:28:42.095273 5378 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-hr6hd" event={"ID":"dc49893f-0dba-4aef-bc77-d6e3c962dd69","Type":"ContainerStarted","Data":"e80258b914c724e2ddf051665ac0c87a393140cd958dc7edbd768d57c3b01fc8"} Oct 30 21:28:42.095415 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:28:42.095332 5378 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-hr6hd" event={"ID":"dc49893f-0dba-4aef-bc77-d6e3c962dd69","Type":"ContainerStarted","Data":"9c5ae594d150ecdc575242c941aef66234b83bdbeedf14632d2eef040337efa1"} Oct 30 21:28:42.213763 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:28:42.213712 5378 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-hr6hd" podStartSLOduration=2.213697573 podStartE2EDuration="2.213697573s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:28:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:28:42.21318634 +0000 UTC m=+3012.641300260" watchObservedRunningTime="2026-10-30 21:28:42.213697573 +0000 UTC m=+3012.641811491" Oct 30 21:28:42.918513 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:28:42.918478 5378 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-hr6hd" Oct 30 21:28:42.919939 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:28:42.919915 5378 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-769df7846c-hr6hd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.172:15021/healthz/ready\": dial tcp 10.133.0.172:15021: connect: connection refused" start-of-body= Oct 30 21:28:42.920062 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:28:42.919964 5378 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-hr6hd" podUID="dc49893f-0dba-4aef-bc77-d6e3c962dd69" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.172:15021/healthz/ready\": dial tcp 10.133.0.172:15021: connect: connection refused" Oct 30 21:28:43.918692 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:28:43.918659 5378 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-769df7846c-hr6hd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.172:15021/healthz/ready\": dial tcp 10.133.0.172:15021: connect: connection refused" start-of-body= Oct 30 21:28:43.919089 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:28:43.918711 5378 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-hr6hd" podUID="dc49893f-0dba-4aef-bc77-d6e3c962dd69" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.172:15021/healthz/ready\": dial tcp 10.133.0.172:15021: connect: connection refused" Oct 30 21:28:44.919090 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:28:44.919049 5378 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-769df7846c-hr6hd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.172:15021/healthz/ready\": dial tcp 10.133.0.172:15021: connect: connection refused" start-of-body= Oct 30 21:28:44.919511 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:28:44.919107 5378 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-hr6hd" podUID="dc49893f-0dba-4aef-bc77-d6e3c962dd69" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.172:15021/healthz/ready\": dial tcp 10.133.0.172:15021: connect: connection refused" Oct 30 21:28:45.919008 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:28:45.918976 5378 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-769df7846c-hr6hd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.172:15021/healthz/ready\": dial tcp 10.133.0.172:15021: connect: connection refused" start-of-body= Oct 30 21:28:45.919193 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:28:45.919028 5378 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-hr6hd" podUID="dc49893f-0dba-4aef-bc77-d6e3c962dd69" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.172:15021/healthz/ready\": dial tcp 10.133.0.172:15021: connect: connection refused" Oct 30 21:28:46.918804 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:28:46.918762 5378 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-769df7846c-hr6hd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.172:15021/healthz/ready\": dial tcp 10.133.0.172:15021: connect: connection refused" start-of-body= Oct 30 21:28:46.918997 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:28:46.918828 5378 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-hr6hd" podUID="dc49893f-0dba-4aef-bc77-d6e3c962dd69" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.172:15021/healthz/ready\": dial tcp 10.133.0.172:15021: connect: connection refused" Oct 30 21:28:47.918304 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:28:47.918261 5378 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-769df7846c-hr6hd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.172:15021/healthz/ready\": dial tcp 10.133.0.172:15021: connect: connection refused" start-of-body= Oct 30 21:28:47.918879 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:28:47.918315 5378 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-hr6hd" podUID="dc49893f-0dba-4aef-bc77-d6e3c962dd69" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.172:15021/healthz/ready\": dial tcp 10.133.0.172:15021: connect: connection refused" Oct 30 21:28:48.839201 ip-10-0-138-149 kubenswrapper[5378]: E1030 21:28:48.839165 5378 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q6mx7" containerName="extractor" Oct 30 21:28:49.564912 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:28:49.564877 5378 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-hr6hd" Oct 30 21:28:49.565383 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:28:49.565334 5378 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-hr6hd" Oct 30 21:28:49.566038 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:28:49.566006 5378 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-hr6hd" Oct 30 21:28:51.184494 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:28:51.184462 5378 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-r9g59"] Oct 30 21:28:53.243314 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:28:53.243283 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:29:00.586189 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:29:00.586158 5378 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-r9g59"] Oct 30 21:29:18.837183 ip-10-0-138-149 kubenswrapper[5378]: E1030 21:29:18.837146 5378 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q6mx7" containerName="extractor" Oct 30 21:29:22.478835 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:29:22.478797 5378 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-r9g59"] Oct 30 21:29:39.348430 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:29:39.348349 5378 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-r9g59"] Oct 30 21:29:48.836178 ip-10-0-138-149 kubenswrapper[5378]: E1030 21:29:48.836134 5378 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q6mx7" containerName="extractor" Oct 30 21:29:53.565818 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:29:53.565786 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:30:00.158618 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:30:00.158579 5378 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889930-9jjd9"] Oct 30 21:30:00.159147 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:30:00.159125 5378 cpu_manager.go:414] "containerMap: removing container" podUID="8252d1e2-e618-4753-a491-fcfe070b0c3c" containerName="revoke-keys" Oct 30 21:30:00.159239 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:30:00.159151 5378 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8252d1e2-e618-4753-a491-fcfe070b0c3c" containerName="revoke-keys" Oct 30 21:30:00.159336 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:30:00.159322 5378 memory_manager.go:365] "RemoveStaleState removing state" podUID="8252d1e2-e618-4753-a491-fcfe070b0c3c" containerName="revoke-keys" Oct 30 21:30:00.162332 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:30:00.162312 5378 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889930-9jjd9" Oct 30 21:30:00.222317 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:30:00.222286 5378 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889930-9jjd9"] Oct 30 21:30:00.233405 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:30:00.233381 5378 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/37d16441-9ce7-45e8-868d-ab0994bca2ba-maas-api-service-ca\") pod \"maas-api-key-cleanup-29889930-9jjd9\" (UID: \"37d16441-9ce7-45e8-868d-ab0994bca2ba\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889930-9jjd9" Oct 30 21:30:00.233519 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:30:00.233437 5378 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kcpmf\" (UniqueName: \"kubernetes.io/projected/37d16441-9ce7-45e8-868d-ab0994bca2ba-kube-api-access-kcpmf\") pod \"maas-api-key-cleanup-29889930-9jjd9\" (UID: \"37d16441-9ce7-45e8-868d-ab0994bca2ba\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889930-9jjd9" Oct 30 21:30:00.270314 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:30:00.270287 5378 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 21:30:00.270474 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:30:00.270456 5378 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-q2mgc\"" Oct 30 21:30:00.270529 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:30:00.270456 5378 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 21:30:00.334938 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:30:00.334907 5378 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/37d16441-9ce7-45e8-868d-ab0994bca2ba-maas-api-service-ca\") pod \"maas-api-key-cleanup-29889930-9jjd9\" (UID: \"37d16441-9ce7-45e8-868d-ab0994bca2ba\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889930-9jjd9" Oct 30 21:30:00.335075 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:30:00.334963 5378 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kcpmf\" (UniqueName: \"kubernetes.io/projected/37d16441-9ce7-45e8-868d-ab0994bca2ba-kube-api-access-kcpmf\") pod \"maas-api-key-cleanup-29889930-9jjd9\" (UID: \"37d16441-9ce7-45e8-868d-ab0994bca2ba\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889930-9jjd9" Oct 30 21:30:00.335535 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:30:00.335515 5378 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/37d16441-9ce7-45e8-868d-ab0994bca2ba-maas-api-service-ca\") pod \"maas-api-key-cleanup-29889930-9jjd9\" (UID: \"37d16441-9ce7-45e8-868d-ab0994bca2ba\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889930-9jjd9" Oct 30 21:30:00.397946 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:30:00.397915 5378 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kcpmf\" (UniqueName: \"kubernetes.io/projected/37d16441-9ce7-45e8-868d-ab0994bca2ba-kube-api-access-kcpmf\") pod \"maas-api-key-cleanup-29889930-9jjd9\" (UID: \"37d16441-9ce7-45e8-868d-ab0994bca2ba\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889930-9jjd9" Oct 30 21:30:00.475501 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:30:00.475439 5378 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889930-9jjd9" Oct 30 21:30:00.844945 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:30:00.844921 5378 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889930-9jjd9"] Oct 30 21:30:00.847357 ip-10-0-138-149 kubenswrapper[5378]: W1030 21:30:00.847330 5378 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod37d16441_9ce7_45e8_868d_ab0994bca2ba.slice/crio-02f9339d5ea08bbfe0d19d8e68031d12e0e2bef30fb0c591d8f6ba18d5bbe565 WatchSource:0}: Error finding container 02f9339d5ea08bbfe0d19d8e68031d12e0e2bef30fb0c591d8f6ba18d5bbe565: Status 404 returned error can't find the container with id 02f9339d5ea08bbfe0d19d8e68031d12e0e2bef30fb0c591d8f6ba18d5bbe565 Oct 30 21:30:01.523557 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:30:01.523451 5378 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889930-9jjd9" event={"ID":"37d16441-9ce7-45e8-868d-ab0994bca2ba","Type":"ContainerStarted","Data":"70f607f8a237c456950f87a28694287b40d9c3b36cf75acaa7ecbfc8f1aa1d17"} Oct 30 21:30:01.523557 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:30:01.523551 5378 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889930-9jjd9" event={"ID":"37d16441-9ce7-45e8-868d-ab0994bca2ba","Type":"ContainerStarted","Data":"02f9339d5ea08bbfe0d19d8e68031d12e0e2bef30fb0c591d8f6ba18d5bbe565"} Oct 30 21:30:01.640250 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:30:01.640194 5378 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889930-9jjd9" podStartSLOduration=1.6401810289999998 podStartE2EDuration="1.640181029s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:30:01.639039018 +0000 UTC m=+3092.067152937" watchObservedRunningTime="2026-10-30 21:30:01.640181029 +0000 UTC m=+3092.068294948" Oct 30 21:30:02.529503 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:30:02.529467 5378 generic.go:356] "Generic (PLEG): container finished" podID="37d16441-9ce7-45e8-868d-ab0994bca2ba" containerID="70f607f8a237c456950f87a28694287b40d9c3b36cf75acaa7ecbfc8f1aa1d17" exitCode=60 Oct 30 21:30:02.530040 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:30:02.529555 5378 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889930-9jjd9" event={"ID":"37d16441-9ce7-45e8-868d-ab0994bca2ba","Type":"ContainerDied","Data":"70f607f8a237c456950f87a28694287b40d9c3b36cf75acaa7ecbfc8f1aa1d17"} Oct 30 21:30:02.530040 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:30:02.529851 5378 scope.go:122] "RemoveContainer" containerID="70f607f8a237c456950f87a28694287b40d9c3b36cf75acaa7ecbfc8f1aa1d17" Oct 30 21:30:03.536425 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:30:03.536391 5378 generic.go:356] "Generic (PLEG): container finished" podID="37d16441-9ce7-45e8-868d-ab0994bca2ba" containerID="89111178af2d4184a9227bd9ce532aa67df437fd322430c9e682567d2476850d" exitCode=60 Oct 30 21:30:03.536851 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:30:03.536478 5378 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889930-9jjd9" event={"ID":"37d16441-9ce7-45e8-868d-ab0994bca2ba","Type":"ContainerDied","Data":"89111178af2d4184a9227bd9ce532aa67df437fd322430c9e682567d2476850d"} Oct 30 21:30:03.536851 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:30:03.536529 5378 scope.go:122] "RemoveContainer" containerID="70f607f8a237c456950f87a28694287b40d9c3b36cf75acaa7ecbfc8f1aa1d17" Oct 30 21:30:03.536851 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:30:03.536798 5378 scope.go:122] "RemoveContainer" containerID="89111178af2d4184a9227bd9ce532aa67df437fd322430c9e682567d2476850d" Oct 30 21:30:03.537061 ip-10-0-138-149 kubenswrapper[5378]: E1030 21:30:03.537036 5378 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29889930-9jjd9_odh-ai-gateway-infra(37d16441-9ce7-45e8-868d-ab0994bca2ba)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889930-9jjd9" podUID="37d16441-9ce7-45e8-868d-ab0994bca2ba" Oct 30 21:30:04.543444 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:30:04.543404 5378 scope.go:122] "RemoveContainer" containerID="89111178af2d4184a9227bd9ce532aa67df437fd322430c9e682567d2476850d" Oct 30 21:30:04.543869 ip-10-0-138-149 kubenswrapper[5378]: E1030 21:30:04.543660 5378 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29889930-9jjd9_odh-ai-gateway-infra(37d16441-9ce7-45e8-868d-ab0994bca2ba)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889930-9jjd9" podUID="37d16441-9ce7-45e8-868d-ab0994bca2ba" Oct 30 21:30:09.954748 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:30:09.954708 5378 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-r9g59"] Oct 30 21:30:11.312208 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:30:11.312155 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:30:11.452056 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:30:11.452023 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:30:13.734491 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:30:13.734453 5378 scope.go:122] "RemoveContainer" containerID="89111178af2d4184a9227bd9ce532aa67df437fd322430c9e682567d2476850d" Oct 30 21:30:14.600959 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:30:14.600931 5378 generic.go:356] "Generic (PLEG): container finished" podID="37d16441-9ce7-45e8-868d-ab0994bca2ba" containerID="5fdd636a75abdf2bb4736ba9740e799a06f805b948abd3ee7eefd3018d450620" exitCode=60 Oct 30 21:30:14.601101 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:30:14.600999 5378 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889930-9jjd9" event={"ID":"37d16441-9ce7-45e8-868d-ab0994bca2ba","Type":"ContainerDied","Data":"5fdd636a75abdf2bb4736ba9740e799a06f805b948abd3ee7eefd3018d450620"} Oct 30 21:30:14.601101 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:30:14.601067 5378 scope.go:122] "RemoveContainer" containerID="89111178af2d4184a9227bd9ce532aa67df437fd322430c9e682567d2476850d" Oct 30 21:30:14.601392 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:30:14.601376 5378 scope.go:122] "RemoveContainer" containerID="5fdd636a75abdf2bb4736ba9740e799a06f805b948abd3ee7eefd3018d450620" Oct 30 21:30:14.601672 ip-10-0-138-149 kubenswrapper[5378]: E1030 21:30:14.601642 5378 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-29889930-9jjd9_odh-ai-gateway-infra(37d16441-9ce7-45e8-868d-ab0994bca2ba)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889930-9jjd9" podUID="37d16441-9ce7-45e8-868d-ab0994bca2ba" Oct 30 21:30:15.725533 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:30:15.725499 5378 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889930-9jjd9"] Oct 30 21:30:15.870214 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:30:15.870187 5378 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889930-9jjd9" Oct 30 21:30:15.951472 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:30:15.951440 5378 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/37d16441-9ce7-45e8-868d-ab0994bca2ba-kube-api-access-kcpmf\" (UniqueName: \"kubernetes.io/projected/37d16441-9ce7-45e8-868d-ab0994bca2ba-kube-api-access-kcpmf\") pod \"37d16441-9ce7-45e8-868d-ab0994bca2ba\" (UID: \"37d16441-9ce7-45e8-868d-ab0994bca2ba\") " Oct 30 21:30:15.951695 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:30:15.951483 5378 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/37d16441-9ce7-45e8-868d-ab0994bca2ba-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/37d16441-9ce7-45e8-868d-ab0994bca2ba-maas-api-service-ca\") pod \"37d16441-9ce7-45e8-868d-ab0994bca2ba\" (UID: \"37d16441-9ce7-45e8-868d-ab0994bca2ba\") " Oct 30 21:30:15.951876 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:30:15.951853 5378 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/37d16441-9ce7-45e8-868d-ab0994bca2ba-maas-api-service-ca" pod "37d16441-9ce7-45e8-868d-ab0994bca2ba" (UID: "37d16441-9ce7-45e8-868d-ab0994bca2ba"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:30:15.953678 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:30:15.953656 5378 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/37d16441-9ce7-45e8-868d-ab0994bca2ba-kube-api-access-kcpmf" pod "37d16441-9ce7-45e8-868d-ab0994bca2ba" (UID: "37d16441-9ce7-45e8-868d-ab0994bca2ba"). InnerVolumeSpecName "kube-api-access-kcpmf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:30:16.052974 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:30:16.052897 5378 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/37d16441-9ce7-45e8-868d-ab0994bca2ba-maas-api-service-ca\") on node \"ip-10-0-138-149.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:30:16.052974 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:30:16.052923 5378 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kcpmf\" (UniqueName: \"kubernetes.io/projected/37d16441-9ce7-45e8-868d-ab0994bca2ba-kube-api-access-kcpmf\") on node \"ip-10-0-138-149.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:30:16.612800 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:30:16.612770 5378 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889930-9jjd9" Oct 30 21:30:16.612955 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:30:16.612770 5378 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889930-9jjd9" event={"ID":"37d16441-9ce7-45e8-868d-ab0994bca2ba","Type":"ContainerDied","Data":"02f9339d5ea08bbfe0d19d8e68031d12e0e2bef30fb0c591d8f6ba18d5bbe565"} Oct 30 21:30:16.612955 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:30:16.612939 5378 scope.go:122] "RemoveContainer" containerID="5fdd636a75abdf2bb4736ba9740e799a06f805b948abd3ee7eefd3018d450620" Oct 30 21:30:16.778106 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:30:16.778073 5378 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889930-9jjd9"] Oct 30 21:30:16.828776 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:30:16.828746 5378 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889930-9jjd9"] Oct 30 21:30:17.739847 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:30:17.739812 5378 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="37d16441-9ce7-45e8-868d-ab0994bca2ba" path="/var/lib/kubelet/pods/37d16441-9ce7-45e8-868d-ab0994bca2ba/volumes" Oct 30 21:30:18.836428 ip-10-0-138-149 kubenswrapper[5378]: E1030 21:30:18.836390 5378 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q6mx7" containerName="extractor" Oct 30 21:30:20.690192 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:30:20.690158 5378 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-r9g59"] Oct 30 21:30:23.388826 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:30:23.388789 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:30:33.241278 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:30:33.241237 5378 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-r9g59"] Oct 30 21:30:48.835598 ip-10-0-138-149 kubenswrapper[5378]: E1030 21:30:48.835546 5378 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q6mx7" containerName="extractor" Oct 30 21:30:55.199694 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:30:55.199657 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:30:55.671528 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:30:55.671495 5378 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-r9g59"] Oct 30 21:31:05.187875 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:31:05.187839 5378 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-r9g59"] Oct 30 21:31:10.982826 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:31:10.982747 5378 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-k97m2"] Oct 30 21:31:10.983452 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:31:10.983428 5378 cpu_manager.go:414] "containerMap: removing container" podUID="37d16441-9ce7-45e8-868d-ab0994bca2ba" containerName="cleanup" Oct 30 21:31:10.983517 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:31:10.983458 5378 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="37d16441-9ce7-45e8-868d-ab0994bca2ba" containerName="cleanup" Oct 30 21:31:10.983517 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:31:10.983482 5378 cpu_manager.go:414] "containerMap: removing container" podUID="37d16441-9ce7-45e8-868d-ab0994bca2ba" containerName="cleanup" Oct 30 21:31:10.983517 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:31:10.983490 5378 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="37d16441-9ce7-45e8-868d-ab0994bca2ba" containerName="cleanup" Oct 30 21:31:10.983658 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:31:10.983602 5378 memory_manager.go:365] "RemoveStaleState removing state" podUID="37d16441-9ce7-45e8-868d-ab0994bca2ba" containerName="cleanup" Oct 30 21:31:10.983658 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:31:10.983615 5378 memory_manager.go:365] "RemoveStaleState removing state" podUID="37d16441-9ce7-45e8-868d-ab0994bca2ba" containerName="cleanup" Oct 30 21:31:10.987047 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:31:10.987029 5378 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-k97m2" Oct 30 21:31:11.042973 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:31:11.042935 5378 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-k97m2"] Oct 30 21:31:11.051037 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:31:11.051003 5378 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cbc20c43-c3d2-483c-946c-6e7b601e4bdb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a4aef7d7-k97m2\" (UID: \"cbc20c43-c3d2-483c-946c-6e7b601e4bdb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-k97m2" Oct 30 21:31:11.051328 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:31:11.051296 5378 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cbc20c43-c3d2-483c-946c-6e7b601e4bdb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a4aef7d7-k97m2\" (UID: \"cbc20c43-c3d2-483c-946c-6e7b601e4bdb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-k97m2" Oct 30 21:31:11.051476 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:31:11.051456 5378 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jmcxp\" (UniqueName: \"kubernetes.io/projected/cbc20c43-c3d2-483c-946c-6e7b601e4bdb-kube-api-access-jmcxp\") pod \"maas-api-revoke-sub-a4aef7d7-k97m2\" (UID: \"cbc20c43-c3d2-483c-946c-6e7b601e4bdb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-k97m2" Oct 30 21:31:11.090447 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:31:11.090399 5378 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 21:31:11.090447 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:31:11.090408 5378 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 21:31:11.090447 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:31:11.090422 5378 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-q2mgc\"" Oct 30 21:31:11.090447 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:31:11.090436 5378 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 21:31:11.152080 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:31:11.152051 5378 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cbc20c43-c3d2-483c-946c-6e7b601e4bdb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a4aef7d7-k97m2\" (UID: \"cbc20c43-c3d2-483c-946c-6e7b601e4bdb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-k97m2" Oct 30 21:31:11.152194 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:31:11.152106 5378 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jmcxp\" (UniqueName: \"kubernetes.io/projected/cbc20c43-c3d2-483c-946c-6e7b601e4bdb-kube-api-access-jmcxp\") pod \"maas-api-revoke-sub-a4aef7d7-k97m2\" (UID: \"cbc20c43-c3d2-483c-946c-6e7b601e4bdb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-k97m2" Oct 30 21:31:11.152194 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:31:11.152164 5378 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cbc20c43-c3d2-483c-946c-6e7b601e4bdb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a4aef7d7-k97m2\" (UID: \"cbc20c43-c3d2-483c-946c-6e7b601e4bdb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-k97m2" Oct 30 21:31:11.152731 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:31:11.152709 5378 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cbc20c43-c3d2-483c-946c-6e7b601e4bdb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a4aef7d7-k97m2\" (UID: \"cbc20c43-c3d2-483c-946c-6e7b601e4bdb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-k97m2" Oct 30 21:31:11.154756 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:31:11.154734 5378 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cbc20c43-c3d2-483c-946c-6e7b601e4bdb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a4aef7d7-k97m2\" (UID: \"cbc20c43-c3d2-483c-946c-6e7b601e4bdb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-k97m2" Oct 30 21:31:11.209954 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:31:11.209929 5378 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jmcxp\" (UniqueName: \"kubernetes.io/projected/cbc20c43-c3d2-483c-946c-6e7b601e4bdb-kube-api-access-jmcxp\") pod \"maas-api-revoke-sub-a4aef7d7-k97m2\" (UID: \"cbc20c43-c3d2-483c-946c-6e7b601e4bdb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-k97m2" Oct 30 21:31:11.299385 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:31:11.299317 5378 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-k97m2" Oct 30 21:31:11.667402 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:31:11.667379 5378 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-k97m2"] Oct 30 21:31:11.669542 ip-10-0-138-149 kubenswrapper[5378]: W1030 21:31:11.669505 5378 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcbc20c43_c3d2_483c_946c_6e7b601e4bdb.slice/crio-9638774c0a99c3d7512c226f27863a6d057023945ce079dc7eff61774041342b WatchSource:0}: Error finding container 9638774c0a99c3d7512c226f27863a6d057023945ce079dc7eff61774041342b: Status 404 returned error can't find the container with id 9638774c0a99c3d7512c226f27863a6d057023945ce079dc7eff61774041342b Oct 30 21:31:11.924979 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:31:11.924905 5378 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-k97m2" event={"ID":"cbc20c43-c3d2-483c-946c-6e7b601e4bdb","Type":"ContainerStarted","Data":"9638774c0a99c3d7512c226f27863a6d057023945ce079dc7eff61774041342b"} Oct 30 21:31:12.932047 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:31:12.932012 5378 generic.go:356] "Generic (PLEG): container finished" podID="cbc20c43-c3d2-483c-946c-6e7b601e4bdb" containerID="0b3341bdb04d660c4b8d97e3221169b925414ca69e962da934cd12a44d2cea6d" exitCode=0 Oct 30 21:31:12.932511 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:31:12.932100 5378 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-k97m2" event={"ID":"cbc20c43-c3d2-483c-946c-6e7b601e4bdb","Type":"ContainerDied","Data":"0b3341bdb04d660c4b8d97e3221169b925414ca69e962da934cd12a44d2cea6d"} Oct 30 21:31:14.069701 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:31:14.068083 5378 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-k97m2" Oct 30 21:31:14.178403 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:31:14.178375 5378 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cbc20c43-c3d2-483c-946c-6e7b601e4bdb-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cbc20c43-c3d2-483c-946c-6e7b601e4bdb-maas-api-openshift-ca\") pod \"cbc20c43-c3d2-483c-946c-6e7b601e4bdb\" (UID: \"cbc20c43-c3d2-483c-946c-6e7b601e4bdb\") " Oct 30 21:31:14.178596 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:31:14.178410 5378 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cbc20c43-c3d2-483c-946c-6e7b601e4bdb-kube-api-access-jmcxp\" (UniqueName: \"kubernetes.io/projected/cbc20c43-c3d2-483c-946c-6e7b601e4bdb-kube-api-access-jmcxp\") pod \"cbc20c43-c3d2-483c-946c-6e7b601e4bdb\" (UID: \"cbc20c43-c3d2-483c-946c-6e7b601e4bdb\") " Oct 30 21:31:14.178596 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:31:14.178435 5378 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/cbc20c43-c3d2-483c-946c-6e7b601e4bdb-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cbc20c43-c3d2-483c-946c-6e7b601e4bdb-maas-api-xks-ca\") pod \"cbc20c43-c3d2-483c-946c-6e7b601e4bdb\" (UID: \"cbc20c43-c3d2-483c-946c-6e7b601e4bdb\") " Oct 30 21:31:14.178747 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:31:14.178716 5378 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cbc20c43-c3d2-483c-946c-6e7b601e4bdb-maas-api-openshift-ca" pod "cbc20c43-c3d2-483c-946c-6e7b601e4bdb" (UID: "cbc20c43-c3d2-483c-946c-6e7b601e4bdb"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:31:14.180460 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:31:14.180436 5378 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cbc20c43-c3d2-483c-946c-6e7b601e4bdb-maas-api-xks-ca" pod "cbc20c43-c3d2-483c-946c-6e7b601e4bdb" (UID: "cbc20c43-c3d2-483c-946c-6e7b601e4bdb"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:31:14.180613 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:31:14.180494 5378 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cbc20c43-c3d2-483c-946c-6e7b601e4bdb-kube-api-access-jmcxp" pod "cbc20c43-c3d2-483c-946c-6e7b601e4bdb" (UID: "cbc20c43-c3d2-483c-946c-6e7b601e4bdb"). InnerVolumeSpecName "kube-api-access-jmcxp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:31:14.279324 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:31:14.279294 5378 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cbc20c43-c3d2-483c-946c-6e7b601e4bdb-maas-api-openshift-ca\") on node \"ip-10-0-138-149.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:31:14.279324 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:31:14.279321 5378 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jmcxp\" (UniqueName: \"kubernetes.io/projected/cbc20c43-c3d2-483c-946c-6e7b601e4bdb-kube-api-access-jmcxp\") on node \"ip-10-0-138-149.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:31:14.279508 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:31:14.279332 5378 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cbc20c43-c3d2-483c-946c-6e7b601e4bdb-maas-api-xks-ca\") on node \"ip-10-0-138-149.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:31:14.941386 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:31:14.941355 5378 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-k97m2" Oct 30 21:31:14.941587 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:31:14.941356 5378 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-k97m2" event={"ID":"cbc20c43-c3d2-483c-946c-6e7b601e4bdb","Type":"ContainerDied","Data":"9638774c0a99c3d7512c226f27863a6d057023945ce079dc7eff61774041342b"} Oct 30 21:31:14.941587 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:31:14.941477 5378 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9638774c0a99c3d7512c226f27863a6d057023945ce079dc7eff61774041342b" Oct 30 21:31:18.835808 ip-10-0-138-149 kubenswrapper[5378]: E1030 21:31:18.835771 5378 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q6mx7" containerName="extractor" Oct 30 21:31:23.852526 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:31:23.852488 5378 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-r9g59"] Oct 30 21:31:32.819033 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:31:32.818942 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:31:36.232619 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:31:36.232585 5378 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-r9g59"] Oct 30 21:31:48.836349 ip-10-0-138-149 kubenswrapper[5378]: E1030 21:31:48.836307 5378 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q6mx7" containerName="extractor" Oct 30 21:31:54.219512 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:31:54.219467 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:31:56.252790 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:31:56.252753 5378 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-r9g59"] Oct 30 21:32:15.350412 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:32:15.350373 5378 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-r9g59"] Oct 30 21:32:17.332703 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:32:17.332665 5378 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-769df7846c-xghmf container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.40:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 21:32:17.333356 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:32:17.333320 5378 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-xghmf" podUID="f93de8dc-026f-4868-94fb-d1b2006dfb63" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.40:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 21:32:18.836612 ip-10-0-138-149 kubenswrapper[5378]: E1030 21:32:18.836558 5378 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q6mx7" containerName="extractor" Oct 30 21:32:24.397281 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:32:24.397248 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:32:28.370193 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:32:28.370161 5378 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-r9g59"] Oct 30 21:32:33.053382 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:32:33.053347 5378 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-x75wp"] Oct 30 21:32:33.102888 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:32:33.102860 5378 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-x75wp"] Oct 30 21:32:33.739895 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:32:33.739863 5378 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ac2d133c-d6a1-4bba-b6f6-a201a06ac557" path="/var/lib/kubelet/pods/ac2d133c-d6a1-4bba-b6f6-a201a06ac557/volumes" Oct 30 21:32:48.836910 ip-10-0-138-149 kubenswrapper[5378]: E1030 21:32:48.836832 5378 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q6mx7" containerName="extractor" Oct 30 21:32:54.553476 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:32:54.553441 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:33:01.185534 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:33:01.185499 5378 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-r9g59"] Oct 30 21:33:18.835896 ip-10-0-138-149 kubenswrapper[5378]: E1030 21:33:18.835857 5378 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q6mx7" containerName="extractor" Oct 30 21:33:20.851651 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:33:20.851614 5378 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-z588j"] Oct 30 21:33:20.852328 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:33:20.852307 5378 cpu_manager.go:414] "containerMap: removing container" podUID="cbc20c43-c3d2-483c-946c-6e7b601e4bdb" containerName="revoke-keys" Oct 30 21:33:20.852424 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:33:20.852331 5378 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cbc20c43-c3d2-483c-946c-6e7b601e4bdb" containerName="revoke-keys" Oct 30 21:33:20.852424 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:33:20.852344 5378 cpu_manager.go:414] "containerMap: removing container" podUID="37d16441-9ce7-45e8-868d-ab0994bca2ba" containerName="cleanup" Oct 30 21:33:20.852424 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:33:20.852351 5378 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="37d16441-9ce7-45e8-868d-ab0994bca2ba" containerName="cleanup" Oct 30 21:33:20.852614 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:33:20.852462 5378 memory_manager.go:365] "RemoveStaleState removing state" podUID="cbc20c43-c3d2-483c-946c-6e7b601e4bdb" containerName="revoke-keys" Oct 30 21:33:20.852614 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:33:20.852476 5378 memory_manager.go:365] "RemoveStaleState removing state" podUID="37d16441-9ce7-45e8-868d-ab0994bca2ba" containerName="cleanup" Oct 30 21:33:20.855879 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:33:20.855852 5378 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-z588j" Oct 30 21:33:20.910836 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:33:20.910795 5378 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-z588j"] Oct 30 21:33:20.959441 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:33:20.959410 5378 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 21:33:20.959634 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:33:20.959454 5378 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 21:33:20.959634 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:33:20.959493 5378 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 21:33:20.959634 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:33:20.959529 5378 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-q2mgc\"" Oct 30 21:33:21.027381 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:33:21.027354 5378 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2zv59\" (UniqueName: \"kubernetes.io/projected/fcfaf80d-b3f8-4cb2-9f58-f1ce1e83b7ae-kube-api-access-2zv59\") pod \"maas-api-revoke-sub-5a29efb3-z588j\" (UID: \"fcfaf80d-b3f8-4cb2-9f58-f1ce1e83b7ae\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-z588j" Oct 30 21:33:21.027545 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:33:21.027415 5378 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fcfaf80d-b3f8-4cb2-9f58-f1ce1e83b7ae-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5a29efb3-z588j\" (UID: \"fcfaf80d-b3f8-4cb2-9f58-f1ce1e83b7ae\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-z588j" Oct 30 21:33:21.027545 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:33:21.027513 5378 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fcfaf80d-b3f8-4cb2-9f58-f1ce1e83b7ae-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5a29efb3-z588j\" (UID: \"fcfaf80d-b3f8-4cb2-9f58-f1ce1e83b7ae\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-z588j" Oct 30 21:33:21.128444 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:33:21.128356 5378 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fcfaf80d-b3f8-4cb2-9f58-f1ce1e83b7ae-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5a29efb3-z588j\" (UID: \"fcfaf80d-b3f8-4cb2-9f58-f1ce1e83b7ae\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-z588j" Oct 30 21:33:21.128444 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:33:21.128416 5378 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2zv59\" (UniqueName: \"kubernetes.io/projected/fcfaf80d-b3f8-4cb2-9f58-f1ce1e83b7ae-kube-api-access-2zv59\") pod \"maas-api-revoke-sub-5a29efb3-z588j\" (UID: \"fcfaf80d-b3f8-4cb2-9f58-f1ce1e83b7ae\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-z588j" Oct 30 21:33:21.128720 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:33:21.128465 5378 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fcfaf80d-b3f8-4cb2-9f58-f1ce1e83b7ae-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5a29efb3-z588j\" (UID: \"fcfaf80d-b3f8-4cb2-9f58-f1ce1e83b7ae\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-z588j" Oct 30 21:33:21.129181 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:33:21.129160 5378 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fcfaf80d-b3f8-4cb2-9f58-f1ce1e83b7ae-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5a29efb3-z588j\" (UID: \"fcfaf80d-b3f8-4cb2-9f58-f1ce1e83b7ae\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-z588j" Oct 30 21:33:21.130837 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:33:21.130804 5378 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fcfaf80d-b3f8-4cb2-9f58-f1ce1e83b7ae-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5a29efb3-z588j\" (UID: \"fcfaf80d-b3f8-4cb2-9f58-f1ce1e83b7ae\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-z588j" Oct 30 21:33:21.186019 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:33:21.185991 5378 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2zv59\" (UniqueName: \"kubernetes.io/projected/fcfaf80d-b3f8-4cb2-9f58-f1ce1e83b7ae-kube-api-access-2zv59\") pod \"maas-api-revoke-sub-5a29efb3-z588j\" (UID: \"fcfaf80d-b3f8-4cb2-9f58-f1ce1e83b7ae\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-z588j" Oct 30 21:33:21.470243 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:33:21.470166 5378 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-z588j" Oct 30 21:33:21.841478 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:33:21.841452 5378 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-z588j"] Oct 30 21:33:21.843637 ip-10-0-138-149 kubenswrapper[5378]: W1030 21:33:21.843609 5378 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfcfaf80d_b3f8_4cb2_9f58_f1ce1e83b7ae.slice/crio-69d4a768fc23613951cc8957e978d5b77de0a057d77f987b35d60a87ea494cd4 WatchSource:0}: Error finding container 69d4a768fc23613951cc8957e978d5b77de0a057d77f987b35d60a87ea494cd4: Status 404 returned error can't find the container with id 69d4a768fc23613951cc8957e978d5b77de0a057d77f987b35d60a87ea494cd4 Oct 30 21:33:22.681288 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:33:22.681247 5378 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-z588j" event={"ID":"fcfaf80d-b3f8-4cb2-9f58-f1ce1e83b7ae","Type":"ContainerStarted","Data":"31f5d5f1e6fb6ba7285844614902b6fcc38cf88a796b9c6cf8cffa31a34f1470"} Oct 30 21:33:22.681827 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:33:22.681306 5378 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-z588j" event={"ID":"fcfaf80d-b3f8-4cb2-9f58-f1ce1e83b7ae","Type":"ContainerStarted","Data":"69d4a768fc23613951cc8957e978d5b77de0a057d77f987b35d60a87ea494cd4"} Oct 30 21:33:22.797020 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:33:22.796968 5378 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-z588j" podStartSLOduration=2.79695447 podStartE2EDuration="2.79695447s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:33:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:33:22.796082434 +0000 UTC m=+3293.224196354" watchObservedRunningTime="2026-10-30 21:33:22.79695447 +0000 UTC m=+3293.225068368" Oct 30 21:33:22.949144 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:33:22.949062 5378 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-r9g59"] Oct 30 21:33:23.691539 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:33:23.691500 5378 generic.go:356] "Generic (PLEG): container finished" podID="fcfaf80d-b3f8-4cb2-9f58-f1ce1e83b7ae" containerID="31f5d5f1e6fb6ba7285844614902b6fcc38cf88a796b9c6cf8cffa31a34f1470" exitCode=0 Oct 30 21:33:23.692137 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:33:23.691690 5378 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-z588j" event={"ID":"fcfaf80d-b3f8-4cb2-9f58-f1ce1e83b7ae","Type":"ContainerDied","Data":"31f5d5f1e6fb6ba7285844614902b6fcc38cf88a796b9c6cf8cffa31a34f1470"} Oct 30 21:33:25.705062 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:33:25.704738 5378 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-z588j" Oct 30 21:33:25.782596 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:33:25.779020 5378 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-z588j" Oct 30 21:33:25.782596 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:33:25.779703 5378 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fcfaf80d-b3f8-4cb2-9f58-f1ce1e83b7ae-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fcfaf80d-b3f8-4cb2-9f58-f1ce1e83b7ae-maas-api-xks-ca\") pod \"fcfaf80d-b3f8-4cb2-9f58-f1ce1e83b7ae\" (UID: \"fcfaf80d-b3f8-4cb2-9f58-f1ce1e83b7ae\") " Oct 30 21:33:25.782596 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:33:25.779763 5378 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fcfaf80d-b3f8-4cb2-9f58-f1ce1e83b7ae-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fcfaf80d-b3f8-4cb2-9f58-f1ce1e83b7ae-maas-api-openshift-ca\") pod \"fcfaf80d-b3f8-4cb2-9f58-f1ce1e83b7ae\" (UID: \"fcfaf80d-b3f8-4cb2-9f58-f1ce1e83b7ae\") " Oct 30 21:33:25.782596 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:33:25.779888 5378 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fcfaf80d-b3f8-4cb2-9f58-f1ce1e83b7ae-kube-api-access-2zv59\" (UniqueName: \"kubernetes.io/projected/fcfaf80d-b3f8-4cb2-9f58-f1ce1e83b7ae-kube-api-access-2zv59\") pod \"fcfaf80d-b3f8-4cb2-9f58-f1ce1e83b7ae\" (UID: \"fcfaf80d-b3f8-4cb2-9f58-f1ce1e83b7ae\") " Oct 30 21:33:25.792712 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:33:25.792682 5378 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fcfaf80d-b3f8-4cb2-9f58-f1ce1e83b7ae-maas-api-openshift-ca" pod "fcfaf80d-b3f8-4cb2-9f58-f1ce1e83b7ae" (UID: "fcfaf80d-b3f8-4cb2-9f58-f1ce1e83b7ae"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:33:25.814016 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:33:25.813986 5378 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fcfaf80d-b3f8-4cb2-9f58-f1ce1e83b7ae-maas-api-xks-ca" pod "fcfaf80d-b3f8-4cb2-9f58-f1ce1e83b7ae" (UID: "fcfaf80d-b3f8-4cb2-9f58-f1ce1e83b7ae"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:33:25.817827 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:33:25.817692 5378 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fcfaf80d-b3f8-4cb2-9f58-f1ce1e83b7ae-kube-api-access-2zv59" pod "fcfaf80d-b3f8-4cb2-9f58-f1ce1e83b7ae" (UID: "fcfaf80d-b3f8-4cb2-9f58-f1ce1e83b7ae"). InnerVolumeSpecName "kube-api-access-2zv59". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:33:25.844989 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:33:25.844944 5378 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-z588j" event={"ID":"fcfaf80d-b3f8-4cb2-9f58-f1ce1e83b7ae","Type":"ContainerDied","Data":"69d4a768fc23613951cc8957e978d5b77de0a057d77f987b35d60a87ea494cd4"} Oct 30 21:33:25.845222 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:33:25.845194 5378 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="69d4a768fc23613951cc8957e978d5b77de0a057d77f987b35d60a87ea494cd4" Oct 30 21:33:25.895885 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:33:25.895728 5378 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fcfaf80d-b3f8-4cb2-9f58-f1ce1e83b7ae-maas-api-xks-ca\") on node \"ip-10-0-138-149.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:33:25.895885 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:33:25.895778 5378 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fcfaf80d-b3f8-4cb2-9f58-f1ce1e83b7ae-maas-api-openshift-ca\") on node \"ip-10-0-138-149.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:33:25.895885 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:33:25.895798 5378 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2zv59\" (UniqueName: \"kubernetes.io/projected/fcfaf80d-b3f8-4cb2-9f58-f1ce1e83b7ae-kube-api-access-2zv59\") on node \"ip-10-0-138-149.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:33:30.202907 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:33:30.202880 5378 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-7x9jg_c2ff359c-cf92-4a5b-8610-f6eb41acbe73/ovn-acl-logging/0.log" Oct 30 21:33:30.215070 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:33:30.215038 5378 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-7x9jg_c2ff359c-cf92-4a5b-8610-f6eb41acbe73/ovn-acl-logging/0.log" Oct 30 21:33:32.105789 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:33:32.105759 5378 scope.go:122] "RemoveContainer" containerID="7ac3c2dc859196e5a12ba602831109ee176c438631b53d1584e4faad487e2c2f" Oct 30 21:33:33.680289 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:33:33.680252 5378 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-r9g59"] Oct 30 21:33:42.043366 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:33:42.043331 5378 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-8r6mj"] Oct 30 21:33:42.093731 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:33:42.093698 5378 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-8r6mj"] Oct 30 21:33:43.740290 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:33:43.740253 5378 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8252d1e2-e618-4753-a491-fcfe070b0c3c" path="/var/lib/kubelet/pods/8252d1e2-e618-4753-a491-fcfe070b0c3c/volumes" Oct 30 21:33:48.836807 ip-10-0-138-149 kubenswrapper[5378]: E1030 21:33:48.836773 5378 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q6mx7" containerName="extractor" Oct 30 21:33:54.263296 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:33:54.263256 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:33:54.764075 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:33:54.764044 5378 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-single-main-89af8e-openshift-default-6cc78f548fcl4bq"] Oct 30 21:33:54.764463 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:33:54.764450 5378 cpu_manager.go:414] "containerMap: removing container" podUID="fcfaf80d-b3f8-4cb2-9f58-f1ce1e83b7ae" containerName="revoke-keys" Oct 30 21:33:54.764514 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:33:54.764465 5378 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fcfaf80d-b3f8-4cb2-9f58-f1ce1e83b7ae" containerName="revoke-keys" Oct 30 21:33:54.764554 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:33:54.764521 5378 memory_manager.go:365] "RemoveStaleState removing state" podUID="fcfaf80d-b3f8-4cb2-9f58-f1ce1e83b7ae" containerName="revoke-keys" Oct 30 21:33:54.767863 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:33:54.767844 5378 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-single-main-89af8e-openshift-default-6cc78f548fcl4bq" Oct 30 21:33:54.851168 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:33:54.851127 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:33:54.879524 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:33:54.879481 5378 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-single-main-89af8e-openshift-default-6cc78f548fcl4bq"] Oct 30 21:33:54.879776 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:33:54.879726 5378 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-single-main-89af8e-openshift-default-dockercfg-8ll58\"" Oct 30 21:33:54.888653 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:33:54.888620 5378 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/67603b6b-0867-4ec2-b55f-1ed68f3b380e-workload-socket\") pod \"e2e-worker-single-main-89af8e-openshift-default-6cc78f548fcl4bq\" (UID: \"67603b6b-0867-4ec2-b55f-1ed68f3b380e\") " pod="openshift-ingress/e2e-worker-single-main-89af8e-openshift-default-6cc78f548fcl4bq" Oct 30 21:33:54.888653 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:33:54.888651 5378 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/67603b6b-0867-4ec2-b55f-1ed68f3b380e-istio-token\") pod \"e2e-worker-single-main-89af8e-openshift-default-6cc78f548fcl4bq\" (UID: \"67603b6b-0867-4ec2-b55f-1ed68f3b380e\") " pod="openshift-ingress/e2e-worker-single-main-89af8e-openshift-default-6cc78f548fcl4bq" Oct 30 21:33:54.888922 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:33:54.888681 5378 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/67603b6b-0867-4ec2-b55f-1ed68f3b380e-istiod-ca-cert\") pod \"e2e-worker-single-main-89af8e-openshift-default-6cc78f548fcl4bq\" (UID: \"67603b6b-0867-4ec2-b55f-1ed68f3b380e\") " pod="openshift-ingress/e2e-worker-single-main-89af8e-openshift-default-6cc78f548fcl4bq" Oct 30 21:33:54.888922 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:33:54.888856 5378 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/67603b6b-0867-4ec2-b55f-1ed68f3b380e-istio-envoy\") pod \"e2e-worker-single-main-89af8e-openshift-default-6cc78f548fcl4bq\" (UID: \"67603b6b-0867-4ec2-b55f-1ed68f3b380e\") " pod="openshift-ingress/e2e-worker-single-main-89af8e-openshift-default-6cc78f548fcl4bq" Oct 30 21:33:54.888922 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:33:54.888887 5378 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/67603b6b-0867-4ec2-b55f-1ed68f3b380e-workload-certs\") pod \"e2e-worker-single-main-89af8e-openshift-default-6cc78f548fcl4bq\" (UID: \"67603b6b-0867-4ec2-b55f-1ed68f3b380e\") " pod="openshift-ingress/e2e-worker-single-main-89af8e-openshift-default-6cc78f548fcl4bq" Oct 30 21:33:54.888922 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:33:54.888907 5378 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vsf6k\" (UniqueName: \"kubernetes.io/projected/67603b6b-0867-4ec2-b55f-1ed68f3b380e-kube-api-access-vsf6k\") pod \"e2e-worker-single-main-89af8e-openshift-default-6cc78f548fcl4bq\" (UID: \"67603b6b-0867-4ec2-b55f-1ed68f3b380e\") " pod="openshift-ingress/e2e-worker-single-main-89af8e-openshift-default-6cc78f548fcl4bq" Oct 30 21:33:54.889167 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:33:54.889003 5378 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/67603b6b-0867-4ec2-b55f-1ed68f3b380e-istio-data\") pod \"e2e-worker-single-main-89af8e-openshift-default-6cc78f548fcl4bq\" (UID: \"67603b6b-0867-4ec2-b55f-1ed68f3b380e\") " pod="openshift-ingress/e2e-worker-single-main-89af8e-openshift-default-6cc78f548fcl4bq" Oct 30 21:33:54.889167 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:33:54.889036 5378 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/67603b6b-0867-4ec2-b55f-1ed68f3b380e-istio-podinfo\") pod \"e2e-worker-single-main-89af8e-openshift-default-6cc78f548fcl4bq\" (UID: \"67603b6b-0867-4ec2-b55f-1ed68f3b380e\") " pod="openshift-ingress/e2e-worker-single-main-89af8e-openshift-default-6cc78f548fcl4bq" Oct 30 21:33:54.889167 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:33:54.889060 5378 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/67603b6b-0867-4ec2-b55f-1ed68f3b380e-credential-socket\") pod \"e2e-worker-single-main-89af8e-openshift-default-6cc78f548fcl4bq\" (UID: \"67603b6b-0867-4ec2-b55f-1ed68f3b380e\") " pod="openshift-ingress/e2e-worker-single-main-89af8e-openshift-default-6cc78f548fcl4bq" Oct 30 21:33:54.990067 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:33:54.990014 5378 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/67603b6b-0867-4ec2-b55f-1ed68f3b380e-istio-envoy\") pod \"e2e-worker-single-main-89af8e-openshift-default-6cc78f548fcl4bq\" (UID: \"67603b6b-0867-4ec2-b55f-1ed68f3b380e\") " pod="openshift-ingress/e2e-worker-single-main-89af8e-openshift-default-6cc78f548fcl4bq" Oct 30 21:33:54.990067 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:33:54.990072 5378 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/67603b6b-0867-4ec2-b55f-1ed68f3b380e-workload-certs\") pod \"e2e-worker-single-main-89af8e-openshift-default-6cc78f548fcl4bq\" (UID: \"67603b6b-0867-4ec2-b55f-1ed68f3b380e\") " pod="openshift-ingress/e2e-worker-single-main-89af8e-openshift-default-6cc78f548fcl4bq" Oct 30 21:33:54.990366 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:33:54.990105 5378 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vsf6k\" (UniqueName: \"kubernetes.io/projected/67603b6b-0867-4ec2-b55f-1ed68f3b380e-kube-api-access-vsf6k\") pod \"e2e-worker-single-main-89af8e-openshift-default-6cc78f548fcl4bq\" (UID: \"67603b6b-0867-4ec2-b55f-1ed68f3b380e\") " pod="openshift-ingress/e2e-worker-single-main-89af8e-openshift-default-6cc78f548fcl4bq" Oct 30 21:33:54.990366 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:33:54.990166 5378 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/67603b6b-0867-4ec2-b55f-1ed68f3b380e-istio-data\") pod \"e2e-worker-single-main-89af8e-openshift-default-6cc78f548fcl4bq\" (UID: \"67603b6b-0867-4ec2-b55f-1ed68f3b380e\") " pod="openshift-ingress/e2e-worker-single-main-89af8e-openshift-default-6cc78f548fcl4bq" Oct 30 21:33:54.990366 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:33:54.990200 5378 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/67603b6b-0867-4ec2-b55f-1ed68f3b380e-istio-podinfo\") pod \"e2e-worker-single-main-89af8e-openshift-default-6cc78f548fcl4bq\" (UID: \"67603b6b-0867-4ec2-b55f-1ed68f3b380e\") " pod="openshift-ingress/e2e-worker-single-main-89af8e-openshift-default-6cc78f548fcl4bq" Oct 30 21:33:54.990366 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:33:54.990237 5378 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/67603b6b-0867-4ec2-b55f-1ed68f3b380e-credential-socket\") pod \"e2e-worker-single-main-89af8e-openshift-default-6cc78f548fcl4bq\" (UID: \"67603b6b-0867-4ec2-b55f-1ed68f3b380e\") " pod="openshift-ingress/e2e-worker-single-main-89af8e-openshift-default-6cc78f548fcl4bq" Oct 30 21:33:54.993488 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:33:54.991821 5378 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/67603b6b-0867-4ec2-b55f-1ed68f3b380e-workload-socket\") pod \"e2e-worker-single-main-89af8e-openshift-default-6cc78f548fcl4bq\" (UID: \"67603b6b-0867-4ec2-b55f-1ed68f3b380e\") " pod="openshift-ingress/e2e-worker-single-main-89af8e-openshift-default-6cc78f548fcl4bq" Oct 30 21:33:54.993488 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:33:54.991857 5378 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/67603b6b-0867-4ec2-b55f-1ed68f3b380e-istio-token\") pod \"e2e-worker-single-main-89af8e-openshift-default-6cc78f548fcl4bq\" (UID: \"67603b6b-0867-4ec2-b55f-1ed68f3b380e\") " pod="openshift-ingress/e2e-worker-single-main-89af8e-openshift-default-6cc78f548fcl4bq" Oct 30 21:33:54.993488 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:33:54.991905 5378 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/67603b6b-0867-4ec2-b55f-1ed68f3b380e-istiod-ca-cert\") pod \"e2e-worker-single-main-89af8e-openshift-default-6cc78f548fcl4bq\" (UID: \"67603b6b-0867-4ec2-b55f-1ed68f3b380e\") " pod="openshift-ingress/e2e-worker-single-main-89af8e-openshift-default-6cc78f548fcl4bq" Oct 30 21:33:54.993488 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:33:54.993298 5378 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/67603b6b-0867-4ec2-b55f-1ed68f3b380e-istiod-ca-cert\") pod \"e2e-worker-single-main-89af8e-openshift-default-6cc78f548fcl4bq\" (UID: \"67603b6b-0867-4ec2-b55f-1ed68f3b380e\") " pod="openshift-ingress/e2e-worker-single-main-89af8e-openshift-default-6cc78f548fcl4bq" Oct 30 21:33:54.993789 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:33:54.993641 5378 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/67603b6b-0867-4ec2-b55f-1ed68f3b380e-istio-data\") pod \"e2e-worker-single-main-89af8e-openshift-default-6cc78f548fcl4bq\" (UID: \"67603b6b-0867-4ec2-b55f-1ed68f3b380e\") " pod="openshift-ingress/e2e-worker-single-main-89af8e-openshift-default-6cc78f548fcl4bq" Oct 30 21:33:54.993789 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:33:54.993648 5378 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/67603b6b-0867-4ec2-b55f-1ed68f3b380e-workload-certs\") pod \"e2e-worker-single-main-89af8e-openshift-default-6cc78f548fcl4bq\" (UID: \"67603b6b-0867-4ec2-b55f-1ed68f3b380e\") " pod="openshift-ingress/e2e-worker-single-main-89af8e-openshift-default-6cc78f548fcl4bq" Oct 30 21:33:54.993789 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:33:54.993765 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:33:54.994085 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:33:54.994062 5378 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/67603b6b-0867-4ec2-b55f-1ed68f3b380e-workload-socket\") pod \"e2e-worker-single-main-89af8e-openshift-default-6cc78f548fcl4bq\" (UID: \"67603b6b-0867-4ec2-b55f-1ed68f3b380e\") " pod="openshift-ingress/e2e-worker-single-main-89af8e-openshift-default-6cc78f548fcl4bq" Oct 30 21:33:54.994305 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:33:54.994286 5378 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/67603b6b-0867-4ec2-b55f-1ed68f3b380e-credential-socket\") pod \"e2e-worker-single-main-89af8e-openshift-default-6cc78f548fcl4bq\" (UID: \"67603b6b-0867-4ec2-b55f-1ed68f3b380e\") " pod="openshift-ingress/e2e-worker-single-main-89af8e-openshift-default-6cc78f548fcl4bq" Oct 30 21:33:54.998403 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:33:54.997963 5378 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/67603b6b-0867-4ec2-b55f-1ed68f3b380e-istio-envoy\") pod \"e2e-worker-single-main-89af8e-openshift-default-6cc78f548fcl4bq\" (UID: \"67603b6b-0867-4ec2-b55f-1ed68f3b380e\") " pod="openshift-ingress/e2e-worker-single-main-89af8e-openshift-default-6cc78f548fcl4bq" Oct 30 21:33:54.998403 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:33:54.998297 5378 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/67603b6b-0867-4ec2-b55f-1ed68f3b380e-istio-podinfo\") pod \"e2e-worker-single-main-89af8e-openshift-default-6cc78f548fcl4bq\" (UID: \"67603b6b-0867-4ec2-b55f-1ed68f3b380e\") " pod="openshift-ingress/e2e-worker-single-main-89af8e-openshift-default-6cc78f548fcl4bq" Oct 30 21:33:55.048465 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:33:55.048385 5378 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vsf6k\" (UniqueName: \"kubernetes.io/projected/67603b6b-0867-4ec2-b55f-1ed68f3b380e-kube-api-access-vsf6k\") pod \"e2e-worker-single-main-89af8e-openshift-default-6cc78f548fcl4bq\" (UID: \"67603b6b-0867-4ec2-b55f-1ed68f3b380e\") " pod="openshift-ingress/e2e-worker-single-main-89af8e-openshift-default-6cc78f548fcl4bq" Oct 30 21:33:55.051068 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:33:55.051041 5378 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/67603b6b-0867-4ec2-b55f-1ed68f3b380e-istio-token\") pod \"e2e-worker-single-main-89af8e-openshift-default-6cc78f548fcl4bq\" (UID: \"67603b6b-0867-4ec2-b55f-1ed68f3b380e\") " pod="openshift-ingress/e2e-worker-single-main-89af8e-openshift-default-6cc78f548fcl4bq" Oct 30 21:33:55.080416 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:33:55.080364 5378 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-single-main-89af8e-openshift-default-6cc78f548fcl4bq" Oct 30 21:33:55.113955 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:33:55.113923 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:33:55.509339 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:33:55.509315 5378 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-single-main-89af8e-openshift-default-6cc78f548fcl4bq"] Oct 30 21:33:55.511117 ip-10-0-138-149 kubenswrapper[5378]: W1030 21:33:55.511089 5378 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod67603b6b_0867_4ec2_b55f_1ed68f3b380e.slice/crio-c73dba65eb877ba63c889ba399b32a18969ae436a017697d922856aa221aa89c WatchSource:0}: Error finding container c73dba65eb877ba63c889ba399b32a18969ae436a017697d922856aa221aa89c: Status 404 returned error can't find the container with id c73dba65eb877ba63c889ba399b32a18969ae436a017697d922856aa221aa89c Oct 30 21:33:55.513340 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:33:55.513302 5378 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 21:33:55.513442 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:33:55.513376 5378 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 21:33:55.513442 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:33:55.513426 5378 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 21:33:55.917886 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:33:55.917846 5378 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-89af8e-openshift-default-6cc78f548fcl4bq" event={"ID":"67603b6b-0867-4ec2-b55f-1ed68f3b380e","Type":"ContainerStarted","Data":"375e56550afb57d3d34656bf8bb7b96819fe0ff7d3b0e97fc1557259d11ff9ef"} Oct 30 21:33:55.918087 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:33:55.917910 5378 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-89af8e-openshift-default-6cc78f548fcl4bq" event={"ID":"67603b6b-0867-4ec2-b55f-1ed68f3b380e","Type":"ContainerStarted","Data":"c73dba65eb877ba63c889ba399b32a18969ae436a017697d922856aa221aa89c"} Oct 30 21:33:56.036748 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:33:56.036698 5378 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-single-main-89af8e-openshift-default-6cc78f548fcl4bq" podStartSLOduration=2.03668391 podStartE2EDuration="2.03668391s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:33:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:33:56.035079335 +0000 UTC m=+3326.463193268" watchObservedRunningTime="2026-10-30 21:33:56.03668391 +0000 UTC m=+3326.464797894" Oct 30 21:33:56.080984 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:33:56.080957 5378 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-single-main-89af8e-openshift-default-6cc78f548fcl4bq" Oct 30 21:33:56.086207 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:33:56.086183 5378 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-single-main-89af8e-openshift-default-6cc78f548fcl4bq" Oct 30 21:33:56.922600 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:33:56.922559 5378 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-single-main-89af8e-openshift-default-6cc78f548fcl4bq" Oct 30 21:33:56.923633 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:33:56.923611 5378 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-single-main-89af8e-openshift-default-6cc78f548fcl4bq" Oct 30 21:33:59.021848 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:33:59.021797 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:33:59.181940 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:33:59.181895 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:33:59.310814 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:33:59.310732 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:34:02.267637 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:34:02.267602 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:34:02.447380 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:34:02.447347 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:34:05.311842 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:34:05.311812 5378 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-89af8e-86747f589-h6p2r"] Oct 30 21:34:05.315377 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:34:05.315357 5378 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-89af8e-86747f589-h6p2r" Oct 30 21:34:05.406046 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:34:05.406010 5378 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9a965ff4-aa23-407d-a1b2-1db32b8488a5-maas-api-tls\") pod \"maas-api-e2e-worker-single-main-89af8e-86747f589-h6p2r\" (UID: \"9a965ff4-aa23-407d-a1b2-1db32b8488a5\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-89af8e-86747f589-h6p2r" Oct 30 21:34:05.406239 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:34:05.406059 5378 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9a965ff4-aa23-407d-a1b2-1db32b8488a5-openshift-service-ca\") pod \"maas-api-e2e-worker-single-main-89af8e-86747f589-h6p2r\" (UID: \"9a965ff4-aa23-407d-a1b2-1db32b8488a5\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-89af8e-86747f589-h6p2r" Oct 30 21:34:05.406239 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:34:05.406137 5378 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9a965ff4-aa23-407d-a1b2-1db32b8488a5-metrics-certs\") pod \"maas-api-e2e-worker-single-main-89af8e-86747f589-h6p2r\" (UID: \"9a965ff4-aa23-407d-a1b2-1db32b8488a5\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-89af8e-86747f589-h6p2r" Oct 30 21:34:05.406239 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:34:05.406173 5378 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5jkhr\" (UniqueName: \"kubernetes.io/projected/9a965ff4-aa23-407d-a1b2-1db32b8488a5-kube-api-access-5jkhr\") pod \"maas-api-e2e-worker-single-main-89af8e-86747f589-h6p2r\" (UID: \"9a965ff4-aa23-407d-a1b2-1db32b8488a5\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-89af8e-86747f589-h6p2r" Oct 30 21:34:05.421003 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:34:05.420970 5378 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-single-main-89af8e-serving-cert\"" Oct 30 21:34:05.421003 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:34:05.420984 5378 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 21:34:05.421225 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:34:05.420988 5378 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 21:34:05.421225 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:34:05.421048 5378 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-6mqfn\"" Oct 30 21:34:05.421225 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:34:05.421079 5378 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\"" Oct 30 21:34:05.422711 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:34:05.422690 5378 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-89af8e-86747f589-h6p2r"] Oct 30 21:34:05.490872 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:34:05.490837 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:34:05.507254 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:34:05.507217 5378 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9a965ff4-aa23-407d-a1b2-1db32b8488a5-maas-api-tls\") pod \"maas-api-e2e-worker-single-main-89af8e-86747f589-h6p2r\" (UID: \"9a965ff4-aa23-407d-a1b2-1db32b8488a5\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-89af8e-86747f589-h6p2r" Oct 30 21:34:05.507490 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:34:05.507280 5378 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9a965ff4-aa23-407d-a1b2-1db32b8488a5-openshift-service-ca\") pod \"maas-api-e2e-worker-single-main-89af8e-86747f589-h6p2r\" (UID: \"9a965ff4-aa23-407d-a1b2-1db32b8488a5\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-89af8e-86747f589-h6p2r" Oct 30 21:34:05.507490 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:34:05.507316 5378 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9a965ff4-aa23-407d-a1b2-1db32b8488a5-metrics-certs\") pod \"maas-api-e2e-worker-single-main-89af8e-86747f589-h6p2r\" (UID: \"9a965ff4-aa23-407d-a1b2-1db32b8488a5\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-89af8e-86747f589-h6p2r" Oct 30 21:34:05.507490 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:34:05.507344 5378 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5jkhr\" (UniqueName: \"kubernetes.io/projected/9a965ff4-aa23-407d-a1b2-1db32b8488a5-kube-api-access-5jkhr\") pod \"maas-api-e2e-worker-single-main-89af8e-86747f589-h6p2r\" (UID: \"9a965ff4-aa23-407d-a1b2-1db32b8488a5\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-89af8e-86747f589-h6p2r" Oct 30 21:34:05.507490 ip-10-0-138-149 kubenswrapper[5378]: E1030 21:34:05.507374 5378 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-worker-single-main-89af8e-serving-cert: secret "maas-api-e2e-worker-single-main-89af8e-serving-cert" not found Oct 30 21:34:05.507490 ip-10-0-138-149 kubenswrapper[5378]: E1030 21:34:05.507444 5378 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9a965ff4-aa23-407d-a1b2-1db32b8488a5-maas-api-tls podName:9a965ff4-aa23-407d-a1b2-1db32b8488a5 nodeName:}" failed. No retries permitted until 2026-10-30 21:34:06.007424971 +0000 UTC m=+3336.435538869 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/9a965ff4-aa23-407d-a1b2-1db32b8488a5-maas-api-tls") pod "maas-api-e2e-worker-single-main-89af8e-86747f589-h6p2r" (UID: "9a965ff4-aa23-407d-a1b2-1db32b8488a5") : secret "maas-api-e2e-worker-single-main-89af8e-serving-cert" not found Oct 30 21:34:05.508064 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:34:05.508040 5378 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9a965ff4-aa23-407d-a1b2-1db32b8488a5-openshift-service-ca\") pod \"maas-api-e2e-worker-single-main-89af8e-86747f589-h6p2r\" (UID: \"9a965ff4-aa23-407d-a1b2-1db32b8488a5\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-89af8e-86747f589-h6p2r" Oct 30 21:34:05.510073 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:34:05.510048 5378 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9a965ff4-aa23-407d-a1b2-1db32b8488a5-metrics-certs\") pod \"maas-api-e2e-worker-single-main-89af8e-86747f589-h6p2r\" (UID: \"9a965ff4-aa23-407d-a1b2-1db32b8488a5\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-89af8e-86747f589-h6p2r" Oct 30 21:34:05.569642 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:34:05.569508 5378 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5jkhr\" (UniqueName: \"kubernetes.io/projected/9a965ff4-aa23-407d-a1b2-1db32b8488a5-kube-api-access-5jkhr\") pod \"maas-api-e2e-worker-single-main-89af8e-86747f589-h6p2r\" (UID: \"9a965ff4-aa23-407d-a1b2-1db32b8488a5\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-89af8e-86747f589-h6p2r" Oct 30 21:34:05.649758 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:34:05.649724 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:34:05.773247 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:34:05.773209 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:34:06.014943 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:34:06.014904 5378 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9a965ff4-aa23-407d-a1b2-1db32b8488a5-maas-api-tls\") pod \"maas-api-e2e-worker-single-main-89af8e-86747f589-h6p2r\" (UID: \"9a965ff4-aa23-407d-a1b2-1db32b8488a5\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-89af8e-86747f589-h6p2r" Oct 30 21:34:06.017515 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:34:06.017484 5378 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9a965ff4-aa23-407d-a1b2-1db32b8488a5-maas-api-tls\") pod \"maas-api-e2e-worker-single-main-89af8e-86747f589-h6p2r\" (UID: \"9a965ff4-aa23-407d-a1b2-1db32b8488a5\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-89af8e-86747f589-h6p2r" Oct 30 21:34:06.228194 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:34:06.228157 5378 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-89af8e-86747f589-h6p2r" Oct 30 21:34:06.603708 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:34:06.603678 5378 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-89af8e-86747f589-h6p2r"] Oct 30 21:34:06.605126 ip-10-0-138-149 kubenswrapper[5378]: W1030 21:34:06.605099 5378 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9a965ff4_aa23_407d_a1b2_1db32b8488a5.slice/crio-7788faabd4693bc131444c3b6afe68994ed8d017d7b98fbf9360fcd26620d7cb WatchSource:0}: Error finding container 7788faabd4693bc131444c3b6afe68994ed8d017d7b98fbf9360fcd26620d7cb: Status 404 returned error can't find the container with id 7788faabd4693bc131444c3b6afe68994ed8d017d7b98fbf9360fcd26620d7cb Oct 30 21:34:06.606467 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:34:06.606448 5378 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 30 21:34:06.969905 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:34:06.969817 5378 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-89af8e-86747f589-h6p2r" event={"ID":"9a965ff4-aa23-407d-a1b2-1db32b8488a5","Type":"ContainerStarted","Data":"7788faabd4693bc131444c3b6afe68994ed8d017d7b98fbf9360fcd26620d7cb"} Oct 30 21:34:07.977060 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:34:07.977014 5378 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-89af8e-86747f589-h6p2r" event={"ID":"9a965ff4-aa23-407d-a1b2-1db32b8488a5","Type":"ContainerStarted","Data":"bbd02bbc5256293ca23c38764c1649ccfc961d82077ba2d82ccd8a7984970ceb"} Oct 30 21:34:07.977533 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:34:07.977098 5378 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-89af8e-86747f589-h6p2r" Oct 30 21:34:08.099301 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:34:08.099248 5378 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-89af8e-86747f589-h6p2r" podStartSLOduration=2.122199707 podStartE2EDuration="3.099235542s" totalImagesPullingTime="977.035835ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:34:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:34:08.098093475 +0000 UTC m=+3338.526207392" watchObservedRunningTime="2026-10-30 21:34:08.099235542 +0000 UTC m=+3338.527349461" Oct 30 21:34:10.484982 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:34:10.484900 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:34:13.370999 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:34:13.370963 5378 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-r9g59"] Oct 30 21:34:13.988966 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:34:13.988929 5378 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-89af8e-86747f589-h6p2r" Oct 30 21:34:15.394673 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:34:15.394632 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:34:18.835636 ip-10-0-138-149 kubenswrapper[5378]: E1030 21:34:18.835598 5378 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q6mx7" containerName="extractor" Oct 30 21:34:20.279263 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:34:20.279130 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:34:23.057953 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:34:23.057858 5378 generic.go:356] "Generic (PLEG): container finished" podID="67603b6b-0867-4ec2-b55f-1ed68f3b380e" containerID="375e56550afb57d3d34656bf8bb7b96819fe0ff7d3b0e97fc1557259d11ff9ef" exitCode=0 Oct 30 21:34:23.057953 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:34:23.057929 5378 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-89af8e-openshift-default-6cc78f548fcl4bq" event={"ID":"67603b6b-0867-4ec2-b55f-1ed68f3b380e","Type":"ContainerDied","Data":"375e56550afb57d3d34656bf8bb7b96819fe0ff7d3b0e97fc1557259d11ff9ef"} Oct 30 21:34:23.058433 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:34:23.058383 5378 scope.go:122] "RemoveContainer" containerID="375e56550afb57d3d34656bf8bb7b96819fe0ff7d3b0e97fc1557259d11ff9ef" Oct 30 21:34:23.059818 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:34:23.059779 5378 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 21:34:23.059929 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:34:23.059874 5378 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 21:34:23.059929 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:34:23.059919 5378 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 21:34:23.869182 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:34:23.869148 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:34:24.080471 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:34:24.080429 5378 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-89af8e-openshift-default-6cc78f548fcl4bq" event={"ID":"67603b6b-0867-4ec2-b55f-1ed68f3b380e","Type":"ContainerStarted","Data":"d86ab3c2b830c49e152500654004c8b3afe7960a659a21d98b1af0fb5c0b0cde"} Oct 30 21:34:24.479713 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:34:24.479679 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:34:24.619266 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:34:24.619231 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:34:24.758175 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:34:24.758061 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:34:25.082885 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:34:25.082789 5378 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-single-main-89af8e-openshift-default-6cc78f548fcl4bq" Oct 30 21:34:25.084708 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:34:25.084677 5378 patch_prober.go:29] interesting pod/e2e-worker-single-main-89af8e-openshift-default-6cc78f548fcl4bq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.176:15021/healthz/ready\": dial tcp 10.133.0.176:15021: connect: connection refused" start-of-body= Oct 30 21:34:25.084833 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:34:25.084744 5378 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-89af8e-openshift-default-6cc78f548fcl4bq" podUID="67603b6b-0867-4ec2-b55f-1ed68f3b380e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.176:15021/healthz/ready\": dial tcp 10.133.0.176:15021: connect: connection refused" Oct 30 21:34:25.416859 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:34:25.416829 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:34:26.081098 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:34:26.081064 5378 patch_prober.go:29] interesting pod/e2e-worker-single-main-89af8e-openshift-default-6cc78f548fcl4bq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.176:15021/healthz/ready\": dial tcp 10.133.0.176:15021: connect: connection refused" start-of-body= Oct 30 21:34:26.081283 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:34:26.081124 5378 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-89af8e-openshift-default-6cc78f548fcl4bq" podUID="67603b6b-0867-4ec2-b55f-1ed68f3b380e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.176:15021/healthz/ready\": dial tcp 10.133.0.176:15021: connect: connection refused" Oct 30 21:34:26.922488 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:34:26.922437 5378 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-single-main-89af8e-openshift-default-6cc78f548fcl4bq" Oct 30 21:34:27.081715 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:34:27.081676 5378 patch_prober.go:29] interesting pod/e2e-worker-single-main-89af8e-openshift-default-6cc78f548fcl4bq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.176:15021/healthz/ready\": dial tcp 10.133.0.176:15021: connect: connection refused" start-of-body= Oct 30 21:34:27.081913 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:34:27.081742 5378 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-89af8e-openshift-default-6cc78f548fcl4bq" podUID="67603b6b-0867-4ec2-b55f-1ed68f3b380e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.176:15021/healthz/ready\": dial tcp 10.133.0.176:15021: connect: connection refused" Oct 30 21:34:27.094363 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:34:27.094328 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:34:28.081291 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:34:28.081255 5378 patch_prober.go:29] interesting pod/e2e-worker-single-main-89af8e-openshift-default-6cc78f548fcl4bq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.176:15021/healthz/ready\": dial tcp 10.133.0.176:15021: connect: connection refused" start-of-body= Oct 30 21:34:28.081736 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:34:28.081313 5378 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-89af8e-openshift-default-6cc78f548fcl4bq" podUID="67603b6b-0867-4ec2-b55f-1ed68f3b380e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.176:15021/healthz/ready\": dial tcp 10.133.0.176:15021: connect: connection refused" Oct 30 21:34:29.081087 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:34:29.081046 5378 patch_prober.go:29] interesting pod/e2e-worker-single-main-89af8e-openshift-default-6cc78f548fcl4bq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.176:15021/healthz/ready\": dial tcp 10.133.0.176:15021: connect: connection refused" start-of-body= Oct 30 21:34:29.081270 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:34:29.081116 5378 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-89af8e-openshift-default-6cc78f548fcl4bq" podUID="67603b6b-0867-4ec2-b55f-1ed68f3b380e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.176:15021/healthz/ready\": dial tcp 10.133.0.176:15021: connect: connection refused" Oct 30 21:34:30.085156 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:34:30.085127 5378 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-single-main-89af8e-openshift-default-6cc78f548fcl4bq" Oct 30 21:34:30.086108 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:34:30.086088 5378 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-single-main-89af8e-openshift-default-6cc78f548fcl4bq" Oct 30 21:34:30.164862 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:34:30.164827 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:34:31.564060 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:34:31.564026 5378 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-r9g59"] Oct 30 21:34:35.047356 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:34:35.047319 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:34:39.858621 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:34:39.858578 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:34:44.660496 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:34:44.660463 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:34:47.132961 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:34:47.132929 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:34:47.253296 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:34:47.253269 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:34:48.835979 ip-10-0-138-149 kubenswrapper[5378]: E1030 21:34:48.835940 5378 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q6mx7" containerName="extractor" Oct 30 21:34:49.458975 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:34:49.458916 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:34:54.256341 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:34:54.256309 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:34:55.226485 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:34:55.226449 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:34:57.777650 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:34:57.777613 5378 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-r9g59"] Oct 30 21:34:59.075630 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:34:59.075599 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:35:03.867096 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:35:03.867053 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:35:08.779683 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:35:08.779649 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:35:13.586289 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:35:13.586257 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:35:18.367038 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:35:18.367001 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:35:18.490795 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:35:18.490761 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:35:18.837118 ip-10-0-138-149 kubenswrapper[5378]: E1030 21:35:18.837040 5378 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q6mx7" containerName="extractor" Oct 30 21:35:23.161899 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:35:23.161864 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:35:23.805788 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:35:23.805733 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:35:26.934468 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:35:26.934432 5378 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-r9g59"] Oct 30 21:35:28.262957 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:35:28.262921 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:35:28.869917 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:35:28.869876 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:35:32.142256 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:35:32.142225 5378 scope.go:122] "RemoveContainer" containerID="d444ca7e546754214817edb86b5868055cfc1a8cd9a9b2ced5cdd549bc87ecda" Oct 30 21:35:33.053781 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:35:33.053734 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:35:36.261428 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:35:36.261385 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:35:36.389066 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:35:36.389036 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:35:37.882371 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:35:37.882327 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:35:42.033963 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:35:42.033886 5378 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-89af8e-86747f589-h6p2r"] Oct 30 21:35:42.034478 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:35:42.034107 5378 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-89af8e-86747f589-h6p2r" podUID="9a965ff4-aa23-407d-a1b2-1db32b8488a5" containerName="maas-api" containerID="cri-o://bbd02bbc5256293ca23c38764c1649ccfc961d82077ba2d82ccd8a7984970ceb" gracePeriod=30 Oct 30 21:35:42.146046 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:35:42.146014 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:35:43.282307 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:35:43.282274 5378 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-89af8e-86747f589-h6p2r" Oct 30 21:35:43.380655 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:35:43.380628 5378 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9a965ff4-aa23-407d-a1b2-1db32b8488a5-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9a965ff4-aa23-407d-a1b2-1db32b8488a5-openshift-service-ca\") pod \"9a965ff4-aa23-407d-a1b2-1db32b8488a5\" (UID: \"9a965ff4-aa23-407d-a1b2-1db32b8488a5\") " Oct 30 21:35:43.380820 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:35:43.380676 5378 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9a965ff4-aa23-407d-a1b2-1db32b8488a5-metrics-certs\" (UniqueName: \"kubernetes.io/secret/9a965ff4-aa23-407d-a1b2-1db32b8488a5-metrics-certs\") pod \"9a965ff4-aa23-407d-a1b2-1db32b8488a5\" (UID: \"9a965ff4-aa23-407d-a1b2-1db32b8488a5\") " Oct 30 21:35:43.380820 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:35:43.380727 5378 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9a965ff4-aa23-407d-a1b2-1db32b8488a5-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9a965ff4-aa23-407d-a1b2-1db32b8488a5-maas-api-tls\") pod \"9a965ff4-aa23-407d-a1b2-1db32b8488a5\" (UID: \"9a965ff4-aa23-407d-a1b2-1db32b8488a5\") " Oct 30 21:35:43.380820 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:35:43.380750 5378 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9a965ff4-aa23-407d-a1b2-1db32b8488a5-kube-api-access-5jkhr\" (UniqueName: \"kubernetes.io/projected/9a965ff4-aa23-407d-a1b2-1db32b8488a5-kube-api-access-5jkhr\") pod \"9a965ff4-aa23-407d-a1b2-1db32b8488a5\" (UID: \"9a965ff4-aa23-407d-a1b2-1db32b8488a5\") " Oct 30 21:35:43.381057 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:35:43.381019 5378 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9a965ff4-aa23-407d-a1b2-1db32b8488a5-openshift-service-ca" pod "9a965ff4-aa23-407d-a1b2-1db32b8488a5" (UID: "9a965ff4-aa23-407d-a1b2-1db32b8488a5"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:35:43.382885 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:35:43.382859 5378 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9a965ff4-aa23-407d-a1b2-1db32b8488a5-metrics-certs" pod "9a965ff4-aa23-407d-a1b2-1db32b8488a5" (UID: "9a965ff4-aa23-407d-a1b2-1db32b8488a5"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:35:43.383011 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:35:43.382983 5378 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9a965ff4-aa23-407d-a1b2-1db32b8488a5-maas-api-tls" pod "9a965ff4-aa23-407d-a1b2-1db32b8488a5" (UID: "9a965ff4-aa23-407d-a1b2-1db32b8488a5"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:35:43.383071 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:35:43.382989 5378 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9a965ff4-aa23-407d-a1b2-1db32b8488a5-kube-api-access-5jkhr" pod "9a965ff4-aa23-407d-a1b2-1db32b8488a5" (UID: "9a965ff4-aa23-407d-a1b2-1db32b8488a5"). InnerVolumeSpecName "kube-api-access-5jkhr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:35:43.469776 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:35:43.469744 5378 generic.go:356] "Generic (PLEG): container finished" podID="9a965ff4-aa23-407d-a1b2-1db32b8488a5" containerID="bbd02bbc5256293ca23c38764c1649ccfc961d82077ba2d82ccd8a7984970ceb" exitCode=0 Oct 30 21:35:43.469957 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:35:43.469787 5378 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-89af8e-86747f589-h6p2r" event={"ID":"9a965ff4-aa23-407d-a1b2-1db32b8488a5","Type":"ContainerDied","Data":"bbd02bbc5256293ca23c38764c1649ccfc961d82077ba2d82ccd8a7984970ceb"} Oct 30 21:35:43.469957 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:35:43.469802 5378 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-89af8e-86747f589-h6p2r" Oct 30 21:35:43.469957 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:35:43.469826 5378 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-89af8e-86747f589-h6p2r" event={"ID":"9a965ff4-aa23-407d-a1b2-1db32b8488a5","Type":"ContainerDied","Data":"7788faabd4693bc131444c3b6afe68994ed8d017d7b98fbf9360fcd26620d7cb"} Oct 30 21:35:43.469957 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:35:43.469853 5378 scope.go:122] "RemoveContainer" containerID="bbd02bbc5256293ca23c38764c1649ccfc961d82077ba2d82ccd8a7984970ceb" Oct 30 21:35:43.479472 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:35:43.479456 5378 scope.go:122] "RemoveContainer" containerID="bbd02bbc5256293ca23c38764c1649ccfc961d82077ba2d82ccd8a7984970ceb" Oct 30 21:35:43.479744 ip-10-0-138-149 kubenswrapper[5378]: E1030 21:35:43.479725 5378 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bbd02bbc5256293ca23c38764c1649ccfc961d82077ba2d82ccd8a7984970ceb\": container with ID starting with bbd02bbc5256293ca23c38764c1649ccfc961d82077ba2d82ccd8a7984970ceb not found: ID does not exist" containerID="bbd02bbc5256293ca23c38764c1649ccfc961d82077ba2d82ccd8a7984970ceb" Oct 30 21:35:43.479791 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:35:43.479754 5378 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bbd02bbc5256293ca23c38764c1649ccfc961d82077ba2d82ccd8a7984970ceb"} err="failed to get container status \"bbd02bbc5256293ca23c38764c1649ccfc961d82077ba2d82ccd8a7984970ceb\": rpc error: code = NotFound desc = could not find container \"bbd02bbc5256293ca23c38764c1649ccfc961d82077ba2d82ccd8a7984970ceb\": container with ID starting with bbd02bbc5256293ca23c38764c1649ccfc961d82077ba2d82ccd8a7984970ceb not found: ID does not exist" Oct 30 21:35:43.482242 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:35:43.482223 5378 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9a965ff4-aa23-407d-a1b2-1db32b8488a5-openshift-service-ca\") on node \"ip-10-0-138-149.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:35:43.482296 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:35:43.482246 5378 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9a965ff4-aa23-407d-a1b2-1db32b8488a5-metrics-certs\") on node \"ip-10-0-138-149.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:35:43.482296 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:35:43.482256 5378 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9a965ff4-aa23-407d-a1b2-1db32b8488a5-maas-api-tls\") on node \"ip-10-0-138-149.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:35:43.482296 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:35:43.482265 5378 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5jkhr\" (UniqueName: \"kubernetes.io/projected/9a965ff4-aa23-407d-a1b2-1db32b8488a5-kube-api-access-5jkhr\") on node \"ip-10-0-138-149.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:35:43.634112 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:35:43.634033 5378 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-89af8e-86747f589-h6p2r"] Oct 30 21:35:43.637791 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:35:43.637765 5378 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-89af8e-86747f589-h6p2r"] Oct 30 21:35:43.654617 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:35:43.654590 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:35:43.740268 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:35:43.740240 5378 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9a965ff4-aa23-407d-a1b2-1db32b8488a5" path="/var/lib/kubelet/pods/9a965ff4-aa23-407d-a1b2-1db32b8488a5/volumes" Oct 30 21:35:43.924123 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:35:43.924046 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:35:44.731746 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:35:44.731708 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:35:46.685410 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:35:46.685381 5378 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-r9g59"] Oct 30 21:35:48.835957 ip-10-0-138-149 kubenswrapper[5378]: E1030 21:35:48.835923 5378 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q6mx7" containerName="extractor" Oct 30 21:35:50.432313 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:35:50.432275 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:35:50.572409 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:35:50.572380 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:35:50.793394 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:35:50.793288 5378 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-single-main-89af8e-openshift-default-6cc78f548fcl4bq"] Oct 30 21:35:50.793546 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:35:50.793510 5378 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-single-main-89af8e-openshift-default-6cc78f548fcl4bq" podUID="67603b6b-0867-4ec2-b55f-1ed68f3b380e" containerName="istio-proxy" containerID="cri-o://d86ab3c2b830c49e152500654004c8b3afe7960a659a21d98b1af0fb5c0b0cde" gracePeriod=30 Oct 30 21:35:50.794998 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:35:50.794968 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:35:55.566656 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:35:55.566622 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:35:56.050023 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:35:56.049993 5378 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-single-main-89af8e-openshift-default-6cc78f548fcl4bq" Oct 30 21:35:56.112038 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:35:56.112014 5378 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/67603b6b-0867-4ec2-b55f-1ed68f3b380e-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/67603b6b-0867-4ec2-b55f-1ed68f3b380e-workload-socket\") pod \"67603b6b-0867-4ec2-b55f-1ed68f3b380e\" (UID: \"67603b6b-0867-4ec2-b55f-1ed68f3b380e\") " Oct 30 21:35:56.112167 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:35:56.112046 5378 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/67603b6b-0867-4ec2-b55f-1ed68f3b380e-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/67603b6b-0867-4ec2-b55f-1ed68f3b380e-workload-certs\") pod \"67603b6b-0867-4ec2-b55f-1ed68f3b380e\" (UID: \"67603b6b-0867-4ec2-b55f-1ed68f3b380e\") " Oct 30 21:35:56.112167 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:35:56.112087 5378 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/67603b6b-0867-4ec2-b55f-1ed68f3b380e-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/67603b6b-0867-4ec2-b55f-1ed68f3b380e-istio-envoy\") pod \"67603b6b-0867-4ec2-b55f-1ed68f3b380e\" (UID: \"67603b6b-0867-4ec2-b55f-1ed68f3b380e\") " Oct 30 21:35:56.112167 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:35:56.112108 5378 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/67603b6b-0867-4ec2-b55f-1ed68f3b380e-kube-api-access-vsf6k\" (UniqueName: \"kubernetes.io/projected/67603b6b-0867-4ec2-b55f-1ed68f3b380e-kube-api-access-vsf6k\") pod \"67603b6b-0867-4ec2-b55f-1ed68f3b380e\" (UID: \"67603b6b-0867-4ec2-b55f-1ed68f3b380e\") " Oct 30 21:35:56.112167 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:35:56.112153 5378 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/67603b6b-0867-4ec2-b55f-1ed68f3b380e-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/67603b6b-0867-4ec2-b55f-1ed68f3b380e-istiod-ca-cert\") pod \"67603b6b-0867-4ec2-b55f-1ed68f3b380e\" (UID: \"67603b6b-0867-4ec2-b55f-1ed68f3b380e\") " Oct 30 21:35:56.112370 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:35:56.112197 5378 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/67603b6b-0867-4ec2-b55f-1ed68f3b380e-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/67603b6b-0867-4ec2-b55f-1ed68f3b380e-istio-data\") pod \"67603b6b-0867-4ec2-b55f-1ed68f3b380e\" (UID: \"67603b6b-0867-4ec2-b55f-1ed68f3b380e\") " Oct 30 21:35:56.112370 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:35:56.112233 5378 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/67603b6b-0867-4ec2-b55f-1ed68f3b380e-istio-token\" (UniqueName: \"kubernetes.io/projected/67603b6b-0867-4ec2-b55f-1ed68f3b380e-istio-token\") pod \"67603b6b-0867-4ec2-b55f-1ed68f3b380e\" (UID: \"67603b6b-0867-4ec2-b55f-1ed68f3b380e\") " Oct 30 21:35:56.112370 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:35:56.112276 5378 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/67603b6b-0867-4ec2-b55f-1ed68f3b380e-workload-socket" pod "67603b6b-0867-4ec2-b55f-1ed68f3b380e" (UID: "67603b6b-0867-4ec2-b55f-1ed68f3b380e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:35:56.112370 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:35:56.112313 5378 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/67603b6b-0867-4ec2-b55f-1ed68f3b380e-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/67603b6b-0867-4ec2-b55f-1ed68f3b380e-istio-podinfo\") pod \"67603b6b-0867-4ec2-b55f-1ed68f3b380e\" (UID: \"67603b6b-0867-4ec2-b55f-1ed68f3b380e\") " Oct 30 21:35:56.112610 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:35:56.112384 5378 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/67603b6b-0867-4ec2-b55f-1ed68f3b380e-workload-certs" pod "67603b6b-0867-4ec2-b55f-1ed68f3b380e" (UID: "67603b6b-0867-4ec2-b55f-1ed68f3b380e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:35:56.112610 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:35:56.112451 5378 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/67603b6b-0867-4ec2-b55f-1ed68f3b380e-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/67603b6b-0867-4ec2-b55f-1ed68f3b380e-credential-socket\") pod \"67603b6b-0867-4ec2-b55f-1ed68f3b380e\" (UID: \"67603b6b-0867-4ec2-b55f-1ed68f3b380e\") " Oct 30 21:35:56.112610 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:35:56.112536 5378 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/67603b6b-0867-4ec2-b55f-1ed68f3b380e-istio-data" pod "67603b6b-0867-4ec2-b55f-1ed68f3b380e" (UID: "67603b6b-0867-4ec2-b55f-1ed68f3b380e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:35:56.112799 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:35:56.112608 5378 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/67603b6b-0867-4ec2-b55f-1ed68f3b380e-istiod-ca-cert" pod "67603b6b-0867-4ec2-b55f-1ed68f3b380e" (UID: "67603b6b-0867-4ec2-b55f-1ed68f3b380e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:35:56.112860 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:35:56.112808 5378 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/67603b6b-0867-4ec2-b55f-1ed68f3b380e-credential-socket" pod "67603b6b-0867-4ec2-b55f-1ed68f3b380e" (UID: "67603b6b-0867-4ec2-b55f-1ed68f3b380e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:35:56.112907 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:35:56.112882 5378 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/67603b6b-0867-4ec2-b55f-1ed68f3b380e-istio-data\") on node \"ip-10-0-138-149.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:35:56.112961 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:35:56.112906 5378 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/67603b6b-0867-4ec2-b55f-1ed68f3b380e-workload-socket\") on node \"ip-10-0-138-149.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:35:56.112961 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:35:56.112921 5378 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/67603b6b-0867-4ec2-b55f-1ed68f3b380e-workload-certs\") on node \"ip-10-0-138-149.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:35:56.112961 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:35:56.112934 5378 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/67603b6b-0867-4ec2-b55f-1ed68f3b380e-istiod-ca-cert\") on node \"ip-10-0-138-149.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:35:56.114377 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:35:56.114355 5378 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/67603b6b-0867-4ec2-b55f-1ed68f3b380e-istio-envoy" pod "67603b6b-0867-4ec2-b55f-1ed68f3b380e" (UID: "67603b6b-0867-4ec2-b55f-1ed68f3b380e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:35:56.114449 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:35:56.114385 5378 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/67603b6b-0867-4ec2-b55f-1ed68f3b380e-kube-api-access-vsf6k" pod "67603b6b-0867-4ec2-b55f-1ed68f3b380e" (UID: "67603b6b-0867-4ec2-b55f-1ed68f3b380e"). InnerVolumeSpecName "kube-api-access-vsf6k". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:35:56.114719 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:35:56.114694 5378 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/67603b6b-0867-4ec2-b55f-1ed68f3b380e-istio-podinfo" pod "67603b6b-0867-4ec2-b55f-1ed68f3b380e" (UID: "67603b6b-0867-4ec2-b55f-1ed68f3b380e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 21:35:56.114814 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:35:56.114701 5378 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/67603b6b-0867-4ec2-b55f-1ed68f3b380e-istio-token" pod "67603b6b-0867-4ec2-b55f-1ed68f3b380e" (UID: "67603b6b-0867-4ec2-b55f-1ed68f3b380e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:35:56.213759 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:35:56.213730 5378 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/67603b6b-0867-4ec2-b55f-1ed68f3b380e-istio-envoy\") on node \"ip-10-0-138-149.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:35:56.213759 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:35:56.213756 5378 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vsf6k\" (UniqueName: \"kubernetes.io/projected/67603b6b-0867-4ec2-b55f-1ed68f3b380e-kube-api-access-vsf6k\") on node \"ip-10-0-138-149.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:35:56.213952 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:35:56.213767 5378 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/67603b6b-0867-4ec2-b55f-1ed68f3b380e-istio-token\") on node \"ip-10-0-138-149.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:35:56.213952 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:35:56.213777 5378 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/67603b6b-0867-4ec2-b55f-1ed68f3b380e-istio-podinfo\") on node \"ip-10-0-138-149.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:35:56.213952 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:35:56.213786 5378 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/67603b6b-0867-4ec2-b55f-1ed68f3b380e-credential-socket\") on node \"ip-10-0-138-149.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:35:56.530282 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:35:56.530191 5378 generic.go:356] "Generic (PLEG): container finished" podID="67603b6b-0867-4ec2-b55f-1ed68f3b380e" containerID="d86ab3c2b830c49e152500654004c8b3afe7960a659a21d98b1af0fb5c0b0cde" exitCode=0 Oct 30 21:35:56.530282 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:35:56.530260 5378 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-single-main-89af8e-openshift-default-6cc78f548fcl4bq" Oct 30 21:35:56.530282 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:35:56.530269 5378 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-89af8e-openshift-default-6cc78f548fcl4bq" event={"ID":"67603b6b-0867-4ec2-b55f-1ed68f3b380e","Type":"ContainerDied","Data":"d86ab3c2b830c49e152500654004c8b3afe7960a659a21d98b1af0fb5c0b0cde"} Oct 30 21:35:56.530613 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:35:56.530323 5378 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-89af8e-openshift-default-6cc78f548fcl4bq" event={"ID":"67603b6b-0867-4ec2-b55f-1ed68f3b380e","Type":"ContainerDied","Data":"c73dba65eb877ba63c889ba399b32a18969ae436a017697d922856aa221aa89c"} Oct 30 21:35:56.530613 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:35:56.530352 5378 scope.go:122] "RemoveContainer" containerID="d86ab3c2b830c49e152500654004c8b3afe7960a659a21d98b1af0fb5c0b0cde" Oct 30 21:35:56.539797 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:35:56.539780 5378 scope.go:122] "RemoveContainer" containerID="375e56550afb57d3d34656bf8bb7b96819fe0ff7d3b0e97fc1557259d11ff9ef" Oct 30 21:35:56.547932 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:35:56.547916 5378 scope.go:122] "RemoveContainer" containerID="d86ab3c2b830c49e152500654004c8b3afe7960a659a21d98b1af0fb5c0b0cde" Oct 30 21:35:56.548216 ip-10-0-138-149 kubenswrapper[5378]: E1030 21:35:56.548198 5378 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d86ab3c2b830c49e152500654004c8b3afe7960a659a21d98b1af0fb5c0b0cde\": container with ID starting with d86ab3c2b830c49e152500654004c8b3afe7960a659a21d98b1af0fb5c0b0cde not found: ID does not exist" containerID="d86ab3c2b830c49e152500654004c8b3afe7960a659a21d98b1af0fb5c0b0cde" Oct 30 21:35:56.548268 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:35:56.548226 5378 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d86ab3c2b830c49e152500654004c8b3afe7960a659a21d98b1af0fb5c0b0cde"} err="failed to get container status \"d86ab3c2b830c49e152500654004c8b3afe7960a659a21d98b1af0fb5c0b0cde\": rpc error: code = NotFound desc = could not find container \"d86ab3c2b830c49e152500654004c8b3afe7960a659a21d98b1af0fb5c0b0cde\": container with ID starting with d86ab3c2b830c49e152500654004c8b3afe7960a659a21d98b1af0fb5c0b0cde not found: ID does not exist" Oct 30 21:35:56.548268 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:35:56.548243 5378 scope.go:122] "RemoveContainer" containerID="375e56550afb57d3d34656bf8bb7b96819fe0ff7d3b0e97fc1557259d11ff9ef" Oct 30 21:35:56.548475 ip-10-0-138-149 kubenswrapper[5378]: E1030 21:35:56.548458 5378 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"375e56550afb57d3d34656bf8bb7b96819fe0ff7d3b0e97fc1557259d11ff9ef\": container with ID starting with 375e56550afb57d3d34656bf8bb7b96819fe0ff7d3b0e97fc1557259d11ff9ef not found: ID does not exist" containerID="375e56550afb57d3d34656bf8bb7b96819fe0ff7d3b0e97fc1557259d11ff9ef" Oct 30 21:35:56.548534 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:35:56.548485 5378 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"375e56550afb57d3d34656bf8bb7b96819fe0ff7d3b0e97fc1557259d11ff9ef"} err="failed to get container status \"375e56550afb57d3d34656bf8bb7b96819fe0ff7d3b0e97fc1557259d11ff9ef\": rpc error: code = NotFound desc = could not find container \"375e56550afb57d3d34656bf8bb7b96819fe0ff7d3b0e97fc1557259d11ff9ef\": container with ID starting with 375e56550afb57d3d34656bf8bb7b96819fe0ff7d3b0e97fc1557259d11ff9ef not found: ID does not exist" Oct 30 21:35:56.698445 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:35:56.698405 5378 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-single-main-89af8e-openshift-default-6cc78f548fcl4bq"] Oct 30 21:35:56.748759 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:35:56.748731 5378 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-single-main-89af8e-openshift-default-6cc78f548fcl4bq"] Oct 30 21:35:57.739301 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:35:57.739263 5378 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="67603b6b-0867-4ec2-b55f-1ed68f3b380e" path="/var/lib/kubelet/pods/67603b6b-0867-4ec2-b55f-1ed68f3b380e/volumes" Oct 30 21:36:05.340428 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:36:05.340374 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:36:05.914714 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:36:05.914672 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:36:11.407039 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:36:11.407005 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:36:15.046556 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:36:15.046511 5378 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-k97m2"] Oct 30 21:36:15.096785 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:36:15.096754 5378 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-k97m2"] Oct 30 21:36:15.740022 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:36:15.739991 5378 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cbc20c43-c3d2-483c-946c-6e7b601e4bdb" path="/var/lib/kubelet/pods/cbc20c43-c3d2-483c-946c-6e7b601e4bdb/volumes" Oct 30 21:36:18.836531 ip-10-0-138-149 kubenswrapper[5378]: E1030 21:36:18.836495 5378 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q6mx7" containerName="extractor" Oct 30 21:36:48.836432 ip-10-0-138-149 kubenswrapper[5378]: E1030 21:36:48.836391 5378 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q6mx7" containerName="extractor" Oct 30 21:36:55.905123 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:36:55.905090 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:37:18.836204 ip-10-0-138-149 kubenswrapper[5378]: E1030 21:37:18.836162 5378 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q6mx7" containerName="extractor" Oct 30 21:37:32.177809 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:37:32.177779 5378 scope.go:122] "RemoveContainer" containerID="0b3341bdb04d660c4b8d97e3221169b925414ca69e962da934cd12a44d2cea6d" Oct 30 21:37:40.155172 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:37:40.155136 5378 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-r9g59"] Oct 30 21:37:44.746472 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:37:44.746436 5378 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-hww5x"] Oct 30 21:37:44.746909 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:37:44.746841 5378 cpu_manager.go:414] "containerMap: removing container" podUID="67603b6b-0867-4ec2-b55f-1ed68f3b380e" containerName="istio-proxy" Oct 30 21:37:44.746909 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:37:44.746852 5378 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="67603b6b-0867-4ec2-b55f-1ed68f3b380e" containerName="istio-proxy" Oct 30 21:37:44.746909 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:37:44.746875 5378 cpu_manager.go:414] "containerMap: removing container" podUID="67603b6b-0867-4ec2-b55f-1ed68f3b380e" containerName="istio-proxy" Oct 30 21:37:44.746909 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:37:44.746880 5378 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="67603b6b-0867-4ec2-b55f-1ed68f3b380e" containerName="istio-proxy" Oct 30 21:37:44.746909 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:37:44.746889 5378 cpu_manager.go:414] "containerMap: removing container" podUID="9a965ff4-aa23-407d-a1b2-1db32b8488a5" containerName="maas-api" Oct 30 21:37:44.746909 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:37:44.746894 5378 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9a965ff4-aa23-407d-a1b2-1db32b8488a5" containerName="maas-api" Oct 30 21:37:44.747114 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:37:44.746954 5378 memory_manager.go:365] "RemoveStaleState removing state" podUID="67603b6b-0867-4ec2-b55f-1ed68f3b380e" containerName="istio-proxy" Oct 30 21:37:44.747114 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:37:44.746963 5378 memory_manager.go:365] "RemoveStaleState removing state" podUID="9a965ff4-aa23-407d-a1b2-1db32b8488a5" containerName="maas-api" Oct 30 21:37:44.751041 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:37:44.751018 5378 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-hww5x" Oct 30 21:37:44.806288 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:37:44.806253 5378 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-hww5x"] Oct 30 21:37:44.830006 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:37:44.829973 5378 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f1f1058e-3063-4153-b2c0-3da42e19520b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d576e68b-hww5x\" (UID: \"f1f1058e-3063-4153-b2c0-3da42e19520b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-hww5x" Oct 30 21:37:44.830158 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:37:44.830010 5378 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f1f1058e-3063-4153-b2c0-3da42e19520b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d576e68b-hww5x\" (UID: \"f1f1058e-3063-4153-b2c0-3da42e19520b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-hww5x" Oct 30 21:37:44.830249 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:37:44.830223 5378 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bxnj2\" (UniqueName: \"kubernetes.io/projected/f1f1058e-3063-4153-b2c0-3da42e19520b-kube-api-access-bxnj2\") pod \"maas-api-revoke-sub-d576e68b-hww5x\" (UID: \"f1f1058e-3063-4153-b2c0-3da42e19520b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-hww5x" Oct 30 21:37:44.854869 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:37:44.854840 5378 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 21:37:44.854869 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:37:44.854867 5378 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 21:37:44.855019 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:37:44.854872 5378 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 21:37:44.855019 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:37:44.854910 5378 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-q2mgc\"" Oct 30 21:37:44.931010 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:37:44.930976 5378 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bxnj2\" (UniqueName: \"kubernetes.io/projected/f1f1058e-3063-4153-b2c0-3da42e19520b-kube-api-access-bxnj2\") pod \"maas-api-revoke-sub-d576e68b-hww5x\" (UID: \"f1f1058e-3063-4153-b2c0-3da42e19520b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-hww5x" Oct 30 21:37:44.931202 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:37:44.931035 5378 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f1f1058e-3063-4153-b2c0-3da42e19520b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d576e68b-hww5x\" (UID: \"f1f1058e-3063-4153-b2c0-3da42e19520b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-hww5x" Oct 30 21:37:44.931202 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:37:44.931062 5378 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f1f1058e-3063-4153-b2c0-3da42e19520b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d576e68b-hww5x\" (UID: \"f1f1058e-3063-4153-b2c0-3da42e19520b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-hww5x" Oct 30 21:37:44.931726 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:37:44.931698 5378 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f1f1058e-3063-4153-b2c0-3da42e19520b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d576e68b-hww5x\" (UID: \"f1f1058e-3063-4153-b2c0-3da42e19520b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-hww5x" Oct 30 21:37:44.933673 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:37:44.933647 5378 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f1f1058e-3063-4153-b2c0-3da42e19520b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d576e68b-hww5x\" (UID: \"f1f1058e-3063-4153-b2c0-3da42e19520b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-hww5x" Oct 30 21:37:44.988424 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:37:44.988397 5378 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bxnj2\" (UniqueName: \"kubernetes.io/projected/f1f1058e-3063-4153-b2c0-3da42e19520b-kube-api-access-bxnj2\") pod \"maas-api-revoke-sub-d576e68b-hww5x\" (UID: \"f1f1058e-3063-4153-b2c0-3da42e19520b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-hww5x" Oct 30 21:37:45.063167 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:37:45.063095 5378 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-hww5x" Oct 30 21:37:45.430546 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:37:45.430524 5378 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-hww5x"] Oct 30 21:37:45.433025 ip-10-0-138-149 kubenswrapper[5378]: W1030 21:37:45.432996 5378 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf1f1058e_3063_4153_b2c0_3da42e19520b.slice/crio-2acb5ce07f7e64a667d0303b2691167eeed04ea1c4602609c379eac4ba86be48 WatchSource:0}: Error finding container 2acb5ce07f7e64a667d0303b2691167eeed04ea1c4602609c379eac4ba86be48: Status 404 returned error can't find the container with id 2acb5ce07f7e64a667d0303b2691167eeed04ea1c4602609c379eac4ba86be48 Oct 30 21:37:46.058784 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:37:46.058704 5378 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-hww5x" event={"ID":"f1f1058e-3063-4153-b2c0-3da42e19520b","Type":"ContainerStarted","Data":"e7907eb781730fea7c8328361e805d3a8a6a7ea8a6dbf06a98a28a6e4110adb7"} Oct 30 21:37:46.058784 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:37:46.058752 5378 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-hww5x" event={"ID":"f1f1058e-3063-4153-b2c0-3da42e19520b","Type":"ContainerStarted","Data":"2acb5ce07f7e64a667d0303b2691167eeed04ea1c4602609c379eac4ba86be48"} Oct 30 21:37:46.173158 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:37:46.173111 5378 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-hww5x" podStartSLOduration=2.173098518 podStartE2EDuration="2.173098518s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:37:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:37:46.172516695 +0000 UTC m=+3556.600630616" watchObservedRunningTime="2026-10-30 21:37:46.173098518 +0000 UTC m=+3556.601212431" Oct 30 21:37:47.067519 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:37:47.067482 5378 generic.go:356] "Generic (PLEG): container finished" podID="f1f1058e-3063-4153-b2c0-3da42e19520b" containerID="e7907eb781730fea7c8328361e805d3a8a6a7ea8a6dbf06a98a28a6e4110adb7" exitCode=0 Oct 30 21:37:47.067942 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:37:47.067518 5378 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-hww5x" event={"ID":"f1f1058e-3063-4153-b2c0-3da42e19520b","Type":"ContainerDied","Data":"e7907eb781730fea7c8328361e805d3a8a6a7ea8a6dbf06a98a28a6e4110adb7"} Oct 30 21:37:48.207213 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:37:48.207188 5378 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-hww5x" Oct 30 21:37:48.255482 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:37:48.255456 5378 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f1f1058e-3063-4153-b2c0-3da42e19520b-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f1f1058e-3063-4153-b2c0-3da42e19520b-maas-api-xks-ca\") pod \"f1f1058e-3063-4153-b2c0-3da42e19520b\" (UID: \"f1f1058e-3063-4153-b2c0-3da42e19520b\") " Oct 30 21:37:48.255634 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:37:48.255523 5378 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f1f1058e-3063-4153-b2c0-3da42e19520b-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f1f1058e-3063-4153-b2c0-3da42e19520b-maas-api-openshift-ca\") pod \"f1f1058e-3063-4153-b2c0-3da42e19520b\" (UID: \"f1f1058e-3063-4153-b2c0-3da42e19520b\") " Oct 30 21:37:48.255634 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:37:48.255580 5378 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f1f1058e-3063-4153-b2c0-3da42e19520b-kube-api-access-bxnj2\" (UniqueName: \"kubernetes.io/projected/f1f1058e-3063-4153-b2c0-3da42e19520b-kube-api-access-bxnj2\") pod \"f1f1058e-3063-4153-b2c0-3da42e19520b\" (UID: \"f1f1058e-3063-4153-b2c0-3da42e19520b\") " Oct 30 21:37:48.255884 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:37:48.255859 5378 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f1f1058e-3063-4153-b2c0-3da42e19520b-maas-api-openshift-ca" pod "f1f1058e-3063-4153-b2c0-3da42e19520b" (UID: "f1f1058e-3063-4153-b2c0-3da42e19520b"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:37:48.257435 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:37:48.257412 5378 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f1f1058e-3063-4153-b2c0-3da42e19520b-maas-api-xks-ca" pod "f1f1058e-3063-4153-b2c0-3da42e19520b" (UID: "f1f1058e-3063-4153-b2c0-3da42e19520b"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:37:48.257503 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:37:48.257488 5378 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f1f1058e-3063-4153-b2c0-3da42e19520b-kube-api-access-bxnj2" pod "f1f1058e-3063-4153-b2c0-3da42e19520b" (UID: "f1f1058e-3063-4153-b2c0-3da42e19520b"). InnerVolumeSpecName "kube-api-access-bxnj2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:37:48.356935 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:37:48.356904 5378 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f1f1058e-3063-4153-b2c0-3da42e19520b-maas-api-xks-ca\") on node \"ip-10-0-138-149.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:37:48.356935 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:37:48.356932 5378 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f1f1058e-3063-4153-b2c0-3da42e19520b-maas-api-openshift-ca\") on node \"ip-10-0-138-149.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:37:48.357119 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:37:48.356945 5378 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bxnj2\" (UniqueName: \"kubernetes.io/projected/f1f1058e-3063-4153-b2c0-3da42e19520b-kube-api-access-bxnj2\") on node \"ip-10-0-138-149.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:37:48.835978 ip-10-0-138-149 kubenswrapper[5378]: E1030 21:37:48.835893 5378 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q6mx7" containerName="extractor" Oct 30 21:37:49.078015 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:37:49.077988 5378 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-hww5x" Oct 30 21:37:49.078015 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:37:49.078014 5378 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-hww5x" event={"ID":"f1f1058e-3063-4153-b2c0-3da42e19520b","Type":"ContainerDied","Data":"2acb5ce07f7e64a667d0303b2691167eeed04ea1c4602609c379eac4ba86be48"} Oct 30 21:37:49.078227 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:37:49.078061 5378 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2acb5ce07f7e64a667d0303b2691167eeed04ea1c4602609c379eac4ba86be48" Oct 30 21:37:56.238748 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:37:56.238709 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:37:57.578161 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:37:57.578127 5378 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-r9g59"] Oct 30 21:38:05.103590 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:38:05.099761 5378 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-769df7846c-99vbx container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.167:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 21:38:05.103590 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:38:05.099835 5378 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-99vbx" podUID="ccc2fbc1-82a9-46e7-a6a6-102582f572d3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.167:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 21:38:05.103590 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:38:05.100167 5378 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-769df7846c-rszcl container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.169:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 21:38:05.103590 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:38:05.100204 5378 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-rszcl" podUID="946e32af-526e-4bae-855d-8062b58d497e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.169:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 21:38:06.929494 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:38:06.929460 5378 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-r9g59"] Oct 30 21:38:18.840726 ip-10-0-138-149 kubenswrapper[5378]: E1030 21:38:18.840688 5378 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q6mx7" containerName="extractor" Oct 30 21:38:26.044816 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:38:26.044781 5378 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-z588j"] Oct 30 21:38:26.094933 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:38:26.094901 5378 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-z588j"] Oct 30 21:38:26.173394 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:38:26.173360 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:38:26.719437 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:38:26.719405 5378 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-single-main-3f401f-openshift-default-7f4658d54fbgjqn"] Oct 30 21:38:26.719825 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:38:26.719811 5378 cpu_manager.go:414] "containerMap: removing container" podUID="f1f1058e-3063-4153-b2c0-3da42e19520b" containerName="revoke-keys" Oct 30 21:38:26.719883 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:38:26.719826 5378 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f1f1058e-3063-4153-b2c0-3da42e19520b" containerName="revoke-keys" Oct 30 21:38:26.719930 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:38:26.719919 5378 memory_manager.go:365] "RemoveStaleState removing state" podUID="67603b6b-0867-4ec2-b55f-1ed68f3b380e" containerName="istio-proxy" Oct 30 21:38:26.719966 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:38:26.719938 5378 memory_manager.go:365] "RemoveStaleState removing state" podUID="f1f1058e-3063-4153-b2c0-3da42e19520b" containerName="revoke-keys" Oct 30 21:38:26.723144 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:38:26.723128 5378 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-single-main-3f401f-openshift-default-7f4658d54fbgjqn" Oct 30 21:38:26.799863 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:38:26.799825 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:38:26.802435 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:38:26.802406 5378 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-single-main-3f401f-openshift-default-dockercfg-kvhpk\"" Oct 30 21:38:26.828295 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:38:26.828263 5378 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-single-main-3f401f-openshift-default-7f4658d54fbgjqn"] Oct 30 21:38:26.848378 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:38:26.848339 5378 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b6a5bc29-f3e8-4c39-9a74-32e1eb0c3ac3-credential-socket\") pod \"e2e-worker-single-main-3f401f-openshift-default-7f4658d54fbgjqn\" (UID: \"b6a5bc29-f3e8-4c39-9a74-32e1eb0c3ac3\") " pod="openshift-ingress/e2e-worker-single-main-3f401f-openshift-default-7f4658d54fbgjqn" Oct 30 21:38:26.848378 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:38:26.848376 5378 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b6a5bc29-f3e8-4c39-9a74-32e1eb0c3ac3-istio-podinfo\") pod \"e2e-worker-single-main-3f401f-openshift-default-7f4658d54fbgjqn\" (UID: \"b6a5bc29-f3e8-4c39-9a74-32e1eb0c3ac3\") " pod="openshift-ingress/e2e-worker-single-main-3f401f-openshift-default-7f4658d54fbgjqn" Oct 30 21:38:26.848634 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:38:26.848399 5378 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b6a5bc29-f3e8-4c39-9a74-32e1eb0c3ac3-workload-certs\") pod \"e2e-worker-single-main-3f401f-openshift-default-7f4658d54fbgjqn\" (UID: \"b6a5bc29-f3e8-4c39-9a74-32e1eb0c3ac3\") " pod="openshift-ingress/e2e-worker-single-main-3f401f-openshift-default-7f4658d54fbgjqn" Oct 30 21:38:26.848634 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:38:26.848491 5378 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b6a5bc29-f3e8-4c39-9a74-32e1eb0c3ac3-istio-token\") pod \"e2e-worker-single-main-3f401f-openshift-default-7f4658d54fbgjqn\" (UID: \"b6a5bc29-f3e8-4c39-9a74-32e1eb0c3ac3\") " pod="openshift-ingress/e2e-worker-single-main-3f401f-openshift-default-7f4658d54fbgjqn" Oct 30 21:38:26.848634 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:38:26.848584 5378 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b6a5bc29-f3e8-4c39-9a74-32e1eb0c3ac3-istio-data\") pod \"e2e-worker-single-main-3f401f-openshift-default-7f4658d54fbgjqn\" (UID: \"b6a5bc29-f3e8-4c39-9a74-32e1eb0c3ac3\") " pod="openshift-ingress/e2e-worker-single-main-3f401f-openshift-default-7f4658d54fbgjqn" Oct 30 21:38:26.848830 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:38:26.848679 5378 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b6a5bc29-f3e8-4c39-9a74-32e1eb0c3ac3-istiod-ca-cert\") pod \"e2e-worker-single-main-3f401f-openshift-default-7f4658d54fbgjqn\" (UID: \"b6a5bc29-f3e8-4c39-9a74-32e1eb0c3ac3\") " pod="openshift-ingress/e2e-worker-single-main-3f401f-openshift-default-7f4658d54fbgjqn" Oct 30 21:38:26.848830 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:38:26.848752 5378 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bbqm8\" (UniqueName: \"kubernetes.io/projected/b6a5bc29-f3e8-4c39-9a74-32e1eb0c3ac3-kube-api-access-bbqm8\") pod \"e2e-worker-single-main-3f401f-openshift-default-7f4658d54fbgjqn\" (UID: \"b6a5bc29-f3e8-4c39-9a74-32e1eb0c3ac3\") " pod="openshift-ingress/e2e-worker-single-main-3f401f-openshift-default-7f4658d54fbgjqn" Oct 30 21:38:26.848830 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:38:26.848800 5378 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b6a5bc29-f3e8-4c39-9a74-32e1eb0c3ac3-workload-socket\") pod \"e2e-worker-single-main-3f401f-openshift-default-7f4658d54fbgjqn\" (UID: \"b6a5bc29-f3e8-4c39-9a74-32e1eb0c3ac3\") " pod="openshift-ingress/e2e-worker-single-main-3f401f-openshift-default-7f4658d54fbgjqn" Oct 30 21:38:26.849062 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:38:26.848879 5378 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b6a5bc29-f3e8-4c39-9a74-32e1eb0c3ac3-istio-envoy\") pod \"e2e-worker-single-main-3f401f-openshift-default-7f4658d54fbgjqn\" (UID: \"b6a5bc29-f3e8-4c39-9a74-32e1eb0c3ac3\") " pod="openshift-ingress/e2e-worker-single-main-3f401f-openshift-default-7f4658d54fbgjqn" Oct 30 21:38:26.878464 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:38:26.878427 5378 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-r9g59"] Oct 30 21:38:26.949534 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:38:26.949502 5378 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bbqm8\" (UniqueName: \"kubernetes.io/projected/b6a5bc29-f3e8-4c39-9a74-32e1eb0c3ac3-kube-api-access-bbqm8\") pod \"e2e-worker-single-main-3f401f-openshift-default-7f4658d54fbgjqn\" (UID: \"b6a5bc29-f3e8-4c39-9a74-32e1eb0c3ac3\") " pod="openshift-ingress/e2e-worker-single-main-3f401f-openshift-default-7f4658d54fbgjqn" Oct 30 21:38:26.949805 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:38:26.949545 5378 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b6a5bc29-f3e8-4c39-9a74-32e1eb0c3ac3-workload-socket\") pod \"e2e-worker-single-main-3f401f-openshift-default-7f4658d54fbgjqn\" (UID: \"b6a5bc29-f3e8-4c39-9a74-32e1eb0c3ac3\") " pod="openshift-ingress/e2e-worker-single-main-3f401f-openshift-default-7f4658d54fbgjqn" Oct 30 21:38:26.949805 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:38:26.949684 5378 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b6a5bc29-f3e8-4c39-9a74-32e1eb0c3ac3-istio-envoy\") pod \"e2e-worker-single-main-3f401f-openshift-default-7f4658d54fbgjqn\" (UID: \"b6a5bc29-f3e8-4c39-9a74-32e1eb0c3ac3\") " pod="openshift-ingress/e2e-worker-single-main-3f401f-openshift-default-7f4658d54fbgjqn" Oct 30 21:38:26.949934 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:38:26.949804 5378 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b6a5bc29-f3e8-4c39-9a74-32e1eb0c3ac3-credential-socket\") pod \"e2e-worker-single-main-3f401f-openshift-default-7f4658d54fbgjqn\" (UID: \"b6a5bc29-f3e8-4c39-9a74-32e1eb0c3ac3\") " pod="openshift-ingress/e2e-worker-single-main-3f401f-openshift-default-7f4658d54fbgjqn" Oct 30 21:38:26.949934 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:38:26.949849 5378 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b6a5bc29-f3e8-4c39-9a74-32e1eb0c3ac3-istio-podinfo\") pod \"e2e-worker-single-main-3f401f-openshift-default-7f4658d54fbgjqn\" (UID: \"b6a5bc29-f3e8-4c39-9a74-32e1eb0c3ac3\") " pod="openshift-ingress/e2e-worker-single-main-3f401f-openshift-default-7f4658d54fbgjqn" Oct 30 21:38:26.949934 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:38:26.949878 5378 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b6a5bc29-f3e8-4c39-9a74-32e1eb0c3ac3-workload-certs\") pod \"e2e-worker-single-main-3f401f-openshift-default-7f4658d54fbgjqn\" (UID: \"b6a5bc29-f3e8-4c39-9a74-32e1eb0c3ac3\") " pod="openshift-ingress/e2e-worker-single-main-3f401f-openshift-default-7f4658d54fbgjqn" Oct 30 21:38:26.949934 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:38:26.949907 5378 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b6a5bc29-f3e8-4c39-9a74-32e1eb0c3ac3-istio-token\") pod \"e2e-worker-single-main-3f401f-openshift-default-7f4658d54fbgjqn\" (UID: \"b6a5bc29-f3e8-4c39-9a74-32e1eb0c3ac3\") " pod="openshift-ingress/e2e-worker-single-main-3f401f-openshift-default-7f4658d54fbgjqn" Oct 30 21:38:26.949934 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:38:26.949924 5378 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b6a5bc29-f3e8-4c39-9a74-32e1eb0c3ac3-istio-data\") pod \"e2e-worker-single-main-3f401f-openshift-default-7f4658d54fbgjqn\" (UID: \"b6a5bc29-f3e8-4c39-9a74-32e1eb0c3ac3\") " pod="openshift-ingress/e2e-worker-single-main-3f401f-openshift-default-7f4658d54fbgjqn" Oct 30 21:38:26.950191 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:38:26.949947 5378 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b6a5bc29-f3e8-4c39-9a74-32e1eb0c3ac3-workload-socket\") pod \"e2e-worker-single-main-3f401f-openshift-default-7f4658d54fbgjqn\" (UID: \"b6a5bc29-f3e8-4c39-9a74-32e1eb0c3ac3\") " pod="openshift-ingress/e2e-worker-single-main-3f401f-openshift-default-7f4658d54fbgjqn" Oct 30 21:38:26.950191 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:38:26.949972 5378 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b6a5bc29-f3e8-4c39-9a74-32e1eb0c3ac3-istiod-ca-cert\") pod \"e2e-worker-single-main-3f401f-openshift-default-7f4658d54fbgjqn\" (UID: \"b6a5bc29-f3e8-4c39-9a74-32e1eb0c3ac3\") " pod="openshift-ingress/e2e-worker-single-main-3f401f-openshift-default-7f4658d54fbgjqn" Oct 30 21:38:26.950312 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:38:26.950219 5378 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b6a5bc29-f3e8-4c39-9a74-32e1eb0c3ac3-credential-socket\") pod \"e2e-worker-single-main-3f401f-openshift-default-7f4658d54fbgjqn\" (UID: \"b6a5bc29-f3e8-4c39-9a74-32e1eb0c3ac3\") " pod="openshift-ingress/e2e-worker-single-main-3f401f-openshift-default-7f4658d54fbgjqn" Oct 30 21:38:26.950312 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:38:26.950292 5378 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b6a5bc29-f3e8-4c39-9a74-32e1eb0c3ac3-workload-certs\") pod \"e2e-worker-single-main-3f401f-openshift-default-7f4658d54fbgjqn\" (UID: \"b6a5bc29-f3e8-4c39-9a74-32e1eb0c3ac3\") " pod="openshift-ingress/e2e-worker-single-main-3f401f-openshift-default-7f4658d54fbgjqn" Oct 30 21:38:26.950466 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:38:26.950415 5378 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b6a5bc29-f3e8-4c39-9a74-32e1eb0c3ac3-istio-data\") pod \"e2e-worker-single-main-3f401f-openshift-default-7f4658d54fbgjqn\" (UID: \"b6a5bc29-f3e8-4c39-9a74-32e1eb0c3ac3\") " pod="openshift-ingress/e2e-worker-single-main-3f401f-openshift-default-7f4658d54fbgjqn" Oct 30 21:38:26.950767 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:38:26.950750 5378 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b6a5bc29-f3e8-4c39-9a74-32e1eb0c3ac3-istiod-ca-cert\") pod \"e2e-worker-single-main-3f401f-openshift-default-7f4658d54fbgjqn\" (UID: \"b6a5bc29-f3e8-4c39-9a74-32e1eb0c3ac3\") " pod="openshift-ingress/e2e-worker-single-main-3f401f-openshift-default-7f4658d54fbgjqn" Oct 30 21:38:26.952519 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:38:26.952490 5378 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b6a5bc29-f3e8-4c39-9a74-32e1eb0c3ac3-istio-envoy\") pod \"e2e-worker-single-main-3f401f-openshift-default-7f4658d54fbgjqn\" (UID: \"b6a5bc29-f3e8-4c39-9a74-32e1eb0c3ac3\") " pod="openshift-ingress/e2e-worker-single-main-3f401f-openshift-default-7f4658d54fbgjqn" Oct 30 21:38:26.952662 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:38:26.952634 5378 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b6a5bc29-f3e8-4c39-9a74-32e1eb0c3ac3-istio-podinfo\") pod \"e2e-worker-single-main-3f401f-openshift-default-7f4658d54fbgjqn\" (UID: \"b6a5bc29-f3e8-4c39-9a74-32e1eb0c3ac3\") " pod="openshift-ingress/e2e-worker-single-main-3f401f-openshift-default-7f4658d54fbgjqn" Oct 30 21:38:27.007436 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:38:27.007365 5378 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bbqm8\" (UniqueName: \"kubernetes.io/projected/b6a5bc29-f3e8-4c39-9a74-32e1eb0c3ac3-kube-api-access-bbqm8\") pod \"e2e-worker-single-main-3f401f-openshift-default-7f4658d54fbgjqn\" (UID: \"b6a5bc29-f3e8-4c39-9a74-32e1eb0c3ac3\") " pod="openshift-ingress/e2e-worker-single-main-3f401f-openshift-default-7f4658d54fbgjqn" Oct 30 21:38:27.014042 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:38:27.013982 5378 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b6a5bc29-f3e8-4c39-9a74-32e1eb0c3ac3-istio-token\") pod \"e2e-worker-single-main-3f401f-openshift-default-7f4658d54fbgjqn\" (UID: \"b6a5bc29-f3e8-4c39-9a74-32e1eb0c3ac3\") " pod="openshift-ingress/e2e-worker-single-main-3f401f-openshift-default-7f4658d54fbgjqn" Oct 30 21:38:27.022121 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:38:27.021784 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:38:27.038788 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:38:27.038743 5378 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-single-main-3f401f-openshift-default-7f4658d54fbgjqn" Oct 30 21:38:27.465025 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:38:27.465001 5378 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-single-main-3f401f-openshift-default-7f4658d54fbgjqn"] Oct 30 21:38:27.466946 ip-10-0-138-149 kubenswrapper[5378]: W1030 21:38:27.466921 5378 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb6a5bc29_f3e8_4c39_9a74_32e1eb0c3ac3.slice/crio-0ff97a7c2c5f9a3e7ddad277dd8f4f2378a099fe4deef3822abf42c68d7d141d WatchSource:0}: Error finding container 0ff97a7c2c5f9a3e7ddad277dd8f4f2378a099fe4deef3822abf42c68d7d141d: Status 404 returned error can't find the container with id 0ff97a7c2c5f9a3e7ddad277dd8f4f2378a099fe4deef3822abf42c68d7d141d Oct 30 21:38:27.469027 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:38:27.468990 5378 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 21:38:27.469136 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:38:27.469069 5378 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 21:38:27.469136 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:38:27.469110 5378 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 21:38:27.739581 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:38:27.739494 5378 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fcfaf80d-b3f8-4cb2-9f58-f1ce1e83b7ae" path="/var/lib/kubelet/pods/fcfaf80d-b3f8-4cb2-9f58-f1ce1e83b7ae/volumes" Oct 30 21:38:28.279880 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:38:28.279845 5378 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-3f401f-openshift-default-7f4658d54fbgjqn" event={"ID":"b6a5bc29-f3e8-4c39-9a74-32e1eb0c3ac3","Type":"ContainerStarted","Data":"9138a9a3cebea4758795e880f39cd8536a5d8f252b97dfd2716927bba0e0427e"} Oct 30 21:38:28.280064 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:38:28.279930 5378 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-3f401f-openshift-default-7f4658d54fbgjqn" event={"ID":"b6a5bc29-f3e8-4c39-9a74-32e1eb0c3ac3","Type":"ContainerStarted","Data":"0ff97a7c2c5f9a3e7ddad277dd8f4f2378a099fe4deef3822abf42c68d7d141d"} Oct 30 21:38:28.398862 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:38:28.398811 5378 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-single-main-3f401f-openshift-default-7f4658d54fbgjqn" podStartSLOduration=2.398796908 podStartE2EDuration="2.398796908s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:38:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:38:28.396937724 +0000 UTC m=+3598.825051645" watchObservedRunningTime="2026-10-30 21:38:28.398796908 +0000 UTC m=+3598.826910901" Oct 30 21:38:29.039472 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:38:29.039435 5378 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-single-main-3f401f-openshift-default-7f4658d54fbgjqn" Oct 30 21:38:29.044605 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:38:29.044580 5378 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-single-main-3f401f-openshift-default-7f4658d54fbgjqn" Oct 30 21:38:29.285129 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:38:29.285100 5378 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-single-main-3f401f-openshift-default-7f4658d54fbgjqn" Oct 30 21:38:29.286135 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:38:29.286115 5378 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-single-main-3f401f-openshift-default-7f4658d54fbgjqn" Oct 30 21:38:30.230966 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:38:30.230938 5378 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-7x9jg_c2ff359c-cf92-4a5b-8610-f6eb41acbe73/ovn-acl-logging/0.log" Oct 30 21:38:30.243586 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:38:30.243547 5378 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-7x9jg_c2ff359c-cf92-4a5b-8610-f6eb41acbe73/ovn-acl-logging/0.log" Oct 30 21:38:31.294052 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:38:31.294018 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:38:31.462078 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:38:31.462043 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:38:31.594253 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:38:31.594175 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:38:34.538898 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:38:34.538866 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:38:34.709378 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:38:34.709349 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:38:37.443211 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:38:37.443178 5378 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-3f401f-6c698d8476-s2qdv"] Oct 30 21:38:37.446749 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:38:37.446730 5378 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-3f401f-6c698d8476-s2qdv" Oct 30 21:38:37.543727 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:38:37.543693 5378 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lc7lv\" (UniqueName: \"kubernetes.io/projected/a19a2e84-eece-4b0e-904c-cda9fbc08cb2-kube-api-access-lc7lv\") pod \"maas-api-e2e-worker-single-main-3f401f-6c698d8476-s2qdv\" (UID: \"a19a2e84-eece-4b0e-904c-cda9fbc08cb2\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-3f401f-6c698d8476-s2qdv" Oct 30 21:38:37.543891 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:38:37.543797 5378 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a19a2e84-eece-4b0e-904c-cda9fbc08cb2-openshift-service-ca\") pod \"maas-api-e2e-worker-single-main-3f401f-6c698d8476-s2qdv\" (UID: \"a19a2e84-eece-4b0e-904c-cda9fbc08cb2\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-3f401f-6c698d8476-s2qdv" Oct 30 21:38:37.543891 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:38:37.543832 5378 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a19a2e84-eece-4b0e-904c-cda9fbc08cb2-metrics-certs\") pod \"maas-api-e2e-worker-single-main-3f401f-6c698d8476-s2qdv\" (UID: \"a19a2e84-eece-4b0e-904c-cda9fbc08cb2\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-3f401f-6c698d8476-s2qdv" Oct 30 21:38:37.543974 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:38:37.543952 5378 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a19a2e84-eece-4b0e-904c-cda9fbc08cb2-maas-api-tls\") pod \"maas-api-e2e-worker-single-main-3f401f-6c698d8476-s2qdv\" (UID: \"a19a2e84-eece-4b0e-904c-cda9fbc08cb2\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-3f401f-6c698d8476-s2qdv" Oct 30 21:38:37.556733 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:38:37.556701 5378 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\"" Oct 30 21:38:37.556733 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:38:37.556704 5378 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 21:38:37.556894 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:38:37.556735 5378 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-6mqfn\"" Oct 30 21:38:37.556894 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:38:37.556739 5378 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-single-main-3f401f-serving-cert\"" Oct 30 21:38:37.556894 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:38:37.556725 5378 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 21:38:37.558269 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:38:37.558251 5378 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-3f401f-6c698d8476-s2qdv"] Oct 30 21:38:37.623953 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:38:37.623924 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:38:37.644882 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:38:37.644854 5378 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a19a2e84-eece-4b0e-904c-cda9fbc08cb2-openshift-service-ca\") pod \"maas-api-e2e-worker-single-main-3f401f-6c698d8476-s2qdv\" (UID: \"a19a2e84-eece-4b0e-904c-cda9fbc08cb2\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-3f401f-6c698d8476-s2qdv" Oct 30 21:38:37.645041 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:38:37.644889 5378 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a19a2e84-eece-4b0e-904c-cda9fbc08cb2-metrics-certs\") pod \"maas-api-e2e-worker-single-main-3f401f-6c698d8476-s2qdv\" (UID: \"a19a2e84-eece-4b0e-904c-cda9fbc08cb2\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-3f401f-6c698d8476-s2qdv" Oct 30 21:38:37.645041 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:38:37.644969 5378 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a19a2e84-eece-4b0e-904c-cda9fbc08cb2-maas-api-tls\") pod \"maas-api-e2e-worker-single-main-3f401f-6c698d8476-s2qdv\" (UID: \"a19a2e84-eece-4b0e-904c-cda9fbc08cb2\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-3f401f-6c698d8476-s2qdv" Oct 30 21:38:37.645169 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:38:37.645069 5378 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lc7lv\" (UniqueName: \"kubernetes.io/projected/a19a2e84-eece-4b0e-904c-cda9fbc08cb2-kube-api-access-lc7lv\") pod \"maas-api-e2e-worker-single-main-3f401f-6c698d8476-s2qdv\" (UID: \"a19a2e84-eece-4b0e-904c-cda9fbc08cb2\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-3f401f-6c698d8476-s2qdv" Oct 30 21:38:37.645169 ip-10-0-138-149 kubenswrapper[5378]: E1030 21:38:37.645127 5378 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-worker-single-main-3f401f-serving-cert: secret "maas-api-e2e-worker-single-main-3f401f-serving-cert" not found Oct 30 21:38:37.645251 ip-10-0-138-149 kubenswrapper[5378]: E1030 21:38:37.645216 5378 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a19a2e84-eece-4b0e-904c-cda9fbc08cb2-maas-api-tls podName:a19a2e84-eece-4b0e-904c-cda9fbc08cb2 nodeName:}" failed. No retries permitted until 2026-10-30 21:38:38.14519462 +0000 UTC m=+3608.573308518 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/a19a2e84-eece-4b0e-904c-cda9fbc08cb2-maas-api-tls") pod "maas-api-e2e-worker-single-main-3f401f-6c698d8476-s2qdv" (UID: "a19a2e84-eece-4b0e-904c-cda9fbc08cb2") : secret "maas-api-e2e-worker-single-main-3f401f-serving-cert" not found Oct 30 21:38:37.645651 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:38:37.645622 5378 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a19a2e84-eece-4b0e-904c-cda9fbc08cb2-openshift-service-ca\") pod \"maas-api-e2e-worker-single-main-3f401f-6c698d8476-s2qdv\" (UID: \"a19a2e84-eece-4b0e-904c-cda9fbc08cb2\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-3f401f-6c698d8476-s2qdv" Oct 30 21:38:37.647267 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:38:37.647245 5378 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a19a2e84-eece-4b0e-904c-cda9fbc08cb2-metrics-certs\") pod \"maas-api-e2e-worker-single-main-3f401f-6c698d8476-s2qdv\" (UID: \"a19a2e84-eece-4b0e-904c-cda9fbc08cb2\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-3f401f-6c698d8476-s2qdv" Oct 30 21:38:37.702922 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:38:37.702867 5378 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lc7lv\" (UniqueName: \"kubernetes.io/projected/a19a2e84-eece-4b0e-904c-cda9fbc08cb2-kube-api-access-lc7lv\") pod \"maas-api-e2e-worker-single-main-3f401f-6c698d8476-s2qdv\" (UID: \"a19a2e84-eece-4b0e-904c-cda9fbc08cb2\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-3f401f-6c698d8476-s2qdv" Oct 30 21:38:37.782044 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:38:37.782007 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:38:37.899030 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:38:37.898990 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:38:38.154196 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:38:38.154159 5378 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a19a2e84-eece-4b0e-904c-cda9fbc08cb2-maas-api-tls\") pod \"maas-api-e2e-worker-single-main-3f401f-6c698d8476-s2qdv\" (UID: \"a19a2e84-eece-4b0e-904c-cda9fbc08cb2\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-3f401f-6c698d8476-s2qdv" Oct 30 21:38:38.156676 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:38:38.156653 5378 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a19a2e84-eece-4b0e-904c-cda9fbc08cb2-maas-api-tls\") pod \"maas-api-e2e-worker-single-main-3f401f-6c698d8476-s2qdv\" (UID: \"a19a2e84-eece-4b0e-904c-cda9fbc08cb2\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-3f401f-6c698d8476-s2qdv" Oct 30 21:38:38.358854 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:38:38.358817 5378 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-3f401f-6c698d8476-s2qdv" Oct 30 21:38:38.728835 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:38:38.728810 5378 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-3f401f-6c698d8476-s2qdv"] Oct 30 21:38:38.730900 ip-10-0-138-149 kubenswrapper[5378]: W1030 21:38:38.730866 5378 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda19a2e84_eece_4b0e_904c_cda9fbc08cb2.slice/crio-35c73d03ef41479773f5f5ea1051c0b143635d320c25deea5f8a982576be96ba WatchSource:0}: Error finding container 35c73d03ef41479773f5f5ea1051c0b143635d320c25deea5f8a982576be96ba: Status 404 returned error can't find the container with id 35c73d03ef41479773f5f5ea1051c0b143635d320c25deea5f8a982576be96ba Oct 30 21:38:39.327916 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:38:39.327879 5378 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-3f401f-6c698d8476-s2qdv" event={"ID":"a19a2e84-eece-4b0e-904c-cda9fbc08cb2","Type":"ContainerStarted","Data":"35c73d03ef41479773f5f5ea1051c0b143635d320c25deea5f8a982576be96ba"} Oct 30 21:38:40.333995 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:38:40.333960 5378 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-3f401f-6c698d8476-s2qdv" event={"ID":"a19a2e84-eece-4b0e-904c-cda9fbc08cb2","Type":"ContainerStarted","Data":"3f48597a22cc1a4c10daea07f4d5464a198c411a92cb0f8c605952a977863a33"} Oct 30 21:38:40.334420 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:38:40.334033 5378 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-3f401f-6c698d8476-s2qdv" Oct 30 21:38:42.550721 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:38:42.550690 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:38:45.657286 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:38:45.653974 5378 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-3f401f-6c698d8476-s2qdv" podStartSLOduration=7.781376116 podStartE2EDuration="8.653951975s" totalImagesPullingTime="872.575859ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:38:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:38:40.456725275 +0000 UTC m=+3610.884839198" watchObservedRunningTime="2026-10-30 21:38:45.653951975 +0000 UTC m=+3616.082065895" Oct 30 21:38:45.657286 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:38:45.655006 5378 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-r9g59"] Oct 30 21:38:46.344201 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:38:46.344170 5378 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-3f401f-6c698d8476-s2qdv" Oct 30 21:38:47.491822 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:38:47.491791 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:38:48.836277 ip-10-0-138-149 kubenswrapper[5378]: E1030 21:38:48.836246 5378 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q6mx7" containerName="extractor" Oct 30 21:38:52.357552 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:38:52.357519 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:38:55.426037 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:38:55.426006 5378 generic.go:356] "Generic (PLEG): container finished" podID="b6a5bc29-f3e8-4c39-9a74-32e1eb0c3ac3" containerID="9138a9a3cebea4758795e880f39cd8536a5d8f252b97dfd2716927bba0e0427e" exitCode=0 Oct 30 21:38:55.426499 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:38:55.426077 5378 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-3f401f-openshift-default-7f4658d54fbgjqn" event={"ID":"b6a5bc29-f3e8-4c39-9a74-32e1eb0c3ac3","Type":"ContainerDied","Data":"9138a9a3cebea4758795e880f39cd8536a5d8f252b97dfd2716927bba0e0427e"} Oct 30 21:38:55.426499 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:38:55.426464 5378 scope.go:122] "RemoveContainer" containerID="9138a9a3cebea4758795e880f39cd8536a5d8f252b97dfd2716927bba0e0427e" Oct 30 21:38:55.427656 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:38:55.427626 5378 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 21:38:55.427757 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:38:55.427700 5378 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 21:38:55.427757 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:38:55.427736 5378 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 21:38:56.291074 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:38:56.291021 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:38:56.432436 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:38:56.432398 5378 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-3f401f-openshift-default-7f4658d54fbgjqn" event={"ID":"b6a5bc29-f3e8-4c39-9a74-32e1eb0c3ac3","Type":"ContainerStarted","Data":"a3048ed4f4a48737a60e7c2755c3074675b9d6eb3ae86b178dd49b45e3ba798b"} Oct 30 21:38:56.579002 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:38:56.578861 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:38:56.879547 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:38:56.879421 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:38:57.002692 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:38:57.002646 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:38:57.039788 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:38:57.039756 5378 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-single-main-3f401f-openshift-default-7f4658d54fbgjqn" Oct 30 21:38:57.040927 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:38:57.040901 5378 patch_prober.go:29] interesting pod/e2e-worker-single-main-3f401f-openshift-default-7f4658d54fbgjqn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.179:15021/healthz/ready\": dial tcp 10.133.0.179:15021: connect: connection refused" start-of-body= Oct 30 21:38:57.041057 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:38:57.040947 5378 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-3f401f-openshift-default-7f4658d54fbgjqn" podUID="b6a5bc29-f3e8-4c39-9a74-32e1eb0c3ac3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.179:15021/healthz/ready\": dial tcp 10.133.0.179:15021: connect: connection refused" Oct 30 21:38:57.133890 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:38:57.133813 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:38:57.501660 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:38:57.501588 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:38:58.040054 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:38:58.040020 5378 patch_prober.go:29] interesting pod/e2e-worker-single-main-3f401f-openshift-default-7f4658d54fbgjqn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.179:15021/healthz/ready\": dial tcp 10.133.0.179:15021: connect: connection refused" start-of-body= Oct 30 21:38:58.040321 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:38:58.040072 5378 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-3f401f-openshift-default-7f4658d54fbgjqn" podUID="b6a5bc29-f3e8-4c39-9a74-32e1eb0c3ac3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.179:15021/healthz/ready\": dial tcp 10.133.0.179:15021: connect: connection refused" Oct 30 21:38:59.039634 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:38:59.039597 5378 patch_prober.go:29] interesting pod/e2e-worker-single-main-3f401f-openshift-default-7f4658d54fbgjqn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.179:15021/healthz/ready\": dial tcp 10.133.0.179:15021: connect: connection refused" start-of-body= Oct 30 21:38:59.040036 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:38:59.039659 5378 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-3f401f-openshift-default-7f4658d54fbgjqn" podUID="b6a5bc29-f3e8-4c39-9a74-32e1eb0c3ac3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.179:15021/healthz/ready\": dial tcp 10.133.0.179:15021: connect: connection refused" Oct 30 21:38:59.285831 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:38:59.285795 5378 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-single-main-3f401f-openshift-default-7f4658d54fbgjqn" Oct 30 21:38:59.635042 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:38:59.635007 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:39:00.040188 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:39:00.040154 5378 patch_prober.go:29] interesting pod/e2e-worker-single-main-3f401f-openshift-default-7f4658d54fbgjqn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.179:15021/healthz/ready\": dial tcp 10.133.0.179:15021: connect: connection refused" start-of-body= Oct 30 21:39:00.040650 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:39:00.040209 5378 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-3f401f-openshift-default-7f4658d54fbgjqn" podUID="b6a5bc29-f3e8-4c39-9a74-32e1eb0c3ac3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.179:15021/healthz/ready\": dial tcp 10.133.0.179:15021: connect: connection refused" Oct 30 21:39:01.039914 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:39:01.039874 5378 patch_prober.go:29] interesting pod/e2e-worker-single-main-3f401f-openshift-default-7f4658d54fbgjqn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.179:15021/healthz/ready\": dial tcp 10.133.0.179:15021: connect: connection refused" start-of-body= Oct 30 21:39:01.040107 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:39:01.039937 5378 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-3f401f-openshift-default-7f4658d54fbgjqn" podUID="b6a5bc29-f3e8-4c39-9a74-32e1eb0c3ac3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.179:15021/healthz/ready\": dial tcp 10.133.0.179:15021: connect: connection refused" Oct 30 21:39:02.043879 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:39:02.043850 5378 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-single-main-3f401f-openshift-default-7f4658d54fbgjqn" Oct 30 21:39:02.044848 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:39:02.044828 5378 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-single-main-3f401f-openshift-default-7f4658d54fbgjqn" Oct 30 21:39:02.353989 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:39:02.353952 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:39:04.233965 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:39:04.233928 5378 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-r9g59"] Oct 30 21:39:07.147607 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:39:07.147555 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:39:11.952178 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:39:11.952140 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:39:16.771439 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:39:16.771410 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:39:18.836317 ip-10-0-138-149 kubenswrapper[5378]: E1030 21:39:18.836280 5378 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q6mx7" containerName="extractor" Oct 30 21:39:21.638000 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:39:21.637964 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:39:26.454635 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:39:26.454599 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:39:30.062821 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:39:30.062786 5378 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-r9g59"] Oct 30 21:39:31.251486 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:39:31.251451 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:39:32.205234 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:39:32.205204 5378 scope.go:122] "RemoveContainer" containerID="31f5d5f1e6fb6ba7285844614902b6fcc38cf88a796b9c6cf8cffa31a34f1470" Oct 30 21:39:36.165504 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:39:36.165463 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:39:40.857703 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:39:40.857666 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:39:45.641710 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:39:45.641677 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:39:48.840696 ip-10-0-138-149 kubenswrapper[5378]: E1030 21:39:48.840660 5378 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q6mx7" containerName="extractor" Oct 30 21:39:50.451477 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:39:50.451443 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:39:55.356934 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:39:55.356891 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:39:56.311926 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:39:56.311884 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:39:56.911634 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:39:56.911598 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:39:58.248954 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:39:58.248879 5378 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-r9g59"] Oct 30 21:40:00.457262 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:40:00.457227 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:40:01.374007 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:40:01.373971 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:40:05.458436 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:40:05.458398 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:40:08.660544 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:40:08.660464 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:40:08.782139 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:40:08.782107 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:40:10.255789 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:40:10.255752 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:40:14.375696 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:40:14.375646 5378 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-3f401f-6c698d8476-s2qdv"] Oct 30 21:40:14.376118 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:40:14.375857 5378 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-3f401f-6c698d8476-s2qdv" podUID="a19a2e84-eece-4b0e-904c-cda9fbc08cb2" containerName="maas-api" containerID="cri-o://3f48597a22cc1a4c10daea07f4d5464a198c411a92cb0f8c605952a977863a33" gracePeriod=30 Oct 30 21:40:14.467677 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:40:14.467638 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:40:15.630056 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:40:15.630031 5378 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-3f401f-6c698d8476-s2qdv" Oct 30 21:40:15.668547 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:40:15.668522 5378 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a19a2e84-eece-4b0e-904c-cda9fbc08cb2-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a19a2e84-eece-4b0e-904c-cda9fbc08cb2-maas-api-tls\") pod \"a19a2e84-eece-4b0e-904c-cda9fbc08cb2\" (UID: \"a19a2e84-eece-4b0e-904c-cda9fbc08cb2\") " Oct 30 21:40:15.668729 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:40:15.668560 5378 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a19a2e84-eece-4b0e-904c-cda9fbc08cb2-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a19a2e84-eece-4b0e-904c-cda9fbc08cb2-openshift-service-ca\") pod \"a19a2e84-eece-4b0e-904c-cda9fbc08cb2\" (UID: \"a19a2e84-eece-4b0e-904c-cda9fbc08cb2\") " Oct 30 21:40:15.668729 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:40:15.668683 5378 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a19a2e84-eece-4b0e-904c-cda9fbc08cb2-kube-api-access-lc7lv\" (UniqueName: \"kubernetes.io/projected/a19a2e84-eece-4b0e-904c-cda9fbc08cb2-kube-api-access-lc7lv\") pod \"a19a2e84-eece-4b0e-904c-cda9fbc08cb2\" (UID: \"a19a2e84-eece-4b0e-904c-cda9fbc08cb2\") " Oct 30 21:40:15.668872 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:40:15.668760 5378 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a19a2e84-eece-4b0e-904c-cda9fbc08cb2-metrics-certs\" (UniqueName: \"kubernetes.io/secret/a19a2e84-eece-4b0e-904c-cda9fbc08cb2-metrics-certs\") pod \"a19a2e84-eece-4b0e-904c-cda9fbc08cb2\" (UID: \"a19a2e84-eece-4b0e-904c-cda9fbc08cb2\") " Oct 30 21:40:15.668953 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:40:15.668931 5378 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a19a2e84-eece-4b0e-904c-cda9fbc08cb2-openshift-service-ca" pod "a19a2e84-eece-4b0e-904c-cda9fbc08cb2" (UID: "a19a2e84-eece-4b0e-904c-cda9fbc08cb2"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:40:15.669130 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:40:15.669109 5378 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a19a2e84-eece-4b0e-904c-cda9fbc08cb2-openshift-service-ca\") on node \"ip-10-0-138-149.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:40:15.670751 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:40:15.670725 5378 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a19a2e84-eece-4b0e-904c-cda9fbc08cb2-metrics-certs" pod "a19a2e84-eece-4b0e-904c-cda9fbc08cb2" (UID: "a19a2e84-eece-4b0e-904c-cda9fbc08cb2"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:40:15.670847 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:40:15.670781 5378 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a19a2e84-eece-4b0e-904c-cda9fbc08cb2-maas-api-tls" pod "a19a2e84-eece-4b0e-904c-cda9fbc08cb2" (UID: "a19a2e84-eece-4b0e-904c-cda9fbc08cb2"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:40:15.670847 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:40:15.670841 5378 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a19a2e84-eece-4b0e-904c-cda9fbc08cb2-kube-api-access-lc7lv" pod "a19a2e84-eece-4b0e-904c-cda9fbc08cb2" (UID: "a19a2e84-eece-4b0e-904c-cda9fbc08cb2"). InnerVolumeSpecName "kube-api-access-lc7lv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:40:15.770635 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:40:15.770604 5378 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lc7lv\" (UniqueName: \"kubernetes.io/projected/a19a2e84-eece-4b0e-904c-cda9fbc08cb2-kube-api-access-lc7lv\") on node \"ip-10-0-138-149.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:40:15.770635 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:40:15.770638 5378 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a19a2e84-eece-4b0e-904c-cda9fbc08cb2-metrics-certs\") on node \"ip-10-0-138-149.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:40:15.770819 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:40:15.770653 5378 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a19a2e84-eece-4b0e-904c-cda9fbc08cb2-maas-api-tls\") on node \"ip-10-0-138-149.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:40:15.836275 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:40:15.836239 5378 generic.go:356] "Generic (PLEG): container finished" podID="a19a2e84-eece-4b0e-904c-cda9fbc08cb2" containerID="3f48597a22cc1a4c10daea07f4d5464a198c411a92cb0f8c605952a977863a33" exitCode=0 Oct 30 21:40:15.836405 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:40:15.836289 5378 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-3f401f-6c698d8476-s2qdv" Oct 30 21:40:15.836405 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:40:15.836321 5378 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-3f401f-6c698d8476-s2qdv" event={"ID":"a19a2e84-eece-4b0e-904c-cda9fbc08cb2","Type":"ContainerDied","Data":"3f48597a22cc1a4c10daea07f4d5464a198c411a92cb0f8c605952a977863a33"} Oct 30 21:40:15.836405 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:40:15.836375 5378 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-3f401f-6c698d8476-s2qdv" event={"ID":"a19a2e84-eece-4b0e-904c-cda9fbc08cb2","Type":"ContainerDied","Data":"35c73d03ef41479773f5f5ea1051c0b143635d320c25deea5f8a982576be96ba"} Oct 30 21:40:15.836405 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:40:15.836399 5378 scope.go:122] "RemoveContainer" containerID="3f48597a22cc1a4c10daea07f4d5464a198c411a92cb0f8c605952a977863a33" Oct 30 21:40:15.845009 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:40:15.844991 5378 scope.go:122] "RemoveContainer" containerID="3f48597a22cc1a4c10daea07f4d5464a198c411a92cb0f8c605952a977863a33" Oct 30 21:40:15.845251 ip-10-0-138-149 kubenswrapper[5378]: E1030 21:40:15.845233 5378 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3f48597a22cc1a4c10daea07f4d5464a198c411a92cb0f8c605952a977863a33\": container with ID starting with 3f48597a22cc1a4c10daea07f4d5464a198c411a92cb0f8c605952a977863a33 not found: ID does not exist" containerID="3f48597a22cc1a4c10daea07f4d5464a198c411a92cb0f8c605952a977863a33" Oct 30 21:40:15.845323 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:40:15.845259 5378 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3f48597a22cc1a4c10daea07f4d5464a198c411a92cb0f8c605952a977863a33"} err="failed to get container status \"3f48597a22cc1a4c10daea07f4d5464a198c411a92cb0f8c605952a977863a33\": rpc error: code = NotFound desc = could not find container \"3f48597a22cc1a4c10daea07f4d5464a198c411a92cb0f8c605952a977863a33\": container with ID starting with 3f48597a22cc1a4c10daea07f4d5464a198c411a92cb0f8c605952a977863a33 not found: ID does not exist" Oct 30 21:40:16.000860 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:40:16.000790 5378 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-3f401f-6c698d8476-s2qdv"] Oct 30 21:40:16.004468 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:40:16.004444 5378 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-3f401f-6c698d8476-s2qdv"] Oct 30 21:40:16.029686 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:40:16.029661 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:40:16.157047 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:40:16.157018 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:40:16.303959 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:40:16.303885 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:40:17.109226 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:40:17.109193 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:40:17.739359 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:40:17.739325 5378 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a19a2e84-eece-4b0e-904c-cda9fbc08cb2" path="/var/lib/kubelet/pods/a19a2e84-eece-4b0e-904c-cda9fbc08cb2/volumes" Oct 30 21:40:18.837223 ip-10-0-138-149 kubenswrapper[5378]: E1030 21:40:18.837191 5378 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q6mx7" containerName="extractor" Oct 30 21:40:19.150233 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:40:19.150201 5378 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-r9g59"] Oct 30 21:40:22.716910 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:40:22.716875 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:40:22.844751 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:40:22.844676 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:40:23.064293 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:40:23.064210 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:40:23.069956 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:40:23.069925 5378 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-single-main-3f401f-openshift-default-7f4658d54fbgjqn"] Oct 30 21:40:23.070394 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:40:23.070355 5378 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-single-main-3f401f-openshift-default-7f4658d54fbgjqn" podUID="b6a5bc29-f3e8-4c39-9a74-32e1eb0c3ac3" containerName="istio-proxy" containerID="cri-o://a3048ed4f4a48737a60e7c2755c3074675b9d6eb3ae86b178dd49b45e3ba798b" gracePeriod=30 Oct 30 21:40:28.325742 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:40:28.325718 5378 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-single-main-3f401f-openshift-default-7f4658d54fbgjqn" Oct 30 21:40:28.394438 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:40:28.394411 5378 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b6a5bc29-f3e8-4c39-9a74-32e1eb0c3ac3-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b6a5bc29-f3e8-4c39-9a74-32e1eb0c3ac3-workload-certs\") pod \"b6a5bc29-f3e8-4c39-9a74-32e1eb0c3ac3\" (UID: \"b6a5bc29-f3e8-4c39-9a74-32e1eb0c3ac3\") " Oct 30 21:40:28.394593 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:40:28.394455 5378 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b6a5bc29-f3e8-4c39-9a74-32e1eb0c3ac3-kube-api-access-bbqm8\" (UniqueName: \"kubernetes.io/projected/b6a5bc29-f3e8-4c39-9a74-32e1eb0c3ac3-kube-api-access-bbqm8\") pod \"b6a5bc29-f3e8-4c39-9a74-32e1eb0c3ac3\" (UID: \"b6a5bc29-f3e8-4c39-9a74-32e1eb0c3ac3\") " Oct 30 21:40:28.394593 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:40:28.394545 5378 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b6a5bc29-f3e8-4c39-9a74-32e1eb0c3ac3-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b6a5bc29-f3e8-4c39-9a74-32e1eb0c3ac3-workload-socket\") pod \"b6a5bc29-f3e8-4c39-9a74-32e1eb0c3ac3\" (UID: \"b6a5bc29-f3e8-4c39-9a74-32e1eb0c3ac3\") " Oct 30 21:40:28.394723 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:40:28.394669 5378 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b6a5bc29-f3e8-4c39-9a74-32e1eb0c3ac3-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b6a5bc29-f3e8-4c39-9a74-32e1eb0c3ac3-istio-data\") pod \"b6a5bc29-f3e8-4c39-9a74-32e1eb0c3ac3\" (UID: \"b6a5bc29-f3e8-4c39-9a74-32e1eb0c3ac3\") " Oct 30 21:40:28.394785 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:40:28.394722 5378 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b6a5bc29-f3e8-4c39-9a74-32e1eb0c3ac3-workload-certs" pod "b6a5bc29-f3e8-4c39-9a74-32e1eb0c3ac3" (UID: "b6a5bc29-f3e8-4c39-9a74-32e1eb0c3ac3"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:40:28.394785 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:40:28.394732 5378 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b6a5bc29-f3e8-4c39-9a74-32e1eb0c3ac3-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b6a5bc29-f3e8-4c39-9a74-32e1eb0c3ac3-istio-envoy\") pod \"b6a5bc29-f3e8-4c39-9a74-32e1eb0c3ac3\" (UID: \"b6a5bc29-f3e8-4c39-9a74-32e1eb0c3ac3\") " Oct 30 21:40:28.394896 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:40:28.394798 5378 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b6a5bc29-f3e8-4c39-9a74-32e1eb0c3ac3-workload-socket" pod "b6a5bc29-f3e8-4c39-9a74-32e1eb0c3ac3" (UID: "b6a5bc29-f3e8-4c39-9a74-32e1eb0c3ac3"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:40:28.394896 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:40:28.394800 5378 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/b6a5bc29-f3e8-4c39-9a74-32e1eb0c3ac3-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b6a5bc29-f3e8-4c39-9a74-32e1eb0c3ac3-istio-podinfo\") pod \"b6a5bc29-f3e8-4c39-9a74-32e1eb0c3ac3\" (UID: \"b6a5bc29-f3e8-4c39-9a74-32e1eb0c3ac3\") " Oct 30 21:40:28.395013 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:40:28.394896 5378 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b6a5bc29-f3e8-4c39-9a74-32e1eb0c3ac3-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b6a5bc29-f3e8-4c39-9a74-32e1eb0c3ac3-istiod-ca-cert\") pod \"b6a5bc29-f3e8-4c39-9a74-32e1eb0c3ac3\" (UID: \"b6a5bc29-f3e8-4c39-9a74-32e1eb0c3ac3\") " Oct 30 21:40:28.395013 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:40:28.394899 5378 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b6a5bc29-f3e8-4c39-9a74-32e1eb0c3ac3-istio-data" pod "b6a5bc29-f3e8-4c39-9a74-32e1eb0c3ac3" (UID: "b6a5bc29-f3e8-4c39-9a74-32e1eb0c3ac3"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:40:28.395013 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:40:28.394919 5378 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b6a5bc29-f3e8-4c39-9a74-32e1eb0c3ac3-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b6a5bc29-f3e8-4c39-9a74-32e1eb0c3ac3-credential-socket\") pod \"b6a5bc29-f3e8-4c39-9a74-32e1eb0c3ac3\" (UID: \"b6a5bc29-f3e8-4c39-9a74-32e1eb0c3ac3\") " Oct 30 21:40:28.395013 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:40:28.394968 5378 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b6a5bc29-f3e8-4c39-9a74-32e1eb0c3ac3-istio-token\" (UniqueName: \"kubernetes.io/projected/b6a5bc29-f3e8-4c39-9a74-32e1eb0c3ac3-istio-token\") pod \"b6a5bc29-f3e8-4c39-9a74-32e1eb0c3ac3\" (UID: \"b6a5bc29-f3e8-4c39-9a74-32e1eb0c3ac3\") " Oct 30 21:40:28.395238 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:40:28.395175 5378 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b6a5bc29-f3e8-4c39-9a74-32e1eb0c3ac3-credential-socket" pod "b6a5bc29-f3e8-4c39-9a74-32e1eb0c3ac3" (UID: "b6a5bc29-f3e8-4c39-9a74-32e1eb0c3ac3"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:40:28.395238 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:40:28.395195 5378 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b6a5bc29-f3e8-4c39-9a74-32e1eb0c3ac3-istiod-ca-cert" pod "b6a5bc29-f3e8-4c39-9a74-32e1eb0c3ac3" (UID: "b6a5bc29-f3e8-4c39-9a74-32e1eb0c3ac3"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:40:28.395354 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:40:28.395298 5378 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b6a5bc29-f3e8-4c39-9a74-32e1eb0c3ac3-istiod-ca-cert\") on node \"ip-10-0-138-149.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:40:28.395354 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:40:28.395313 5378 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b6a5bc29-f3e8-4c39-9a74-32e1eb0c3ac3-credential-socket\") on node \"ip-10-0-138-149.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:40:28.395354 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:40:28.395322 5378 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b6a5bc29-f3e8-4c39-9a74-32e1eb0c3ac3-workload-certs\") on node \"ip-10-0-138-149.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:40:28.395354 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:40:28.395335 5378 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b6a5bc29-f3e8-4c39-9a74-32e1eb0c3ac3-workload-socket\") on node \"ip-10-0-138-149.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:40:28.395354 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:40:28.395349 5378 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b6a5bc29-f3e8-4c39-9a74-32e1eb0c3ac3-istio-data\") on node \"ip-10-0-138-149.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:40:28.396899 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:40:28.396873 5378 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b6a5bc29-f3e8-4c39-9a74-32e1eb0c3ac3-kube-api-access-bbqm8" pod "b6a5bc29-f3e8-4c39-9a74-32e1eb0c3ac3" (UID: "b6a5bc29-f3e8-4c39-9a74-32e1eb0c3ac3"). InnerVolumeSpecName "kube-api-access-bbqm8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:40:28.396899 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:40:28.396877 5378 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b6a5bc29-f3e8-4c39-9a74-32e1eb0c3ac3-istio-podinfo" pod "b6a5bc29-f3e8-4c39-9a74-32e1eb0c3ac3" (UID: "b6a5bc29-f3e8-4c39-9a74-32e1eb0c3ac3"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 21:40:28.396899 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:40:28.396902 5378 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b6a5bc29-f3e8-4c39-9a74-32e1eb0c3ac3-istio-envoy" pod "b6a5bc29-f3e8-4c39-9a74-32e1eb0c3ac3" (UID: "b6a5bc29-f3e8-4c39-9a74-32e1eb0c3ac3"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:40:28.397084 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:40:28.396977 5378 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b6a5bc29-f3e8-4c39-9a74-32e1eb0c3ac3-istio-token" pod "b6a5bc29-f3e8-4c39-9a74-32e1eb0c3ac3" (UID: "b6a5bc29-f3e8-4c39-9a74-32e1eb0c3ac3"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:40:28.496263 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:40:28.496237 5378 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b6a5bc29-f3e8-4c39-9a74-32e1eb0c3ac3-istio-token\") on node \"ip-10-0-138-149.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:40:28.496263 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:40:28.496260 5378 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bbqm8\" (UniqueName: \"kubernetes.io/projected/b6a5bc29-f3e8-4c39-9a74-32e1eb0c3ac3-kube-api-access-bbqm8\") on node \"ip-10-0-138-149.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:40:28.496448 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:40:28.496272 5378 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b6a5bc29-f3e8-4c39-9a74-32e1eb0c3ac3-istio-envoy\") on node \"ip-10-0-138-149.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:40:28.496448 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:40:28.496280 5378 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b6a5bc29-f3e8-4c39-9a74-32e1eb0c3ac3-istio-podinfo\") on node \"ip-10-0-138-149.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:40:28.896124 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:40:28.896093 5378 generic.go:356] "Generic (PLEG): container finished" podID="b6a5bc29-f3e8-4c39-9a74-32e1eb0c3ac3" containerID="a3048ed4f4a48737a60e7c2755c3074675b9d6eb3ae86b178dd49b45e3ba798b" exitCode=0 Oct 30 21:40:28.896335 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:40:28.896146 5378 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-single-main-3f401f-openshift-default-7f4658d54fbgjqn" Oct 30 21:40:28.896335 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:40:28.896180 5378 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-3f401f-openshift-default-7f4658d54fbgjqn" event={"ID":"b6a5bc29-f3e8-4c39-9a74-32e1eb0c3ac3","Type":"ContainerDied","Data":"a3048ed4f4a48737a60e7c2755c3074675b9d6eb3ae86b178dd49b45e3ba798b"} Oct 30 21:40:28.896335 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:40:28.896247 5378 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-3f401f-openshift-default-7f4658d54fbgjqn" event={"ID":"b6a5bc29-f3e8-4c39-9a74-32e1eb0c3ac3","Type":"ContainerDied","Data":"0ff97a7c2c5f9a3e7ddad277dd8f4f2378a099fe4deef3822abf42c68d7d141d"} Oct 30 21:40:28.896335 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:40:28.896278 5378 scope.go:122] "RemoveContainer" containerID="a3048ed4f4a48737a60e7c2755c3074675b9d6eb3ae86b178dd49b45e3ba798b" Oct 30 21:40:28.908346 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:40:28.908331 5378 scope.go:122] "RemoveContainer" containerID="9138a9a3cebea4758795e880f39cd8536a5d8f252b97dfd2716927bba0e0427e" Oct 30 21:40:28.916050 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:40:28.916034 5378 scope.go:122] "RemoveContainer" containerID="a3048ed4f4a48737a60e7c2755c3074675b9d6eb3ae86b178dd49b45e3ba798b" Oct 30 21:40:28.916313 ip-10-0-138-149 kubenswrapper[5378]: E1030 21:40:28.916294 5378 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a3048ed4f4a48737a60e7c2755c3074675b9d6eb3ae86b178dd49b45e3ba798b\": container with ID starting with a3048ed4f4a48737a60e7c2755c3074675b9d6eb3ae86b178dd49b45e3ba798b not found: ID does not exist" containerID="a3048ed4f4a48737a60e7c2755c3074675b9d6eb3ae86b178dd49b45e3ba798b" Oct 30 21:40:28.916392 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:40:28.916328 5378 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a3048ed4f4a48737a60e7c2755c3074675b9d6eb3ae86b178dd49b45e3ba798b"} err="failed to get container status \"a3048ed4f4a48737a60e7c2755c3074675b9d6eb3ae86b178dd49b45e3ba798b\": rpc error: code = NotFound desc = could not find container \"a3048ed4f4a48737a60e7c2755c3074675b9d6eb3ae86b178dd49b45e3ba798b\": container with ID starting with a3048ed4f4a48737a60e7c2755c3074675b9d6eb3ae86b178dd49b45e3ba798b not found: ID does not exist" Oct 30 21:40:28.916392 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:40:28.916352 5378 scope.go:122] "RemoveContainer" containerID="9138a9a3cebea4758795e880f39cd8536a5d8f252b97dfd2716927bba0e0427e" Oct 30 21:40:28.916614 ip-10-0-138-149 kubenswrapper[5378]: E1030 21:40:28.916594 5378 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9138a9a3cebea4758795e880f39cd8536a5d8f252b97dfd2716927bba0e0427e\": container with ID starting with 9138a9a3cebea4758795e880f39cd8536a5d8f252b97dfd2716927bba0e0427e not found: ID does not exist" containerID="9138a9a3cebea4758795e880f39cd8536a5d8f252b97dfd2716927bba0e0427e" Oct 30 21:40:28.916665 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:40:28.916621 5378 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9138a9a3cebea4758795e880f39cd8536a5d8f252b97dfd2716927bba0e0427e"} err="failed to get container status \"9138a9a3cebea4758795e880f39cd8536a5d8f252b97dfd2716927bba0e0427e\": rpc error: code = NotFound desc = could not find container \"9138a9a3cebea4758795e880f39cd8536a5d8f252b97dfd2716927bba0e0427e\": container with ID starting with 9138a9a3cebea4758795e880f39cd8536a5d8f252b97dfd2716927bba0e0427e not found: ID does not exist" Oct 30 21:40:29.061787 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:40:29.061756 5378 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-single-main-3f401f-openshift-default-7f4658d54fbgjqn"] Oct 30 21:40:29.112315 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:40:29.112284 5378 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-single-main-3f401f-openshift-default-7f4658d54fbgjqn"] Oct 30 21:40:29.739526 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:40:29.739480 5378 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b6a5bc29-f3e8-4c39-9a74-32e1eb0c3ac3" path="/var/lib/kubelet/pods/b6a5bc29-f3e8-4c39-9a74-32e1eb0c3ac3/volumes" Oct 30 21:40:30.220625 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:40:30.220590 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:40:34.637484 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:40:34.637447 5378 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-r9g59"] Oct 30 21:40:38.111695 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:40:38.111664 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:40:38.232164 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:40:38.232123 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:40:48.836038 ip-10-0-138-149 kubenswrapper[5378]: E1030 21:40:48.836001 5378 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q6mx7" containerName="extractor" Oct 30 21:40:52.863536 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:40:52.863493 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:40:57.242378 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:40:57.242338 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:41:12.707124 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:41:12.707086 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:41:15.626551 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:41:15.626515 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:41:17.356177 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:41:17.356141 5378 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-r9g59"] Oct 30 21:41:18.848929 ip-10-0-138-149 kubenswrapper[5378]: E1030 21:41:18.848885 5378 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q6mx7" containerName="extractor" Oct 30 21:41:20.000588 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:41:20.000446 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:41:20.090588 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:41:20.089637 5378 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-769df7846c-rszcl container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.169:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 21:41:20.090588 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:41:20.089703 5378 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-rszcl" podUID="946e32af-526e-4bae-855d-8062b58d497e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.169:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 21:41:20.110597 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:41:20.101989 5378 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-769df7846c-99vbx container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.167:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 21:41:20.110597 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:41:20.102071 5378 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-99vbx" podUID="ccc2fbc1-82a9-46e7-a6a6-102582f572d3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.167:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 21:41:20.615716 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:41:20.615676 5378 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-769df7846c-hr6hd container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.172:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 21:41:20.616047 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:41:20.616015 5378 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-hr6hd" podUID="dc49893f-0dba-4aef-bc77-d6e3c962dd69" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.172:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 21:41:41.753877 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:41:41.753804 5378 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-tvb85"] Oct 30 21:41:41.754258 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:41:41.754175 5378 cpu_manager.go:414] "containerMap: removing container" podUID="a19a2e84-eece-4b0e-904c-cda9fbc08cb2" containerName="maas-api" Oct 30 21:41:41.754258 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:41:41.754186 5378 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a19a2e84-eece-4b0e-904c-cda9fbc08cb2" containerName="maas-api" Oct 30 21:41:41.754258 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:41:41.754195 5378 cpu_manager.go:414] "containerMap: removing container" podUID="b6a5bc29-f3e8-4c39-9a74-32e1eb0c3ac3" containerName="istio-proxy" Oct 30 21:41:41.754258 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:41:41.754200 5378 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b6a5bc29-f3e8-4c39-9a74-32e1eb0c3ac3" containerName="istio-proxy" Oct 30 21:41:41.754258 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:41:41.754219 5378 cpu_manager.go:414] "containerMap: removing container" podUID="b6a5bc29-f3e8-4c39-9a74-32e1eb0c3ac3" containerName="istio-proxy" Oct 30 21:41:41.754258 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:41:41.754224 5378 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b6a5bc29-f3e8-4c39-9a74-32e1eb0c3ac3" containerName="istio-proxy" Oct 30 21:41:41.754471 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:41:41.754281 5378 memory_manager.go:365] "RemoveStaleState removing state" podUID="a19a2e84-eece-4b0e-904c-cda9fbc08cb2" containerName="maas-api" Oct 30 21:41:41.754471 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:41:41.754291 5378 memory_manager.go:365] "RemoveStaleState removing state" podUID="b6a5bc29-f3e8-4c39-9a74-32e1eb0c3ac3" containerName="istio-proxy" Oct 30 21:41:41.757188 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:41:41.757171 5378 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-tvb85" Oct 30 21:41:41.810855 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:41:41.810826 5378 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 21:41:41.861067 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:41:41.861038 5378 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 21:41:41.861275 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:41:41.861151 5378 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 21:41:41.861275 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:41:41.861195 5378 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-q2mgc\"" Oct 30 21:41:41.862634 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:41:41.862613 5378 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-tvb85"] Oct 30 21:41:41.888168 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:41:41.888140 5378 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f91373c9-d082-49f4-a400-e153af14c5fa-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-452b6d7b-tvb85\" (UID: \"f91373c9-d082-49f4-a400-e153af14c5fa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-tvb85" Oct 30 21:41:41.888362 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:41:41.888341 5378 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f91373c9-d082-49f4-a400-e153af14c5fa-maas-api-xks-ca\") pod \"maas-api-revoke-sub-452b6d7b-tvb85\" (UID: \"f91373c9-d082-49f4-a400-e153af14c5fa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-tvb85" Oct 30 21:41:41.888492 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:41:41.888468 5378 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qf8c2\" (UniqueName: \"kubernetes.io/projected/f91373c9-d082-49f4-a400-e153af14c5fa-kube-api-access-qf8c2\") pod \"maas-api-revoke-sub-452b6d7b-tvb85\" (UID: \"f91373c9-d082-49f4-a400-e153af14c5fa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-tvb85" Oct 30 21:41:41.989403 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:41:41.989369 5378 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f91373c9-d082-49f4-a400-e153af14c5fa-maas-api-xks-ca\") pod \"maas-api-revoke-sub-452b6d7b-tvb85\" (UID: \"f91373c9-d082-49f4-a400-e153af14c5fa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-tvb85" Oct 30 21:41:41.989403 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:41:41.989407 5378 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qf8c2\" (UniqueName: \"kubernetes.io/projected/f91373c9-d082-49f4-a400-e153af14c5fa-kube-api-access-qf8c2\") pod \"maas-api-revoke-sub-452b6d7b-tvb85\" (UID: \"f91373c9-d082-49f4-a400-e153af14c5fa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-tvb85" Oct 30 21:41:41.989725 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:41:41.989470 5378 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f91373c9-d082-49f4-a400-e153af14c5fa-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-452b6d7b-tvb85\" (UID: \"f91373c9-d082-49f4-a400-e153af14c5fa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-tvb85" Oct 30 21:41:41.990204 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:41:41.990180 5378 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f91373c9-d082-49f4-a400-e153af14c5fa-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-452b6d7b-tvb85\" (UID: \"f91373c9-d082-49f4-a400-e153af14c5fa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-tvb85" Oct 30 21:41:41.991884 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:41:41.991863 5378 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f91373c9-d082-49f4-a400-e153af14c5fa-maas-api-xks-ca\") pod \"maas-api-revoke-sub-452b6d7b-tvb85\" (UID: \"f91373c9-d082-49f4-a400-e153af14c5fa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-tvb85" Oct 30 21:41:42.046738 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:41:42.046676 5378 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qf8c2\" (UniqueName: \"kubernetes.io/projected/f91373c9-d082-49f4-a400-e153af14c5fa-kube-api-access-qf8c2\") pod \"maas-api-revoke-sub-452b6d7b-tvb85\" (UID: \"f91373c9-d082-49f4-a400-e153af14c5fa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-tvb85" Oct 30 21:41:42.069346 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:41:42.069322 5378 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-tvb85" Oct 30 21:41:42.438517 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:41:42.438491 5378 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-tvb85"] Oct 30 21:41:42.440754 ip-10-0-138-149 kubenswrapper[5378]: W1030 21:41:42.440724 5378 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf91373c9_d082_49f4_a400_e153af14c5fa.slice/crio-6e16ee5ab30380da0ff93099a1bf862bb9d223d00070e1c3d577a1eb9322ba6c WatchSource:0}: Error finding container 6e16ee5ab30380da0ff93099a1bf862bb9d223d00070e1c3d577a1eb9322ba6c: Status 404 returned error can't find the container with id 6e16ee5ab30380da0ff93099a1bf862bb9d223d00070e1c3d577a1eb9322ba6c Oct 30 21:41:43.127623 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:41:43.127591 5378 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-r9g59"] Oct 30 21:41:43.276345 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:41:43.276312 5378 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-tvb85" event={"ID":"f91373c9-d082-49f4-a400-e153af14c5fa","Type":"ContainerStarted","Data":"7bb672010600936a302b62b2f1d334b1edd38f0ed6a1ee1439c4918843231847"} Oct 30 21:41:43.276515 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:41:43.276362 5378 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-tvb85" event={"ID":"f91373c9-d082-49f4-a400-e153af14c5fa","Type":"ContainerStarted","Data":"6e16ee5ab30380da0ff93099a1bf862bb9d223d00070e1c3d577a1eb9322ba6c"} Oct 30 21:41:43.390962 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:41:43.390875 5378 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-tvb85" podStartSLOduration=2.390860735 podStartE2EDuration="2.390860735s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:41:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:41:43.389595317 +0000 UTC m=+3793.817709238" watchObservedRunningTime="2026-10-30 21:41:43.390860735 +0000 UTC m=+3793.818974673" Oct 30 21:41:44.284637 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:41:44.284595 5378 generic.go:356] "Generic (PLEG): container finished" podID="f91373c9-d082-49f4-a400-e153af14c5fa" containerID="7bb672010600936a302b62b2f1d334b1edd38f0ed6a1ee1439c4918843231847" exitCode=0 Oct 30 21:41:44.285281 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:41:44.284680 5378 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-tvb85" event={"ID":"f91373c9-d082-49f4-a400-e153af14c5fa","Type":"ContainerDied","Data":"7bb672010600936a302b62b2f1d334b1edd38f0ed6a1ee1439c4918843231847"} Oct 30 21:41:46.258492 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:41:46.258460 5378 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-tvb85" Oct 30 21:41:46.337830 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:41:46.337794 5378 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-tvb85" event={"ID":"f91373c9-d082-49f4-a400-e153af14c5fa","Type":"ContainerDied","Data":"6e16ee5ab30380da0ff93099a1bf862bb9d223d00070e1c3d577a1eb9322ba6c"} Oct 30 21:41:46.338188 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:41:46.338170 5378 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6e16ee5ab30380da0ff93099a1bf862bb9d223d00070e1c3d577a1eb9322ba6c" Oct 30 21:41:46.338334 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:41:46.338111 5378 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-tvb85" Oct 30 21:41:46.356925 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:41:46.356893 5378 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f91373c9-d082-49f4-a400-e153af14c5fa-kube-api-access-qf8c2\" (UniqueName: \"kubernetes.io/projected/f91373c9-d082-49f4-a400-e153af14c5fa-kube-api-access-qf8c2\") pod \"f91373c9-d082-49f4-a400-e153af14c5fa\" (UID: \"f91373c9-d082-49f4-a400-e153af14c5fa\") " Oct 30 21:41:46.357169 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:41:46.357150 5378 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f91373c9-d082-49f4-a400-e153af14c5fa-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f91373c9-d082-49f4-a400-e153af14c5fa-maas-api-openshift-ca\") pod \"f91373c9-d082-49f4-a400-e153af14c5fa\" (UID: \"f91373c9-d082-49f4-a400-e153af14c5fa\") " Oct 30 21:41:46.357334 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:41:46.357317 5378 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f91373c9-d082-49f4-a400-e153af14c5fa-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f91373c9-d082-49f4-a400-e153af14c5fa-maas-api-xks-ca\") pod \"f91373c9-d082-49f4-a400-e153af14c5fa\" (UID: \"f91373c9-d082-49f4-a400-e153af14c5fa\") " Oct 30 21:41:46.358777 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:41:46.358752 5378 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f91373c9-d082-49f4-a400-e153af14c5fa-maas-api-openshift-ca" pod "f91373c9-d082-49f4-a400-e153af14c5fa" (UID: "f91373c9-d082-49f4-a400-e153af14c5fa"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:41:46.365996 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:41:46.365970 5378 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f91373c9-d082-49f4-a400-e153af14c5fa-maas-api-xks-ca" pod "f91373c9-d082-49f4-a400-e153af14c5fa" (UID: "f91373c9-d082-49f4-a400-e153af14c5fa"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:41:46.366552 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:41:46.366531 5378 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f91373c9-d082-49f4-a400-e153af14c5fa-kube-api-access-qf8c2" pod "f91373c9-d082-49f4-a400-e153af14c5fa" (UID: "f91373c9-d082-49f4-a400-e153af14c5fa"). InnerVolumeSpecName "kube-api-access-qf8c2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:41:46.458778 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:41:46.458738 5378 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f91373c9-d082-49f4-a400-e153af14c5fa-maas-api-xks-ca\") on node \"ip-10-0-138-149.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:41:46.459971 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:41:46.459903 5378 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qf8c2\" (UniqueName: \"kubernetes.io/projected/f91373c9-d082-49f4-a400-e153af14c5fa-kube-api-access-qf8c2\") on node \"ip-10-0-138-149.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:41:46.460144 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:41:46.460128 5378 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f91373c9-d082-49f4-a400-e153af14c5fa-maas-api-openshift-ca\") on node \"ip-10-0-138-149.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:41:48.836652 ip-10-0-138-149 kubenswrapper[5378]: E1030 21:41:48.836616 5378 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q6mx7" containerName="extractor" Oct 30 21:41:54.667521 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:41:54.667490 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:41:57.580572 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:41:57.580540 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:41:58.528693 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:41:58.528658 5378 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-r9g59"] Oct 30 21:42:07.117093 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:42:07.117062 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:42:11.463100 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:42:11.463063 5378 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-r9g59"] Oct 30 21:42:18.835912 ip-10-0-138-149 kubenswrapper[5378]: E1030 21:42:18.835876 5378 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q6mx7" containerName="extractor" Oct 30 21:42:24.148407 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:42:24.148370 5378 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-r9g59"] Oct 30 21:42:35.046167 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:42:35.046136 5378 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-r9g59"] Oct 30 21:42:48.839393 ip-10-0-138-149 kubenswrapper[5378]: E1030 21:42:48.839359 5378 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q6mx7" containerName="extractor" Oct 30 21:42:49.049234 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:42:49.049198 5378 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-hww5x"] Oct 30 21:42:49.099205 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:42:49.099179 5378 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-hww5x"] Oct 30 21:42:49.739182 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:42:49.739146 5378 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f1f1058e-3063-4153-b2c0-3da42e19520b" path="/var/lib/kubelet/pods/f1f1058e-3063-4153-b2c0-3da42e19520b/volumes" Oct 30 21:42:57.922749 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:42:57.922711 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:42:58.040327 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:42:58.040288 5378 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-r9g59"] Oct 30 21:42:58.428872 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:42:58.428829 5378 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-56b9cf7d95-dmq6m"] Oct 30 21:42:58.429512 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:42:58.429493 5378 cpu_manager.go:414] "containerMap: removing container" podUID="f91373c9-d082-49f4-a400-e153af14c5fa" containerName="revoke-keys" Oct 30 21:42:58.429599 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:42:58.429515 5378 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f91373c9-d082-49f4-a400-e153af14c5fa" containerName="revoke-keys" Oct 30 21:42:58.438582 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:42:58.438537 5378 memory_manager.go:365] "RemoveStaleState removing state" podUID="b6a5bc29-f3e8-4c39-9a74-32e1eb0c3ac3" containerName="istio-proxy" Oct 30 21:42:58.438745 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:42:58.438608 5378 memory_manager.go:365] "RemoveStaleState removing state" podUID="f91373c9-d082-49f4-a400-e153af14c5fa" containerName="revoke-keys" Oct 30 21:42:58.442180 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:42:58.442147 5378 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-56b9cf7d95-dmq6m" Oct 30 21:42:58.443986 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:42:58.443962 5378 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-6b964d9667-s42ks"] Oct 30 21:42:58.447717 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:42:58.447700 5378 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-6b964d9667-s42ks" Oct 30 21:42:58.485284 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:42:58.485246 5378 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-56b9cf7d95-dmq6m"] Oct 30 21:42:58.495325 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:42:58.495291 5378 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-6mqfn\"" Oct 30 21:42:58.495515 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:42:58.495390 5378 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\"" Oct 30 21:42:58.495515 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:42:58.495430 5378 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-serving-cert\"" Oct 30 21:42:58.530136 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:42:58.530098 5378 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cwsw8\" (UniqueName: \"kubernetes.io/projected/21cae4d5-8463-4e3d-80b7-273bc59743aa-kube-api-access-cwsw8\") pod \"maas-api-56b9cf7d95-dmq6m\" (UID: \"21cae4d5-8463-4e3d-80b7-273bc59743aa\") " pod="odh-ai-gateway-infra/maas-api-56b9cf7d95-dmq6m" Oct 30 21:42:58.530136 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:42:58.530137 5378 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b389974d-172e-4040-bb3f-34f077b65d28-metrics-certs\") pod \"maas-api-6b964d9667-s42ks\" (UID: \"b389974d-172e-4040-bb3f-34f077b65d28\") " pod="odh-ai-gateway-infra/maas-api-6b964d9667-s42ks" Oct 30 21:42:58.530411 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:42:58.530258 5378 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4cf7l\" (UniqueName: \"kubernetes.io/projected/b389974d-172e-4040-bb3f-34f077b65d28-kube-api-access-4cf7l\") pod \"maas-api-6b964d9667-s42ks\" (UID: \"b389974d-172e-4040-bb3f-34f077b65d28\") " pod="odh-ai-gateway-infra/maas-api-6b964d9667-s42ks" Oct 30 21:42:58.530411 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:42:58.530342 5378 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/21cae4d5-8463-4e3d-80b7-273bc59743aa-maas-api-tls\") pod \"maas-api-56b9cf7d95-dmq6m\" (UID: \"21cae4d5-8463-4e3d-80b7-273bc59743aa\") " pod="odh-ai-gateway-infra/maas-api-56b9cf7d95-dmq6m" Oct 30 21:42:58.530411 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:42:58.530364 5378 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b389974d-172e-4040-bb3f-34f077b65d28-openshift-service-ca\") pod \"maas-api-6b964d9667-s42ks\" (UID: \"b389974d-172e-4040-bb3f-34f077b65d28\") " pod="odh-ai-gateway-infra/maas-api-6b964d9667-s42ks" Oct 30 21:42:58.530641 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:42:58.530443 5378 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/21cae4d5-8463-4e3d-80b7-273bc59743aa-openshift-service-ca\") pod \"maas-api-56b9cf7d95-dmq6m\" (UID: \"21cae4d5-8463-4e3d-80b7-273bc59743aa\") " pod="odh-ai-gateway-infra/maas-api-56b9cf7d95-dmq6m" Oct 30 21:42:58.530641 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:42:58.530499 5378 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/21cae4d5-8463-4e3d-80b7-273bc59743aa-metrics-certs\") pod \"maas-api-56b9cf7d95-dmq6m\" (UID: \"21cae4d5-8463-4e3d-80b7-273bc59743aa\") " pod="odh-ai-gateway-infra/maas-api-56b9cf7d95-dmq6m" Oct 30 21:42:58.530641 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:42:58.530515 5378 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b389974d-172e-4040-bb3f-34f077b65d28-maas-api-tls\") pod \"maas-api-6b964d9667-s42ks\" (UID: \"b389974d-172e-4040-bb3f-34f077b65d28\") " pod="odh-ai-gateway-infra/maas-api-6b964d9667-s42ks" Oct 30 21:42:58.535364 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:42:58.535334 5378 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-6b964d9667-s42ks"] Oct 30 21:42:58.578621 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:42:58.578587 5378 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 21:42:58.578621 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:42:58.578603 5378 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 21:42:58.631618 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:42:58.631594 5378 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/21cae4d5-8463-4e3d-80b7-273bc59743aa-maas-api-tls\") pod \"maas-api-56b9cf7d95-dmq6m\" (UID: \"21cae4d5-8463-4e3d-80b7-273bc59743aa\") " pod="odh-ai-gateway-infra/maas-api-56b9cf7d95-dmq6m" Oct 30 21:42:58.631770 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:42:58.631624 5378 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b389974d-172e-4040-bb3f-34f077b65d28-openshift-service-ca\") pod \"maas-api-6b964d9667-s42ks\" (UID: \"b389974d-172e-4040-bb3f-34f077b65d28\") " pod="odh-ai-gateway-infra/maas-api-6b964d9667-s42ks" Oct 30 21:42:58.631770 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:42:58.631650 5378 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/21cae4d5-8463-4e3d-80b7-273bc59743aa-openshift-service-ca\") pod \"maas-api-56b9cf7d95-dmq6m\" (UID: \"21cae4d5-8463-4e3d-80b7-273bc59743aa\") " pod="odh-ai-gateway-infra/maas-api-56b9cf7d95-dmq6m" Oct 30 21:42:58.631770 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:42:58.631700 5378 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/21cae4d5-8463-4e3d-80b7-273bc59743aa-metrics-certs\") pod \"maas-api-56b9cf7d95-dmq6m\" (UID: \"21cae4d5-8463-4e3d-80b7-273bc59743aa\") " pod="odh-ai-gateway-infra/maas-api-56b9cf7d95-dmq6m" Oct 30 21:42:58.631770 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:42:58.631722 5378 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b389974d-172e-4040-bb3f-34f077b65d28-maas-api-tls\") pod \"maas-api-6b964d9667-s42ks\" (UID: \"b389974d-172e-4040-bb3f-34f077b65d28\") " pod="odh-ai-gateway-infra/maas-api-6b964d9667-s42ks" Oct 30 21:42:58.632005 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:42:58.631804 5378 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cwsw8\" (UniqueName: \"kubernetes.io/projected/21cae4d5-8463-4e3d-80b7-273bc59743aa-kube-api-access-cwsw8\") pod \"maas-api-56b9cf7d95-dmq6m\" (UID: \"21cae4d5-8463-4e3d-80b7-273bc59743aa\") " pod="odh-ai-gateway-infra/maas-api-56b9cf7d95-dmq6m" Oct 30 21:42:58.632005 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:42:58.631825 5378 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b389974d-172e-4040-bb3f-34f077b65d28-metrics-certs\") pod \"maas-api-6b964d9667-s42ks\" (UID: \"b389974d-172e-4040-bb3f-34f077b65d28\") " pod="odh-ai-gateway-infra/maas-api-6b964d9667-s42ks" Oct 30 21:42:58.632005 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:42:58.631851 5378 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4cf7l\" (UniqueName: \"kubernetes.io/projected/b389974d-172e-4040-bb3f-34f077b65d28-kube-api-access-4cf7l\") pod \"maas-api-6b964d9667-s42ks\" (UID: \"b389974d-172e-4040-bb3f-34f077b65d28\") " pod="odh-ai-gateway-infra/maas-api-6b964d9667-s42ks" Oct 30 21:42:58.632440 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:42:58.632413 5378 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b389974d-172e-4040-bb3f-34f077b65d28-openshift-service-ca\") pod \"maas-api-6b964d9667-s42ks\" (UID: \"b389974d-172e-4040-bb3f-34f077b65d28\") " pod="odh-ai-gateway-infra/maas-api-6b964d9667-s42ks" Oct 30 21:42:58.632533 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:42:58.632499 5378 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/21cae4d5-8463-4e3d-80b7-273bc59743aa-openshift-service-ca\") pod \"maas-api-56b9cf7d95-dmq6m\" (UID: \"21cae4d5-8463-4e3d-80b7-273bc59743aa\") " pod="odh-ai-gateway-infra/maas-api-56b9cf7d95-dmq6m" Oct 30 21:42:58.634682 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:42:58.634649 5378 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b389974d-172e-4040-bb3f-34f077b65d28-maas-api-tls\") pod \"maas-api-6b964d9667-s42ks\" (UID: \"b389974d-172e-4040-bb3f-34f077b65d28\") " pod="odh-ai-gateway-infra/maas-api-6b964d9667-s42ks" Oct 30 21:42:58.634792 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:42:58.634648 5378 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b389974d-172e-4040-bb3f-34f077b65d28-metrics-certs\") pod \"maas-api-6b964d9667-s42ks\" (UID: \"b389974d-172e-4040-bb3f-34f077b65d28\") " pod="odh-ai-gateway-infra/maas-api-6b964d9667-s42ks" Oct 30 21:42:58.634792 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:42:58.634716 5378 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/21cae4d5-8463-4e3d-80b7-273bc59743aa-metrics-certs\") pod \"maas-api-56b9cf7d95-dmq6m\" (UID: \"21cae4d5-8463-4e3d-80b7-273bc59743aa\") " pod="odh-ai-gateway-infra/maas-api-56b9cf7d95-dmq6m" Oct 30 21:42:58.634880 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:42:58.634788 5378 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/21cae4d5-8463-4e3d-80b7-273bc59743aa-maas-api-tls\") pod \"maas-api-56b9cf7d95-dmq6m\" (UID: \"21cae4d5-8463-4e3d-80b7-273bc59743aa\") " pod="odh-ai-gateway-infra/maas-api-56b9cf7d95-dmq6m" Oct 30 21:42:58.689210 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:42:58.689150 5378 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cwsw8\" (UniqueName: \"kubernetes.io/projected/21cae4d5-8463-4e3d-80b7-273bc59743aa-kube-api-access-cwsw8\") pod \"maas-api-56b9cf7d95-dmq6m\" (UID: \"21cae4d5-8463-4e3d-80b7-273bc59743aa\") " pod="odh-ai-gateway-infra/maas-api-56b9cf7d95-dmq6m" Oct 30 21:42:58.689321 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:42:58.689244 5378 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4cf7l\" (UniqueName: \"kubernetes.io/projected/b389974d-172e-4040-bb3f-34f077b65d28-kube-api-access-4cf7l\") pod \"maas-api-6b964d9667-s42ks\" (UID: \"b389974d-172e-4040-bb3f-34f077b65d28\") " pod="odh-ai-gateway-infra/maas-api-6b964d9667-s42ks" Oct 30 21:42:58.758497 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:42:58.758470 5378 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-56b9cf7d95-dmq6m" Oct 30 21:42:58.767391 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:42:58.767362 5378 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-6b964d9667-s42ks" Oct 30 21:42:59.152764 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:42:59.152735 5378 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-56b9cf7d95-dmq6m"] Oct 30 21:42:59.154445 ip-10-0-138-149 kubenswrapper[5378]: W1030 21:42:59.154417 5378 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod21cae4d5_8463_4e3d_80b7_273bc59743aa.slice/crio-cd863ffccfef12a6c9f96bc256772b25a5e839a5abe617e30361aeba8bcf4ef1 WatchSource:0}: Error finding container cd863ffccfef12a6c9f96bc256772b25a5e839a5abe617e30361aeba8bcf4ef1: Status 404 returned error can't find the container with id cd863ffccfef12a6c9f96bc256772b25a5e839a5abe617e30361aeba8bcf4ef1 Oct 30 21:42:59.155911 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:42:59.155894 5378 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 30 21:42:59.190033 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:42:59.190006 5378 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-6b964d9667-s42ks"] Oct 30 21:42:59.203118 ip-10-0-138-149 kubenswrapper[5378]: W1030 21:42:59.203091 5378 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb389974d_172e_4040_bb3f_34f077b65d28.slice/crio-632838d7ffdbbdb771ee24abbdb253f0539f0b92af800360c199c0990df7824e WatchSource:0}: Error finding container 632838d7ffdbbdb771ee24abbdb253f0539f0b92af800360c199c0990df7824e: Status 404 returned error can't find the container with id 632838d7ffdbbdb771ee24abbdb253f0539f0b92af800360c199c0990df7824e Oct 30 21:42:59.740079 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:42:59.740042 5378 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-56b9cf7d95-dmq6m" event={"ID":"21cae4d5-8463-4e3d-80b7-273bc59743aa","Type":"ContainerStarted","Data":"cd863ffccfef12a6c9f96bc256772b25a5e839a5abe617e30361aeba8bcf4ef1"} Oct 30 21:42:59.740422 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:42:59.740393 5378 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-6b964d9667-s42ks" event={"ID":"b389974d-172e-4040-bb3f-34f077b65d28","Type":"ContainerStarted","Data":"632838d7ffdbbdb771ee24abbdb253f0539f0b92af800360c199c0990df7824e"} Oct 30 21:43:00.746883 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:43:00.746843 5378 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-56b9cf7d95-dmq6m" event={"ID":"21cae4d5-8463-4e3d-80b7-273bc59743aa","Type":"ContainerStarted","Data":"ef314ad0f7f66ca78d3e99a8376ea60dc68fb4ad6c90cc1dd7c8688639951b18"} Oct 30 21:43:00.747291 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:43:00.746905 5378 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-56b9cf7d95-dmq6m" Oct 30 21:43:00.748464 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:43:00.748437 5378 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-6b964d9667-s42ks" event={"ID":"b389974d-172e-4040-bb3f-34f077b65d28","Type":"ContainerStarted","Data":"7731f17b93741db28d666ca5119b18629111f3a09ea5ab167f3147c185b8a4f6"} Oct 30 21:43:00.748652 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:43:00.748516 5378 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-6b964d9667-s42ks" Oct 30 21:43:00.862666 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:43:00.862621 5378 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-56b9cf7d95-dmq6m" podStartSLOduration=2.052962162 podStartE2EDuration="2.862608855s" totalImagesPullingTime="809.646693ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:42:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:43:00.861513527 +0000 UTC m=+3871.289627445" watchObservedRunningTime="2026-10-30 21:43:00.862608855 +0000 UTC m=+3871.290722775" Oct 30 21:43:00.977423 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:43:00.977371 5378 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-6b964d9667-s42ks" podStartSLOduration=2.159661934 podStartE2EDuration="2.977357025s" totalImagesPullingTime="817.695091ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:42:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:43:00.975987222 +0000 UTC m=+3871.404101155" watchObservedRunningTime="2026-10-30 21:43:00.977357025 +0000 UTC m=+3871.405470969" Oct 30 21:43:05.311291 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:43:05.311256 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:43:06.018175 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:43:06.018144 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:43:06.314136 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:43:06.314047 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:43:06.760321 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:43:06.760292 5378 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-6b964d9667-s42ks" Oct 30 21:43:06.760646 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:43:06.760624 5378 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-56b9cf7d95-dmq6m" Oct 30 21:43:06.930101 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:43:06.930071 5378 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-56b9cf7d95-dmq6m"] Oct 30 21:43:06.930269 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:43:06.930253 5378 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-56b9cf7d95-dmq6m" podUID="21cae4d5-8463-4e3d-80b7-273bc59743aa" containerName="maas-api" containerID="cri-o://ef314ad0f7f66ca78d3e99a8376ea60dc68fb4ad6c90cc1dd7c8688639951b18" gracePeriod=30 Oct 30 21:43:07.179838 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:43:07.179808 5378 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-56b9cf7d95-dmq6m" Oct 30 21:43:07.217060 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:43:07.217026 5378 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/21cae4d5-8463-4e3d-80b7-273bc59743aa-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/21cae4d5-8463-4e3d-80b7-273bc59743aa-maas-api-tls\") pod \"21cae4d5-8463-4e3d-80b7-273bc59743aa\" (UID: \"21cae4d5-8463-4e3d-80b7-273bc59743aa\") " Oct 30 21:43:07.217224 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:43:07.217078 5378 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/21cae4d5-8463-4e3d-80b7-273bc59743aa-metrics-certs\" (UniqueName: \"kubernetes.io/secret/21cae4d5-8463-4e3d-80b7-273bc59743aa-metrics-certs\") pod \"21cae4d5-8463-4e3d-80b7-273bc59743aa\" (UID: \"21cae4d5-8463-4e3d-80b7-273bc59743aa\") " Oct 30 21:43:07.217224 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:43:07.217140 5378 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/21cae4d5-8463-4e3d-80b7-273bc59743aa-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/21cae4d5-8463-4e3d-80b7-273bc59743aa-openshift-service-ca\") pod \"21cae4d5-8463-4e3d-80b7-273bc59743aa\" (UID: \"21cae4d5-8463-4e3d-80b7-273bc59743aa\") " Oct 30 21:43:07.217224 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:43:07.217212 5378 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/21cae4d5-8463-4e3d-80b7-273bc59743aa-kube-api-access-cwsw8\" (UniqueName: \"kubernetes.io/projected/21cae4d5-8463-4e3d-80b7-273bc59743aa-kube-api-access-cwsw8\") pod \"21cae4d5-8463-4e3d-80b7-273bc59743aa\" (UID: \"21cae4d5-8463-4e3d-80b7-273bc59743aa\") " Oct 30 21:43:07.217550 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:43:07.217525 5378 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/21cae4d5-8463-4e3d-80b7-273bc59743aa-openshift-service-ca" pod "21cae4d5-8463-4e3d-80b7-273bc59743aa" (UID: "21cae4d5-8463-4e3d-80b7-273bc59743aa"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:43:07.219255 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:43:07.219233 5378 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/21cae4d5-8463-4e3d-80b7-273bc59743aa-maas-api-tls" pod "21cae4d5-8463-4e3d-80b7-273bc59743aa" (UID: "21cae4d5-8463-4e3d-80b7-273bc59743aa"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:43:07.219367 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:43:07.219345 5378 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/21cae4d5-8463-4e3d-80b7-273bc59743aa-kube-api-access-cwsw8" pod "21cae4d5-8463-4e3d-80b7-273bc59743aa" (UID: "21cae4d5-8463-4e3d-80b7-273bc59743aa"). InnerVolumeSpecName "kube-api-access-cwsw8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:43:07.219408 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:43:07.219346 5378 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/21cae4d5-8463-4e3d-80b7-273bc59743aa-metrics-certs" pod "21cae4d5-8463-4e3d-80b7-273bc59743aa" (UID: "21cae4d5-8463-4e3d-80b7-273bc59743aa"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:43:07.318260 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:43:07.318181 5378 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/21cae4d5-8463-4e3d-80b7-273bc59743aa-maas-api-tls\") on node \"ip-10-0-138-149.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:43:07.318260 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:43:07.318208 5378 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/21cae4d5-8463-4e3d-80b7-273bc59743aa-metrics-certs\") on node \"ip-10-0-138-149.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:43:07.318260 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:43:07.318218 5378 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/21cae4d5-8463-4e3d-80b7-273bc59743aa-openshift-service-ca\") on node \"ip-10-0-138-149.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:43:07.318260 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:43:07.318226 5378 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cwsw8\" (UniqueName: \"kubernetes.io/projected/21cae4d5-8463-4e3d-80b7-273bc59743aa-kube-api-access-cwsw8\") on node \"ip-10-0-138-149.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:43:07.780722 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:43:07.780687 5378 generic.go:356] "Generic (PLEG): container finished" podID="21cae4d5-8463-4e3d-80b7-273bc59743aa" containerID="ef314ad0f7f66ca78d3e99a8376ea60dc68fb4ad6c90cc1dd7c8688639951b18" exitCode=0 Oct 30 21:43:07.780883 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:43:07.780757 5378 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-56b9cf7d95-dmq6m" Oct 30 21:43:07.780883 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:43:07.780760 5378 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-56b9cf7d95-dmq6m" event={"ID":"21cae4d5-8463-4e3d-80b7-273bc59743aa","Type":"ContainerDied","Data":"ef314ad0f7f66ca78d3e99a8376ea60dc68fb4ad6c90cc1dd7c8688639951b18"} Oct 30 21:43:07.780883 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:43:07.780881 5378 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-56b9cf7d95-dmq6m" event={"ID":"21cae4d5-8463-4e3d-80b7-273bc59743aa","Type":"ContainerDied","Data":"cd863ffccfef12a6c9f96bc256772b25a5e839a5abe617e30361aeba8bcf4ef1"} Oct 30 21:43:07.781032 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:43:07.780912 5378 scope.go:122] "RemoveContainer" containerID="ef314ad0f7f66ca78d3e99a8376ea60dc68fb4ad6c90cc1dd7c8688639951b18" Oct 30 21:43:07.789513 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:43:07.789492 5378 scope.go:122] "RemoveContainer" containerID="ef314ad0f7f66ca78d3e99a8376ea60dc68fb4ad6c90cc1dd7c8688639951b18" Oct 30 21:43:07.789817 ip-10-0-138-149 kubenswrapper[5378]: E1030 21:43:07.789796 5378 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ef314ad0f7f66ca78d3e99a8376ea60dc68fb4ad6c90cc1dd7c8688639951b18\": container with ID starting with ef314ad0f7f66ca78d3e99a8376ea60dc68fb4ad6c90cc1dd7c8688639951b18 not found: ID does not exist" containerID="ef314ad0f7f66ca78d3e99a8376ea60dc68fb4ad6c90cc1dd7c8688639951b18" Oct 30 21:43:07.789890 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:43:07.789824 5378 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ef314ad0f7f66ca78d3e99a8376ea60dc68fb4ad6c90cc1dd7c8688639951b18"} err="failed to get container status \"ef314ad0f7f66ca78d3e99a8376ea60dc68fb4ad6c90cc1dd7c8688639951b18\": rpc error: code = NotFound desc = could not find container \"ef314ad0f7f66ca78d3e99a8376ea60dc68fb4ad6c90cc1dd7c8688639951b18\": container with ID starting with ef314ad0f7f66ca78d3e99a8376ea60dc68fb4ad6c90cc1dd7c8688639951b18 not found: ID does not exist" Oct 30 21:43:07.946486 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:43:07.946448 5378 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-56b9cf7d95-dmq6m"] Oct 30 21:43:07.948445 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:43:07.948421 5378 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-56b9cf7d95-dmq6m"] Oct 30 21:43:09.425973 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:43:09.425893 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:43:09.621853 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:43:09.621823 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:43:09.740023 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:43:09.739948 5378 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="21cae4d5-8463-4e3d-80b7-273bc59743aa" path="/var/lib/kubelet/pods/21cae4d5-8463-4e3d-80b7-273bc59743aa/volumes" Oct 30 21:43:09.786269 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:43:09.786239 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:43:12.714228 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:43:12.714199 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:43:12.872777 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:43:12.872749 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:43:13.032073 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:43:13.031982 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:43:13.775374 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:43:13.775345 5378 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-1a1f4ae2-6c4dc6b768-t2ww6"] Oct 30 21:43:13.775846 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:43:13.775743 5378 cpu_manager.go:414] "containerMap: removing container" podUID="21cae4d5-8463-4e3d-80b7-273bc59743aa" containerName="maas-api" Oct 30 21:43:13.775846 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:43:13.775755 5378 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="21cae4d5-8463-4e3d-80b7-273bc59743aa" containerName="maas-api" Oct 30 21:43:13.775846 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:43:13.775842 5378 memory_manager.go:365] "RemoveStaleState removing state" podUID="21cae4d5-8463-4e3d-80b7-273bc59743aa" containerName="maas-api" Oct 30 21:43:13.780226 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:43:13.780205 5378 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-1a1f4ae2-6c4dc6b768-t2ww6" Oct 30 21:43:13.870823 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:43:13.870789 5378 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7dcaf495-0291-4336-8ee1-2b646a36d5c8-metrics-certs\") pod \"maas-api-e2e-ait-1a1f4ae2-6c4dc6b768-t2ww6\" (UID: \"7dcaf495-0291-4336-8ee1-2b646a36d5c8\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-1a1f4ae2-6c4dc6b768-t2ww6" Oct 30 21:43:13.870823 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:43:13.870828 5378 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n8kwr\" (UniqueName: \"kubernetes.io/projected/7dcaf495-0291-4336-8ee1-2b646a36d5c8-kube-api-access-n8kwr\") pod \"maas-api-e2e-ait-1a1f4ae2-6c4dc6b768-t2ww6\" (UID: \"7dcaf495-0291-4336-8ee1-2b646a36d5c8\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-1a1f4ae2-6c4dc6b768-t2ww6" Oct 30 21:43:13.871026 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:43:13.870932 5378 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7dcaf495-0291-4336-8ee1-2b646a36d5c8-maas-api-tls\") pod \"maas-api-e2e-ait-1a1f4ae2-6c4dc6b768-t2ww6\" (UID: \"7dcaf495-0291-4336-8ee1-2b646a36d5c8\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-1a1f4ae2-6c4dc6b768-t2ww6" Oct 30 21:43:13.871026 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:43:13.870981 5378 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7dcaf495-0291-4336-8ee1-2b646a36d5c8-openshift-service-ca\") pod \"maas-api-e2e-ait-1a1f4ae2-6c4dc6b768-t2ww6\" (UID: \"7dcaf495-0291-4336-8ee1-2b646a36d5c8\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-1a1f4ae2-6c4dc6b768-t2ww6" Oct 30 21:43:13.884607 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:43:13.884585 5378 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-1a1f4ae2-serving-cert\"" Oct 30 21:43:13.886208 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:43:13.886188 5378 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-1a1f4ae2-6c4dc6b768-t2ww6"] Oct 30 21:43:13.969504 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:43:13.969474 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:43:13.971680 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:43:13.971653 5378 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7dcaf495-0291-4336-8ee1-2b646a36d5c8-metrics-certs\") pod \"maas-api-e2e-ait-1a1f4ae2-6c4dc6b768-t2ww6\" (UID: \"7dcaf495-0291-4336-8ee1-2b646a36d5c8\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-1a1f4ae2-6c4dc6b768-t2ww6" Oct 30 21:43:13.971746 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:43:13.971704 5378 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n8kwr\" (UniqueName: \"kubernetes.io/projected/7dcaf495-0291-4336-8ee1-2b646a36d5c8-kube-api-access-n8kwr\") pod \"maas-api-e2e-ait-1a1f4ae2-6c4dc6b768-t2ww6\" (UID: \"7dcaf495-0291-4336-8ee1-2b646a36d5c8\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-1a1f4ae2-6c4dc6b768-t2ww6" Oct 30 21:43:13.971804 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:43:13.971783 5378 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7dcaf495-0291-4336-8ee1-2b646a36d5c8-maas-api-tls\") pod \"maas-api-e2e-ait-1a1f4ae2-6c4dc6b768-t2ww6\" (UID: \"7dcaf495-0291-4336-8ee1-2b646a36d5c8\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-1a1f4ae2-6c4dc6b768-t2ww6" Oct 30 21:43:13.971841 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:43:13.971817 5378 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7dcaf495-0291-4336-8ee1-2b646a36d5c8-openshift-service-ca\") pod \"maas-api-e2e-ait-1a1f4ae2-6c4dc6b768-t2ww6\" (UID: \"7dcaf495-0291-4336-8ee1-2b646a36d5c8\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-1a1f4ae2-6c4dc6b768-t2ww6" Oct 30 21:43:13.971953 ip-10-0-138-149 kubenswrapper[5378]: E1030 21:43:13.971933 5378 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ait-1a1f4ae2-serving-cert: secret "maas-api-e2e-ait-1a1f4ae2-serving-cert" not found Oct 30 21:43:13.972866 ip-10-0-138-149 kubenswrapper[5378]: E1030 21:43:13.972739 5378 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7dcaf495-0291-4336-8ee1-2b646a36d5c8-maas-api-tls podName:7dcaf495-0291-4336-8ee1-2b646a36d5c8 nodeName:}" failed. No retries permitted until 2026-10-30 21:43:14.472710062 +0000 UTC m=+3884.900823970 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/7dcaf495-0291-4336-8ee1-2b646a36d5c8-maas-api-tls") pod "maas-api-e2e-ait-1a1f4ae2-6c4dc6b768-t2ww6" (UID: "7dcaf495-0291-4336-8ee1-2b646a36d5c8") : secret "maas-api-e2e-ait-1a1f4ae2-serving-cert" not found Oct 30 21:43:13.980796 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:43:13.973431 5378 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7dcaf495-0291-4336-8ee1-2b646a36d5c8-openshift-service-ca\") pod \"maas-api-e2e-ait-1a1f4ae2-6c4dc6b768-t2ww6\" (UID: \"7dcaf495-0291-4336-8ee1-2b646a36d5c8\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-1a1f4ae2-6c4dc6b768-t2ww6" Oct 30 21:43:13.982222 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:43:13.982197 5378 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7dcaf495-0291-4336-8ee1-2b646a36d5c8-metrics-certs\") pod \"maas-api-e2e-ait-1a1f4ae2-6c4dc6b768-t2ww6\" (UID: \"7dcaf495-0291-4336-8ee1-2b646a36d5c8\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-1a1f4ae2-6c4dc6b768-t2ww6" Oct 30 21:43:14.030154 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:43:14.030094 5378 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n8kwr\" (UniqueName: \"kubernetes.io/projected/7dcaf495-0291-4336-8ee1-2b646a36d5c8-kube-api-access-n8kwr\") pod \"maas-api-e2e-ait-1a1f4ae2-6c4dc6b768-t2ww6\" (UID: \"7dcaf495-0291-4336-8ee1-2b646a36d5c8\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-1a1f4ae2-6c4dc6b768-t2ww6" Oct 30 21:43:14.145583 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:43:14.145543 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:43:14.280703 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:43:14.280626 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:43:14.476384 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:43:14.476349 5378 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7dcaf495-0291-4336-8ee1-2b646a36d5c8-maas-api-tls\") pod \"maas-api-e2e-ait-1a1f4ae2-6c4dc6b768-t2ww6\" (UID: \"7dcaf495-0291-4336-8ee1-2b646a36d5c8\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-1a1f4ae2-6c4dc6b768-t2ww6" Oct 30 21:43:14.480294 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:43:14.480261 5378 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7dcaf495-0291-4336-8ee1-2b646a36d5c8-maas-api-tls\") pod \"maas-api-e2e-ait-1a1f4ae2-6c4dc6b768-t2ww6\" (UID: \"7dcaf495-0291-4336-8ee1-2b646a36d5c8\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-1a1f4ae2-6c4dc6b768-t2ww6" Oct 30 21:43:14.692969 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:43:14.692937 5378 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-1a1f4ae2-6c4dc6b768-t2ww6" Oct 30 21:43:15.064274 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:43:15.064248 5378 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-1a1f4ae2-6c4dc6b768-t2ww6"] Oct 30 21:43:15.069699 ip-10-0-138-149 kubenswrapper[5378]: W1030 21:43:15.069668 5378 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7dcaf495_0291_4336_8ee1_2b646a36d5c8.slice/crio-ad6c75b60f6567b8148f57317462ab79619b1c547041a5388036fea6e36d2fae WatchSource:0}: Error finding container ad6c75b60f6567b8148f57317462ab79619b1c547041a5388036fea6e36d2fae: Status 404 returned error can't find the container with id ad6c75b60f6567b8148f57317462ab79619b1c547041a5388036fea6e36d2fae Oct 30 21:43:15.820951 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:43:15.820914 5378 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-1a1f4ae2-6c4dc6b768-t2ww6" event={"ID":"7dcaf495-0291-4336-8ee1-2b646a36d5c8","Type":"ContainerStarted","Data":"ad6c75b60f6567b8148f57317462ab79619b1c547041a5388036fea6e36d2fae"} Oct 30 21:43:16.828924 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:43:16.828717 5378 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-1a1f4ae2-6c4dc6b768-t2ww6" event={"ID":"7dcaf495-0291-4336-8ee1-2b646a36d5c8","Type":"ContainerStarted","Data":"233b0e53d4d33ed53fe81e46088bbdeb580890f0719914027db4ccef8ca6d161"} Oct 30 21:43:16.829354 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:43:16.828985 5378 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-1a1f4ae2-6c4dc6b768-t2ww6" Oct 30 21:43:16.946420 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:43:16.946371 5378 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-1a1f4ae2-6c4dc6b768-t2ww6" podStartSLOduration=3.128980287 podStartE2EDuration="3.94635162s" totalImagesPullingTime="817.371333ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:43:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:43:16.945199813 +0000 UTC m=+3887.373313733" watchObservedRunningTime="2026-10-30 21:43:16.94635162 +0000 UTC m=+3887.374465539" Oct 30 21:43:18.835692 ip-10-0-138-149 kubenswrapper[5378]: E1030 21:43:18.835653 5378 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q6mx7" containerName="extractor" Oct 30 21:43:19.180115 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:43:19.180080 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:43:22.390884 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:43:22.390849 5378 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-r9g59"] Oct 30 21:43:22.839677 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:43:22.839597 5378 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-1a1f4ae2-6c4dc6b768-t2ww6" Oct 30 21:43:24.066403 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:43:24.066360 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:43:28.868620 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:43:28.868581 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:43:30.261971 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:43:30.261943 5378 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-7x9jg_c2ff359c-cf92-4a5b-8610-f6eb41acbe73/ovn-acl-logging/0.log" Oct 30 21:43:30.274266 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:43:30.274235 5378 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-7x9jg_c2ff359c-cf92-4a5b-8610-f6eb41acbe73/ovn-acl-logging/0.log" Oct 30 21:43:34.056750 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:43:34.056715 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:43:39.215733 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:43:39.215700 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:43:44.135466 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:43:44.135430 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:43:48.836374 ip-10-0-138-149 kubenswrapper[5378]: E1030 21:43:48.836337 5378 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q6mx7" containerName="extractor" Oct 30 21:43:48.982347 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:43:48.982311 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:43:53.851787 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:43:53.851750 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:43:58.033421 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:43:58.033375 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:43:58.158166 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:43:58.158131 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:43:58.279485 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:43:58.279456 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:43:58.666508 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:43:58.666472 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:44:02.817693 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:44:02.817661 5378 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-1a1f4ae2-6c4dc6b768-t2ww6"] Oct 30 21:44:02.818134 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:44:02.817877 5378 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-1a1f4ae2-6c4dc6b768-t2ww6" podUID="7dcaf495-0291-4336-8ee1-2b646a36d5c8" containerName="maas-api" containerID="cri-o://233b0e53d4d33ed53fe81e46088bbdeb580890f0719914027db4ccef8ca6d161" gracePeriod=30 Oct 30 21:44:02.834676 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:44:02.834650 5378 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-ait-1a1f4ae2-6c4dc6b768-t2ww6" podUID="7dcaf495-0291-4336-8ee1-2b646a36d5c8" containerName="maas-api" probeResult="failure" output="Get \"https://10.133.0.184:8443/health\": dial tcp 10.133.0.184:8443: connect: connection refused" Oct 30 21:44:02.902897 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:44:02.902818 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:44:03.065352 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:44:03.065319 5378 generic.go:356] "Generic (PLEG): container finished" podID="7dcaf495-0291-4336-8ee1-2b646a36d5c8" containerID="233b0e53d4d33ed53fe81e46088bbdeb580890f0719914027db4ccef8ca6d161" exitCode=0 Oct 30 21:44:03.065526 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:44:03.065378 5378 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-1a1f4ae2-6c4dc6b768-t2ww6" event={"ID":"7dcaf495-0291-4336-8ee1-2b646a36d5c8","Type":"ContainerDied","Data":"233b0e53d4d33ed53fe81e46088bbdeb580890f0719914027db4ccef8ca6d161"} Oct 30 21:44:03.098792 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:44:03.098764 5378 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-1a1f4ae2-6c4dc6b768-t2ww6" Oct 30 21:44:03.165874 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:44:03.165849 5378 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7dcaf495-0291-4336-8ee1-2b646a36d5c8-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7dcaf495-0291-4336-8ee1-2b646a36d5c8-openshift-service-ca\") pod \"7dcaf495-0291-4336-8ee1-2b646a36d5c8\" (UID: \"7dcaf495-0291-4336-8ee1-2b646a36d5c8\") " Oct 30 21:44:03.166041 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:44:03.165926 5378 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7dcaf495-0291-4336-8ee1-2b646a36d5c8-metrics-certs\" (UniqueName: \"kubernetes.io/secret/7dcaf495-0291-4336-8ee1-2b646a36d5c8-metrics-certs\") pod \"7dcaf495-0291-4336-8ee1-2b646a36d5c8\" (UID: \"7dcaf495-0291-4336-8ee1-2b646a36d5c8\") " Oct 30 21:44:03.166041 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:44:03.166019 5378 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7dcaf495-0291-4336-8ee1-2b646a36d5c8-kube-api-access-n8kwr\" (UniqueName: \"kubernetes.io/projected/7dcaf495-0291-4336-8ee1-2b646a36d5c8-kube-api-access-n8kwr\") pod \"7dcaf495-0291-4336-8ee1-2b646a36d5c8\" (UID: \"7dcaf495-0291-4336-8ee1-2b646a36d5c8\") " Oct 30 21:44:03.166175 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:44:03.166072 5378 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7dcaf495-0291-4336-8ee1-2b646a36d5c8-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7dcaf495-0291-4336-8ee1-2b646a36d5c8-maas-api-tls\") pod \"7dcaf495-0291-4336-8ee1-2b646a36d5c8\" (UID: \"7dcaf495-0291-4336-8ee1-2b646a36d5c8\") " Oct 30 21:44:03.166261 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:44:03.166238 5378 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7dcaf495-0291-4336-8ee1-2b646a36d5c8-openshift-service-ca" pod "7dcaf495-0291-4336-8ee1-2b646a36d5c8" (UID: "7dcaf495-0291-4336-8ee1-2b646a36d5c8"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:44:03.166372 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:44:03.166347 5378 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7dcaf495-0291-4336-8ee1-2b646a36d5c8-openshift-service-ca\") on node \"ip-10-0-138-149.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:44:03.168164 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:44:03.168137 5378 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7dcaf495-0291-4336-8ee1-2b646a36d5c8-metrics-certs" pod "7dcaf495-0291-4336-8ee1-2b646a36d5c8" (UID: "7dcaf495-0291-4336-8ee1-2b646a36d5c8"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:44:03.168268 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:44:03.168192 5378 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7dcaf495-0291-4336-8ee1-2b646a36d5c8-maas-api-tls" pod "7dcaf495-0291-4336-8ee1-2b646a36d5c8" (UID: "7dcaf495-0291-4336-8ee1-2b646a36d5c8"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:44:03.168268 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:44:03.168212 5378 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7dcaf495-0291-4336-8ee1-2b646a36d5c8-kube-api-access-n8kwr" pod "7dcaf495-0291-4336-8ee1-2b646a36d5c8" (UID: "7dcaf495-0291-4336-8ee1-2b646a36d5c8"). InnerVolumeSpecName "kube-api-access-n8kwr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:44:03.267326 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:44:03.267293 5378 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n8kwr\" (UniqueName: \"kubernetes.io/projected/7dcaf495-0291-4336-8ee1-2b646a36d5c8-kube-api-access-n8kwr\") on node \"ip-10-0-138-149.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:44:03.267326 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:44:03.267325 5378 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7dcaf495-0291-4336-8ee1-2b646a36d5c8-maas-api-tls\") on node \"ip-10-0-138-149.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:44:03.267326 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:44:03.267335 5378 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7dcaf495-0291-4336-8ee1-2b646a36d5c8-metrics-certs\") on node \"ip-10-0-138-149.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:44:04.072627 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:44:04.072541 5378 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-1a1f4ae2-6c4dc6b768-t2ww6" Oct 30 21:44:04.073074 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:44:04.072532 5378 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-1a1f4ae2-6c4dc6b768-t2ww6" event={"ID":"7dcaf495-0291-4336-8ee1-2b646a36d5c8","Type":"ContainerDied","Data":"ad6c75b60f6567b8148f57317462ab79619b1c547041a5388036fea6e36d2fae"} Oct 30 21:44:04.073074 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:44:04.072698 5378 scope.go:122] "RemoveContainer" containerID="233b0e53d4d33ed53fe81e46088bbdeb580890f0719914027db4ccef8ca6d161" Oct 30 21:44:04.237877 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:44:04.237834 5378 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-1a1f4ae2-6c4dc6b768-t2ww6"] Oct 30 21:44:04.288241 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:44:04.288201 5378 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-1a1f4ae2-6c4dc6b768-t2ww6"] Oct 30 21:44:04.438812 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:44:04.438776 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:44:04.806763 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:44:04.806680 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:44:05.741104 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:44:05.741053 5378 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7dcaf495-0291-4336-8ee1-2b646a36d5c8" path="/var/lib/kubelet/pods/7dcaf495-0291-4336-8ee1-2b646a36d5c8/volumes" Oct 30 21:44:07.207398 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:44:07.207367 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:44:07.370096 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:44:07.370059 5378 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-r9g59"] Oct 30 21:44:08.527968 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:44:08.527930 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:44:09.156851 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:44:09.156819 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:44:09.326698 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:44:09.326662 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:44:13.544726 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:44:13.544687 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:44:13.721346 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:44:13.721315 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:44:13.859747 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:44:13.859717 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:44:16.787633 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:44:16.787591 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:44:16.949913 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:44:16.949877 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:44:17.522421 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:44:17.522383 5378 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-authcfg-http-0e8a1a-757f7548cf-9c982"] Oct 30 21:44:17.523395 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:44:17.523353 5378 cpu_manager.go:414] "containerMap: removing container" podUID="7dcaf495-0291-4336-8ee1-2b646a36d5c8" containerName="maas-api" Oct 30 21:44:17.523625 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:44:17.523600 5378 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7dcaf495-0291-4336-8ee1-2b646a36d5c8" containerName="maas-api" Oct 30 21:44:17.524148 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:44:17.524128 5378 memory_manager.go:365] "RemoveStaleState removing state" podUID="7dcaf495-0291-4336-8ee1-2b646a36d5c8" containerName="maas-api" Oct 30 21:44:17.529956 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:44:17.529935 5378 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-0e8a1a-757f7548cf-9c982" Oct 30 21:44:17.583216 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:44:17.583173 5378 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-authcfg-http-0e8a1a-757f7548cf-9c982"] Oct 30 21:44:17.618411 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:44:17.618372 5378 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ea89ddbc-b1a4-434b-9100-50dae9deb8bc-plugins-config-volume\") pod \"payload-pre-processing-e2e-authcfg-http-0e8a1a-757f7548cf-9c982\" (UID: \"ea89ddbc-b1a4-434b-9100-50dae9deb8bc\") " pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-0e8a1a-757f7548cf-9c982" Oct 30 21:44:17.618630 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:44:17.618545 5378 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lcggr\" (UniqueName: \"kubernetes.io/projected/ea89ddbc-b1a4-434b-9100-50dae9deb8bc-kube-api-access-lcggr\") pod \"payload-pre-processing-e2e-authcfg-http-0e8a1a-757f7548cf-9c982\" (UID: \"ea89ddbc-b1a4-434b-9100-50dae9deb8bc\") " pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-0e8a1a-757f7548cf-9c982" Oct 30 21:44:17.631886 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:44:17.631861 5378 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-authcfg-http-0e8a1a\"" Oct 30 21:44:17.631966 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:44:17.631878 5378 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-authcfg-http-0e8a1a-dockercfg-nk8st\"" Oct 30 21:44:17.719341 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:44:17.719292 5378 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lcggr\" (UniqueName: \"kubernetes.io/projected/ea89ddbc-b1a4-434b-9100-50dae9deb8bc-kube-api-access-lcggr\") pod \"payload-pre-processing-e2e-authcfg-http-0e8a1a-757f7548cf-9c982\" (UID: \"ea89ddbc-b1a4-434b-9100-50dae9deb8bc\") " pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-0e8a1a-757f7548cf-9c982" Oct 30 21:44:17.719341 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:44:17.719348 5378 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ea89ddbc-b1a4-434b-9100-50dae9deb8bc-plugins-config-volume\") pod \"payload-pre-processing-e2e-authcfg-http-0e8a1a-757f7548cf-9c982\" (UID: \"ea89ddbc-b1a4-434b-9100-50dae9deb8bc\") " pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-0e8a1a-757f7548cf-9c982" Oct 30 21:44:17.720073 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:44:17.720050 5378 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ea89ddbc-b1a4-434b-9100-50dae9deb8bc-plugins-config-volume\") pod \"payload-pre-processing-e2e-authcfg-http-0e8a1a-757f7548cf-9c982\" (UID: \"ea89ddbc-b1a4-434b-9100-50dae9deb8bc\") " pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-0e8a1a-757f7548cf-9c982" Oct 30 21:44:17.778172 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:44:17.778095 5378 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lcggr\" (UniqueName: \"kubernetes.io/projected/ea89ddbc-b1a4-434b-9100-50dae9deb8bc-kube-api-access-lcggr\") pod \"payload-pre-processing-e2e-authcfg-http-0e8a1a-757f7548cf-9c982\" (UID: \"ea89ddbc-b1a4-434b-9100-50dae9deb8bc\") " pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-0e8a1a-757f7548cf-9c982" Oct 30 21:44:17.843858 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:44:17.843818 5378 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-0e8a1a-757f7548cf-9c982" Oct 30 21:44:18.211515 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:44:18.211489 5378 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-authcfg-http-0e8a1a-757f7548cf-9c982"] Oct 30 21:44:18.214132 ip-10-0-138-149 kubenswrapper[5378]: W1030 21:44:18.214104 5378 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podea89ddbc_b1a4_434b_9100_50dae9deb8bc.slice/crio-ce30e071a143ec573c73d992393d58ce0f95c9f4b5e7818fcd3b53981cb84fa4 WatchSource:0}: Error finding container ce30e071a143ec573c73d992393d58ce0f95c9f4b5e7818fcd3b53981cb84fa4: Status 404 returned error can't find the container with id ce30e071a143ec573c73d992393d58ce0f95c9f4b5e7818fcd3b53981cb84fa4 Oct 30 21:44:18.836461 ip-10-0-138-149 kubenswrapper[5378]: E1030 21:44:18.836431 5378 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q6mx7" containerName="extractor" Oct 30 21:44:19.147690 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:44:19.147655 5378 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-0e8a1a-757f7548cf-9c982" event={"ID":"ea89ddbc-b1a4-434b-9100-50dae9deb8bc","Type":"ContainerStarted","Data":"704d00361831f14e977e5330a620d772ad418a8363f00008afef58595462254d"} Oct 30 21:44:19.148115 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:44:19.147707 5378 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-0e8a1a-757f7548cf-9c982" event={"ID":"ea89ddbc-b1a4-434b-9100-50dae9deb8bc","Type":"ContainerStarted","Data":"ce30e071a143ec573c73d992393d58ce0f95c9f4b5e7818fcd3b53981cb84fa4"} Oct 30 21:44:19.148115 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:44:19.147774 5378 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-0e8a1a-757f7548cf-9c982" Oct 30 21:44:19.263643 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:44:19.263596 5378 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-0e8a1a-757f7548cf-9c982" podStartSLOduration=2.263556056 podStartE2EDuration="2.263556056s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:44:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:44:19.26241651 +0000 UTC m=+3949.690530430" watchObservedRunningTime="2026-10-30 21:44:19.263556056 +0000 UTC m=+3949.691669976" Oct 30 21:44:19.811378 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:44:19.811340 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:44:19.971457 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:44:19.971425 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:44:20.097659 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:44:20.097628 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:44:24.756987 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:44:24.756955 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:44:27.858676 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:44:27.858643 5378 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-r9g59"] Oct 30 21:44:29.742856 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:44:29.742824 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:44:30.155819 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:44:30.155790 5378 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-0e8a1a-757f7548cf-9c982" Oct 30 21:44:31.479142 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:44:31.479108 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:44:32.264292 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:44:32.264257 5378 scope.go:122] "RemoveContainer" containerID="e7907eb781730fea7c8328361e805d3a8a6a7ea8a6dbf06a98a28a6e4110adb7" Oct 30 21:44:34.572584 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:44:34.572537 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:44:38.467628 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:44:38.467532 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:44:39.672954 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:44:39.672920 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:44:44.463975 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:44:44.463940 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:44:47.639947 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:44:47.639912 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:44:48.837348 ip-10-0-138-149 kubenswrapper[5378]: E1030 21:44:48.837313 5378 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q6mx7" containerName="extractor" Oct 30 21:44:49.348170 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:44:49.348133 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:44:54.176256 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:44:54.176222 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:44:58.589458 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:44:58.589418 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:44:59.054382 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:44:59.054341 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:45:03.870035 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:45:03.870001 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:45:08.674249 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:45:08.674210 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:45:09.710764 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:45:09.710722 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:45:13.463807 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:45:13.463775 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:45:18.269194 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:45:18.269160 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:45:18.835787 ip-10-0-138-149 kubenswrapper[5378]: E1030 21:45:18.835752 5378 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q6mx7" containerName="extractor" Oct 30 21:45:23.055421 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:45:23.055382 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:45:27.872334 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:45:27.872303 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:45:32.662528 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:45:32.662484 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:45:37.466129 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:45:37.466092 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:45:42.260634 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:45:42.260601 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:45:47.072879 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:45:47.072842 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:45:48.836132 ip-10-0-138-149 kubenswrapper[5378]: E1030 21:45:48.836094 5378 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q6mx7" containerName="extractor" Oct 30 21:45:51.855298 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:45:51.855250 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:45:56.768915 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:45:56.768851 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:45:58.929862 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:45:58.929831 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:45:59.284997 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:45:59.284881 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:45:59.405177 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:45:59.405140 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:46:01.854281 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:46:01.854244 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:46:06.767734 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:46:06.767703 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:46:10.795639 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:46:10.795542 5378 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-authcfg-http-0e8a1a-757f7548cf-9c982"] Oct 30 21:46:11.038726 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:46:11.038691 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:46:11.152999 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:46:11.152930 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:46:11.570218 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:46:11.570100 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:46:15.796222 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:46:15.796163 5378 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-0e8a1a-757f7548cf-9c982" podUID="ea89ddbc-b1a4-434b-9100-50dae9deb8bc" containerName="payload-pre-processing" containerID="cri-o://704d00361831f14e977e5330a620d772ad418a8363f00008afef58595462254d" gracePeriod=25 Oct 30 21:46:16.047083 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:46:16.047014 5378 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-0e8a1a-757f7548cf-9c982" Oct 30 21:46:16.065552 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:46:16.065519 5378 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ea89ddbc-b1a4-434b-9100-50dae9deb8bc-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ea89ddbc-b1a4-434b-9100-50dae9deb8bc-plugins-config-volume\") pod \"ea89ddbc-b1a4-434b-9100-50dae9deb8bc\" (UID: \"ea89ddbc-b1a4-434b-9100-50dae9deb8bc\") " Oct 30 21:46:16.065718 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:46:16.065688 5378 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ea89ddbc-b1a4-434b-9100-50dae9deb8bc-kube-api-access-lcggr\" (UniqueName: \"kubernetes.io/projected/ea89ddbc-b1a4-434b-9100-50dae9deb8bc-kube-api-access-lcggr\") pod \"ea89ddbc-b1a4-434b-9100-50dae9deb8bc\" (UID: \"ea89ddbc-b1a4-434b-9100-50dae9deb8bc\") " Oct 30 21:46:16.065919 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:46:16.065895 5378 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ea89ddbc-b1a4-434b-9100-50dae9deb8bc-plugins-config-volume" pod "ea89ddbc-b1a4-434b-9100-50dae9deb8bc" (UID: "ea89ddbc-b1a4-434b-9100-50dae9deb8bc"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:46:16.066067 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:46:16.066045 5378 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ea89ddbc-b1a4-434b-9100-50dae9deb8bc-plugins-config-volume\") on node \"ip-10-0-138-149.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:46:16.067850 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:46:16.067826 5378 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ea89ddbc-b1a4-434b-9100-50dae9deb8bc-kube-api-access-lcggr" pod "ea89ddbc-b1a4-434b-9100-50dae9deb8bc" (UID: "ea89ddbc-b1a4-434b-9100-50dae9deb8bc"). InnerVolumeSpecName "kube-api-access-lcggr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:46:16.167132 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:46:16.167102 5378 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lcggr\" (UniqueName: \"kubernetes.io/projected/ea89ddbc-b1a4-434b-9100-50dae9deb8bc-kube-api-access-lcggr\") on node \"ip-10-0-138-149.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:46:16.699643 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:46:16.699607 5378 generic.go:356] "Generic (PLEG): container finished" podID="ea89ddbc-b1a4-434b-9100-50dae9deb8bc" containerID="704d00361831f14e977e5330a620d772ad418a8363f00008afef58595462254d" exitCode=0 Oct 30 21:46:16.699818 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:46:16.699652 5378 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-0e8a1a-757f7548cf-9c982" Oct 30 21:46:16.699818 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:46:16.699672 5378 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-0e8a1a-757f7548cf-9c982" event={"ID":"ea89ddbc-b1a4-434b-9100-50dae9deb8bc","Type":"ContainerDied","Data":"704d00361831f14e977e5330a620d772ad418a8363f00008afef58595462254d"} Oct 30 21:46:16.699818 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:46:16.699718 5378 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-0e8a1a-757f7548cf-9c982" event={"ID":"ea89ddbc-b1a4-434b-9100-50dae9deb8bc","Type":"ContainerDied","Data":"ce30e071a143ec573c73d992393d58ce0f95c9f4b5e7818fcd3b53981cb84fa4"} Oct 30 21:46:16.699818 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:46:16.699745 5378 scope.go:122] "RemoveContainer" containerID="704d00361831f14e977e5330a620d772ad418a8363f00008afef58595462254d" Oct 30 21:46:16.708967 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:46:16.708947 5378 scope.go:122] "RemoveContainer" containerID="704d00361831f14e977e5330a620d772ad418a8363f00008afef58595462254d" Oct 30 21:46:16.709234 ip-10-0-138-149 kubenswrapper[5378]: E1030 21:46:16.709214 5378 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"704d00361831f14e977e5330a620d772ad418a8363f00008afef58595462254d\": container with ID starting with 704d00361831f14e977e5330a620d772ad418a8363f00008afef58595462254d not found: ID does not exist" containerID="704d00361831f14e977e5330a620d772ad418a8363f00008afef58595462254d" Oct 30 21:46:16.709305 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:46:16.709242 5378 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"704d00361831f14e977e5330a620d772ad418a8363f00008afef58595462254d"} err="failed to get container status \"704d00361831f14e977e5330a620d772ad418a8363f00008afef58595462254d\": rpc error: code = NotFound desc = could not find container \"704d00361831f14e977e5330a620d772ad418a8363f00008afef58595462254d\": container with ID starting with 704d00361831f14e977e5330a620d772ad418a8363f00008afef58595462254d not found: ID does not exist" Oct 30 21:46:16.863775 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:46:16.863741 5378 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-authcfg-http-0e8a1a-757f7548cf-9c982"] Oct 30 21:46:16.914261 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:46:16.914236 5378 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-authcfg-http-0e8a1a-757f7548cf-9c982"] Oct 30 21:46:17.739299 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:46:17.739268 5378 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ea89ddbc-b1a4-434b-9100-50dae9deb8bc" path="/var/lib/kubelet/pods/ea89ddbc-b1a4-434b-9100-50dae9deb8bc/volumes" Oct 30 21:46:18.837074 ip-10-0-138-149 kubenswrapper[5378]: E1030 21:46:18.837034 5378 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q6mx7" containerName="extractor" Oct 30 21:46:20.833996 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:46:20.833965 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:46:22.381472 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:46:22.381433 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:46:22.660141 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:46:22.660069 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:46:23.544144 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:46:23.544107 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:46:27.039157 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:46:27.039123 5378 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-r9g59"] Oct 30 21:46:29.211533 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:46:29.211499 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:46:29.337264 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:46:29.337229 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:46:29.560609 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:46:29.560475 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:46:39.956631 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:46:39.956594 5378 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-r9g59"] Oct 30 21:46:46.049920 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:46:46.049885 5378 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-tvb85"] Oct 30 21:46:46.100103 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:46:46.100075 5378 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-tvb85"] Oct 30 21:46:47.739171 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:46:47.739141 5378 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f91373c9-d082-49f4-a400-e153af14c5fa" path="/var/lib/kubelet/pods/f91373c9-d082-49f4-a400-e153af14c5fa/volumes" Oct 30 21:46:48.835691 ip-10-0-138-149 kubenswrapper[5378]: E1030 21:46:48.835656 5378 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q6mx7" containerName="extractor" Oct 30 21:46:52.036456 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:46:52.036423 5378 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-r9g59"] Oct 30 21:46:59.269222 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:46:59.269188 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:47:13.159547 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:47:13.159508 5378 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-r9g59"] Oct 30 21:47:18.850206 ip-10-0-138-149 kubenswrapper[5378]: E1030 21:47:18.850157 5378 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q6mx7" containerName="extractor" Oct 30 21:47:20.095761 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:47:20.095711 5378 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-769df7846c-rszcl container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.169:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 21:47:20.096413 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:47:20.096362 5378 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-rszcl" podUID="946e32af-526e-4bae-855d-8062b58d497e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.169:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 21:47:20.100725 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:47:20.100700 5378 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-769df7846c-99vbx container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.167:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 21:47:20.100914 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:47:20.100891 5378 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-99vbx" podUID="ccc2fbc1-82a9-46e7-a6a6-102582f572d3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.167:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 21:47:20.612588 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:47:20.611687 5378 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-769df7846c-hr6hd container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.172:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 21:47:20.612588 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:47:20.611799 5378 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-hr6hd" podUID="dc49893f-0dba-4aef-bc77-d6e3c962dd69" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.172:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 21:47:44.930082 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:47:44.930004 5378 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-6d699dfc88-qgrhg_7f07de12-d7f7-42db-a68f-d65bb9e84c48/authorino/0.log" Oct 30 21:47:45.664162 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:47:45.664128 5378 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-r9g59"] Oct 30 21:47:47.331718 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:47:47.331678 5378 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-769df7846c-xghmf container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.40:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 21:47:47.332353 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:47:47.332317 5378 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-xghmf" podUID="f93de8dc-026f-4868-94fb-d1b2006dfb63" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.40:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 21:47:48.836199 ip-10-0-138-149 kubenswrapper[5378]: E1030 21:47:48.836163 5378 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q6mx7" containerName="extractor" Oct 30 21:47:55.711285 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:47:55.711252 5378 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_ai-gateway-controller-6c6c76f5b9-7zvnk_a1568576-80d5-42ec-af7b-99d0dd79fa20/manager/0.log" Oct 30 21:47:56.069626 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:47:56.069540 5378 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-776cc66d47-vk9qt_7bdaf64e-4296-4533-890f-c664325df4af/manager/0.log" Oct 30 21:47:57.221992 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:47:57.221962 5378 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-6cb6cd7964-tnzlz_a4403f57-3bb7-4040-80ef-e582130b97cc/manager/0.log" Oct 30 21:47:57.643573 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:47:57.643544 5378 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-75bd89fdfc-gj4j5_e598f3d1-bf40-44c3-acdb-12dcdb3a7153/server/0.log" Oct 30 21:47:57.862298 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:47:57.862264 5378 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-769875db56-66h86_cadcf041-ccba-4462-80a8-2d273479f50f/manager/0.log" Oct 30 21:47:59.127718 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:47:59.127685 5378 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-6b964d9667-s42ks_b389974d-172e-4040-bb3f-34f077b65d28/maas-api/0.log" Oct 30 21:47:59.614471 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:47:59.614439 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:48:01.728963 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:48:01.728929 5378 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-6d699dfc88-qgrhg_7f07de12-d7f7-42db-a68f-d65bb9e84c48/authorino/0.log" Oct 30 21:48:02.629526 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:48:02.629498 5378 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-66c4fb575d-b2sv2_dd4ee482-64ad-45a3-bad6-2ad6e4a3693d/kuadrant-console-plugin/0.log" Oct 30 21:48:03.334217 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:48:03.334189 5378 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-7c9b87d574-r9g59_c980ce98-3651-4f7c-bc05-467c9e14f9c2/limitador/0.log" Oct 30 21:48:04.641138 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:48:04.641107 5378 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-7ddd9cfdc8-5h7rm_7a313ddc-716a-4e94-83a5-0e2550272032/discovery/0.log" Oct 30 21:48:04.950952 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:48:04.950877 5378 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-7988bdb8dc-9bf7w_31b20f62-29dc-4320-b844-92916489fb18/kube-auth-proxy/0.log" Oct 30 21:48:05.922741 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:48:05.922711 5378 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-769df7846c-99vbx_ccc2fbc1-82a9-46e7-a6a6-102582f572d3/istio-proxy/0.log" Oct 30 21:48:06.387139 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:48:06.387113 5378 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-769df7846c-hr6hd_dc49893f-0dba-4aef-bc77-d6e3c962dd69/istio-proxy/0.log" Oct 30 21:48:06.839166 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:48:06.839087 5378 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-769df7846c-rszcl_946e32af-526e-4bae-855d-8062b58d497e/istio-proxy/0.log" Oct 30 21:48:07.615139 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:48:07.615101 5378 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-769df7846c-xghmf_f93de8dc-026f-4868-94fb-d1b2006dfb63/istio-proxy/0.log" Oct 30 21:48:08.142901 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:48:08.142873 5378 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-7b98bb678c-7l6vk_4c52be8c-3199-4691-a6ea-4f98842987c4/payload-pre-processing/0.log" Oct 30 21:48:08.367854 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:48:08.367818 5378 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5f78687d5b-2r7nt_273afb91-cad1-42d5-89c7-20d2521e082e/payload-processing/0.log" Oct 30 21:48:09.485850 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:48:09.485819 5378 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-2-simulated-kserve-cbd4f5586-xdmrj_c9c625ce-f432-4bd9-8a6f-0a01335b886b/main/0.log" Oct 30 21:48:10.219759 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:48:10.219728 5378 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-trlp-test-simulated-kserve-69b588897d-tkbxl_13d6b67c-8825-489e-91af-407eb360cae7/main/0.log" Oct 30 21:48:10.687417 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:48:10.687385 5378 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_echo-simulator-kserve-64b6f8f96c-hjktp_52a53c39-758b-41c5-be5d-729a097e58fc/main/0.log" Oct 30 21:48:12.082267 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:48:12.082240 5378 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-6d699dfc88-qgrhg_7f07de12-d7f7-42db-a68f-d65bb9e84c48/authorino/0.log" Oct 30 21:48:18.836121 ip-10-0-138-149 kubenswrapper[5378]: E1030 21:48:18.836076 5378 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q6mx7" containerName="extractor" Oct 30 21:48:21.414489 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:48:21.414455 5378 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_ai-gateway-controller-6c6c76f5b9-7zvnk_a1568576-80d5-42ec-af7b-99d0dd79fa20/manager/0.log" Oct 30 21:48:21.785034 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:48:21.784950 5378 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-776cc66d47-vk9qt_7bdaf64e-4296-4533-890f-c664325df4af/manager/0.log" Oct 30 21:48:22.705536 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:48:22.705505 5378 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-6cb6cd7964-tnzlz_a4403f57-3bb7-4040-80ef-e582130b97cc/manager/0.log" Oct 30 21:48:23.126952 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:48:23.126929 5378 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-75bd89fdfc-gj4j5_e598f3d1-bf40-44c3-acdb-12dcdb3a7153/server/0.log" Oct 30 21:48:23.369489 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:48:23.369461 5378 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-769875db56-66h86_cadcf041-ccba-4462-80a8-2d273479f50f/manager/0.log" Oct 30 21:48:24.573579 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:48:24.573539 5378 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-6b964d9667-s42ks_b389974d-172e-4040-bb3f-34f077b65d28/maas-api/0.log" Oct 30 21:48:26.736721 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:48:26.736692 5378 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-6d699dfc88-qgrhg_7f07de12-d7f7-42db-a68f-d65bb9e84c48/authorino/0.log" Oct 30 21:48:27.756122 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:48:27.756094 5378 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-66c4fb575d-b2sv2_dd4ee482-64ad-45a3-bad6-2ad6e4a3693d/kuadrant-console-plugin/0.log" Oct 30 21:48:28.413122 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:48:28.413089 5378 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-7c9b87d574-r9g59_c980ce98-3651-4f7c-bc05-467c9e14f9c2/limitador/0.log" Oct 30 21:48:29.564710 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:48:29.564681 5378 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-7ddd9cfdc8-5h7rm_7a313ddc-716a-4e94-83a5-0e2550272032/discovery/0.log" Oct 30 21:48:29.886343 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:48:29.886315 5378 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-7988bdb8dc-9bf7w_31b20f62-29dc-4320-b844-92916489fb18/kube-auth-proxy/0.log" Oct 30 21:48:30.289387 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:48:30.289312 5378 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-7x9jg_c2ff359c-cf92-4a5b-8610-f6eb41acbe73/ovn-acl-logging/0.log" Oct 30 21:48:30.310954 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:48:30.310929 5378 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-7x9jg_c2ff359c-cf92-4a5b-8610-f6eb41acbe73/ovn-acl-logging/0.log" Oct 30 21:48:30.853354 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:48:30.853318 5378 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-769df7846c-99vbx_ccc2fbc1-82a9-46e7-a6a6-102582f572d3/istio-proxy/0.log" Oct 30 21:48:31.325629 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:48:31.325517 5378 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-769df7846c-hr6hd_dc49893f-0dba-4aef-bc77-d6e3c962dd69/istio-proxy/0.log" Oct 30 21:48:31.783395 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:48:31.783368 5378 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-769df7846c-rszcl_946e32af-526e-4bae-855d-8062b58d497e/istio-proxy/0.log" Oct 30 21:48:32.313690 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:48:32.313661 5378 scope.go:122] "RemoveContainer" containerID="7bb672010600936a302b62b2f1d334b1edd38f0ed6a1ee1439c4918843231847" Oct 30 21:48:32.541590 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:48:32.541549 5378 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-769df7846c-xghmf_f93de8dc-026f-4868-94fb-d1b2006dfb63/istio-proxy/0.log" Oct 30 21:48:33.054273 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:48:33.054244 5378 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-7b98bb678c-7l6vk_4c52be8c-3199-4691-a6ea-4f98842987c4/payload-pre-processing/0.log" Oct 30 21:48:33.276157 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:48:33.276130 5378 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5f78687d5b-2r7nt_273afb91-cad1-42d5-89c7-20d2521e082e/payload-processing/0.log" Oct 30 21:48:34.185050 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:48:34.185023 5378 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-2-simulated-kserve-cbd4f5586-xdmrj_c9c625ce-f432-4bd9-8a6f-0a01335b886b/main/0.log" Oct 30 21:48:34.927064 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:48:34.927032 5378 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-trlp-test-simulated-kserve-69b588897d-tkbxl_13d6b67c-8825-489e-91af-407eb360cae7/main/0.log" Oct 30 21:48:35.381748 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:48:35.381721 5378 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_echo-simulator-kserve-64b6f8f96c-hjktp_52a53c39-758b-41c5-be5d-729a097e58fc/main/0.log" Oct 30 21:48:40.143520 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:48:40.143487 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:48:48.836054 ip-10-0-138-149 kubenswrapper[5378]: E1030 21:48:48.836016 5378 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q6mx7" containerName="extractor" Oct 30 21:49:00.000719 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:49:00.000675 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:49:04.217329 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:49:04.217301 5378 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-6cb6cd7964-tnzlz_a4403f57-3bb7-4040-80ef-e582130b97cc/manager/0.log" Oct 30 21:49:04.805088 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:49:04.805052 5378 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-6cb6cd7964-tnzlz_a4403f57-3bb7-4040-80ef-e582130b97cc/manager/0.log" Oct 30 21:49:05.590507 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:49:05.590479 5378 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5f78687d5b-2r7nt_273afb91-cad1-42d5-89c7-20d2521e082e/payload-processing/0.log" Oct 30 21:49:05.811548 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:49:05.811522 5378 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5f78687d5b-2r7nt_273afb91-cad1-42d5-89c7-20d2521e082e/payload-processing/0.log" Oct 30 21:49:18.836325 ip-10-0-138-149 kubenswrapper[5378]: E1030 21:49:18.836244 5378 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q6mx7" containerName="extractor" Oct 30 21:49:22.267223 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:49:22.267185 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:49:42.252606 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:49:42.252574 5378 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-6cb6cd7964-tnzlz_a4403f57-3bb7-4040-80ef-e582130b97cc/manager/0.log" Oct 30 21:49:42.827936 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:49:42.827901 5378 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-6cb6cd7964-tnzlz_a4403f57-3bb7-4040-80ef-e582130b97cc/manager/0.log" Oct 30 21:49:43.603351 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:49:43.603321 5378 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5f78687d5b-2r7nt_273afb91-cad1-42d5-89c7-20d2521e082e/payload-processing/0.log" Oct 30 21:49:43.820603 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:49:43.820552 5378 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5f78687d5b-2r7nt_273afb91-cad1-42d5-89c7-20d2521e082e/payload-processing/0.log" Oct 30 21:49:44.168269 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:49:44.168236 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:49:44.301421 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:49:44.301390 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:49:48.836669 ip-10-0-138-149 kubenswrapper[5378]: E1030 21:49:48.836634 5378 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q6mx7" containerName="extractor" Oct 30 21:49:52.858125 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:49:52.858090 5378 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-bcmwc_0d6769f4-dc38-45ac-863b-c75fa672866a/global-pull-secret-syncer/0.log" Oct 30 21:49:55.916087 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:49:55.916044 5378 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" Oct 30 21:49:57.713096 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:49:57.713066 5378 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-138-149.us-west-1.compute.internal_0d2e0d42ed810f19933d4cd655966ed4/haproxy/0.log" Oct 30 21:50:00.276331 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:50:00.276292 5378 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cjrkj_497c44c4-2ece-4009-b88c-6ecfa9920110/konnectivity-agent/0.log" limit=1024 Oct 30 21:50:03.094377 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:50:03.094333 5378 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-6d699dfc88-qgrhg_7f07de12-d7f7-42db-a68f-d65bb9e84c48/authorino/0.log" Oct 30 21:50:03.537426 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:50:03.537389 5378 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-must-gather-8xd6d/perf-node-gather-daemonset-njq2l"] Oct 30 21:50:03.537875 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:50:03.537862 5378 cpu_manager.go:414] "containerMap: removing container" podUID="ea89ddbc-b1a4-434b-9100-50dae9deb8bc" containerName="payload-pre-processing" Oct 30 21:50:03.537928 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:50:03.537877 5378 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ea89ddbc-b1a4-434b-9100-50dae9deb8bc" containerName="payload-pre-processing" Oct 30 21:50:03.537966 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:50:03.537939 5378 memory_manager.go:365] "RemoveStaleState removing state" podUID="ea89ddbc-b1a4-434b-9100-50dae9deb8bc" containerName="payload-pre-processing" Oct 30 21:50:03.541223 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:50:03.541203 5378 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-8xd6d/perf-node-gather-daemonset-njq2l" Oct 30 21:50:03.596367 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:50:03.596324 5378 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-8xd6d\"/\"openshift-service-ca.crt\"" Oct 30 21:50:03.644712 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:50:03.644681 5378 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/90952299-205c-4fd8-bb69-c6e590381228-proc\") pod \"perf-node-gather-daemonset-njq2l\" (UID: \"90952299-205c-4fd8-bb69-c6e590381228\") " pod="openshift-must-gather-8xd6d/perf-node-gather-daemonset-njq2l" Oct 30 21:50:03.644896 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:50:03.644721 5378 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-822dc\" (UniqueName: \"kubernetes.io/projected/90952299-205c-4fd8-bb69-c6e590381228-kube-api-access-822dc\") pod \"perf-node-gather-daemonset-njq2l\" (UID: \"90952299-205c-4fd8-bb69-c6e590381228\") " pod="openshift-must-gather-8xd6d/perf-node-gather-daemonset-njq2l" Oct 30 21:50:03.644896 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:50:03.644743 5378 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/90952299-205c-4fd8-bb69-c6e590381228-sys\") pod \"perf-node-gather-daemonset-njq2l\" (UID: \"90952299-205c-4fd8-bb69-c6e590381228\") " pod="openshift-must-gather-8xd6d/perf-node-gather-daemonset-njq2l" Oct 30 21:50:03.644896 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:50:03.644820 5378 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/90952299-205c-4fd8-bb69-c6e590381228-podres\") pod \"perf-node-gather-daemonset-njq2l\" (UID: \"90952299-205c-4fd8-bb69-c6e590381228\") " pod="openshift-must-gather-8xd6d/perf-node-gather-daemonset-njq2l" Oct 30 21:50:03.645028 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:50:03.644903 5378 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/90952299-205c-4fd8-bb69-c6e590381228-lib-modules\") pod \"perf-node-gather-daemonset-njq2l\" (UID: \"90952299-205c-4fd8-bb69-c6e590381228\") " pod="openshift-must-gather-8xd6d/perf-node-gather-daemonset-njq2l" Oct 30 21:50:03.646970 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:50:03.646948 5378 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-8xd6d\"/\"kube-root-ca.crt\"" Oct 30 21:50:03.646970 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:50:03.646949 5378 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-8xd6d\"/\"default-dockercfg-hwbdz\"" Oct 30 21:50:03.648313 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:50:03.648279 5378 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-8xd6d/perf-node-gather-daemonset-njq2l"] Oct 30 21:50:03.746125 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:50:03.746095 5378 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/90952299-205c-4fd8-bb69-c6e590381228-proc\") pod \"perf-node-gather-daemonset-njq2l\" (UID: \"90952299-205c-4fd8-bb69-c6e590381228\") " pod="openshift-must-gather-8xd6d/perf-node-gather-daemonset-njq2l" Oct 30 21:50:03.746327 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:50:03.746142 5378 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-822dc\" (UniqueName: \"kubernetes.io/projected/90952299-205c-4fd8-bb69-c6e590381228-kube-api-access-822dc\") pod \"perf-node-gather-daemonset-njq2l\" (UID: \"90952299-205c-4fd8-bb69-c6e590381228\") " pod="openshift-must-gather-8xd6d/perf-node-gather-daemonset-njq2l" Oct 30 21:50:03.746327 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:50:03.746166 5378 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/90952299-205c-4fd8-bb69-c6e590381228-sys\") pod \"perf-node-gather-daemonset-njq2l\" (UID: \"90952299-205c-4fd8-bb69-c6e590381228\") " pod="openshift-must-gather-8xd6d/perf-node-gather-daemonset-njq2l" Oct 30 21:50:03.746327 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:50:03.746213 5378 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/90952299-205c-4fd8-bb69-c6e590381228-proc\") pod \"perf-node-gather-daemonset-njq2l\" (UID: \"90952299-205c-4fd8-bb69-c6e590381228\") " pod="openshift-must-gather-8xd6d/perf-node-gather-daemonset-njq2l" Oct 30 21:50:03.746327 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:50:03.746229 5378 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/90952299-205c-4fd8-bb69-c6e590381228-sys\") pod \"perf-node-gather-daemonset-njq2l\" (UID: \"90952299-205c-4fd8-bb69-c6e590381228\") " pod="openshift-must-gather-8xd6d/perf-node-gather-daemonset-njq2l" Oct 30 21:50:03.746510 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:50:03.746329 5378 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/90952299-205c-4fd8-bb69-c6e590381228-podres\") pod \"perf-node-gather-daemonset-njq2l\" (UID: \"90952299-205c-4fd8-bb69-c6e590381228\") " pod="openshift-must-gather-8xd6d/perf-node-gather-daemonset-njq2l" Oct 30 21:50:03.746510 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:50:03.746407 5378 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/90952299-205c-4fd8-bb69-c6e590381228-podres\") pod \"perf-node-gather-daemonset-njq2l\" (UID: \"90952299-205c-4fd8-bb69-c6e590381228\") " pod="openshift-must-gather-8xd6d/perf-node-gather-daemonset-njq2l" Oct 30 21:50:03.746510 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:50:03.746425 5378 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/90952299-205c-4fd8-bb69-c6e590381228-lib-modules\") pod \"perf-node-gather-daemonset-njq2l\" (UID: \"90952299-205c-4fd8-bb69-c6e590381228\") " pod="openshift-must-gather-8xd6d/perf-node-gather-daemonset-njq2l" Oct 30 21:50:03.746632 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:50:03.746521 5378 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/90952299-205c-4fd8-bb69-c6e590381228-lib-modules\") pod \"perf-node-gather-daemonset-njq2l\" (UID: \"90952299-205c-4fd8-bb69-c6e590381228\") " pod="openshift-must-gather-8xd6d/perf-node-gather-daemonset-njq2l" Oct 30 21:50:03.807129 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:50:03.807058 5378 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-822dc\" (UniqueName: \"kubernetes.io/projected/90952299-205c-4fd8-bb69-c6e590381228-kube-api-access-822dc\") pod \"perf-node-gather-daemonset-njq2l\" (UID: \"90952299-205c-4fd8-bb69-c6e590381228\") " pod="openshift-must-gather-8xd6d/perf-node-gather-daemonset-njq2l" Oct 30 21:50:03.853193 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:50:03.853152 5378 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-8xd6d/perf-node-gather-daemonset-njq2l" Oct 30 21:50:04.222178 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:50:04.222154 5378 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-8xd6d/perf-node-gather-daemonset-njq2l"] Oct 30 21:50:04.224055 ip-10-0-138-149 kubenswrapper[5378]: W1030 21:50:04.224026 5378 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod90952299_205c_4fd8_bb69_c6e590381228.slice/crio-6fa04353d16e1c662cbb10d01224c16b05cfe553d219bb1a37de55acc2b96935 WatchSource:0}: Error finding container 6fa04353d16e1c662cbb10d01224c16b05cfe553d219bb1a37de55acc2b96935: Status 404 returned error can't find the container with id 6fa04353d16e1c662cbb10d01224c16b05cfe553d219bb1a37de55acc2b96935 Oct 30 21:50:04.249063 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:50:04.249039 5378 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-66c4fb575d-b2sv2_dd4ee482-64ad-45a3-bad6-2ad6e4a3693d/kuadrant-console-plugin/0.log" Oct 30 21:50:04.796616 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:50:04.796558 5378 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-8xd6d/perf-node-gather-daemonset-njq2l" event={"ID":"90952299-205c-4fd8-bb69-c6e590381228","Type":"ContainerStarted","Data":"25932299ef3ce11dca120fd064b2232c4f2604112317fd7b14934b9e9589a142"} Oct 30 21:50:04.796809 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:50:04.796675 5378 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-8xd6d/perf-node-gather-daemonset-njq2l" event={"ID":"90952299-205c-4fd8-bb69-c6e590381228","Type":"ContainerStarted","Data":"6fa04353d16e1c662cbb10d01224c16b05cfe553d219bb1a37de55acc2b96935"} Oct 30 21:50:04.796809 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:50:04.796706 5378 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-8xd6d/perf-node-gather-daemonset-njq2l" Oct 30 21:50:04.911372 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:50:04.911318 5378 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-must-gather-8xd6d/perf-node-gather-daemonset-njq2l" podStartSLOduration=1.911306267 podStartE2EDuration="1.911306267s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 21:50:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:50:04.909981394 +0000 UTC m=+4295.338095313" watchObservedRunningTime="2026-10-30 21:50:04.911306267 +0000 UTC m=+4295.339420186" Oct 30 21:50:05.233644 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:50:05.233610 5378 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-7c9b87d574-r9g59_c980ce98-3651-4f7c-bc05-467c9e14f9c2/limitador/0.log" Oct 30 21:50:10.812351 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:50:10.812322 5378 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-8xd6d/perf-node-gather-daemonset-njq2l" Oct 30 21:50:13.419779 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:50:13.419753 5378 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_monitoring-plugin-77f4f76594-64rbk_42765893-c3e3-41ab-811f-3135d4c74b07/monitoring-plugin/0.log" Oct 30 21:50:13.839823 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:50:13.839741 5378 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-jzxqq_7df66b3c-abe2-4d20-abda-45320cd4af1b/node-exporter/0.log" Oct 30 21:50:14.264376 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:50:14.264334 5378 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-jzxqq_7df66b3c-abe2-4d20-abda-45320cd4af1b/kube-rbac-proxy/0.log" Oct 30 21:50:14.678458 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:50:14.678426 5378 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-jzxqq_7df66b3c-abe2-4d20-abda-45320cd4af1b/init-textfile/0.log" Oct 30 21:50:18.836585 ip-10-0-138-149 kubenswrapper[5378]: E1030 21:50:18.836530 5378 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q6mx7" containerName="extractor" Oct 30 21:50:21.790735 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:50:21.790709 5378 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-9cd89b74c-2xw4j_5186676f-8c82-4667-a013-3efc22d2c4e7/prometheus-operator/0.log" Oct 30 21:50:22.204653 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:50:22.204620 5378 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-9cd89b74c-2xw4j_5186676f-8c82-4667-a013-3efc22d2c4e7/kube-rbac-proxy/0.log" Oct 30 21:50:24.285041 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:50:24.285008 5378 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-7d8d77f945-zfsgx_3bcf59e7-befb-4623-9a0f-f01b70963b3e/thanos-query/0.log" Oct 30 21:50:24.696738 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:50:24.696695 5378 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-7d8d77f945-zfsgx_3bcf59e7-befb-4623-9a0f-f01b70963b3e/kube-rbac-proxy-web/0.log" Oct 30 21:50:25.108551 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:50:25.108506 5378 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-7d8d77f945-zfsgx_3bcf59e7-befb-4623-9a0f-f01b70963b3e/kube-rbac-proxy/0.log" Oct 30 21:50:25.522314 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:50:25.522238 5378 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-7d8d77f945-zfsgx_3bcf59e7-befb-4623-9a0f-f01b70963b3e/prom-label-proxy/0.log" Oct 30 21:50:25.935966 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:50:25.935937 5378 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-7d8d77f945-zfsgx_3bcf59e7-befb-4623-9a0f-f01b70963b3e/kube-rbac-proxy-rules/0.log" Oct 30 21:50:26.350858 ip-10-0-138-149 kubenswrapper[5378]: I1030 21:50:26.350833 5378 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-7d8d77f945-zfsgx_3bcf59e7-befb-4623-9a0f-f01b70963b3e/kube-rbac-proxy-metrics/0.log"